Sep 16 18:38:32.674478 ip-10-0-142-80 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 18:38:32.674615 ip-10-0-142-80 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 18:38:32.674709 ip-10-0-142-80 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 18:38:32.675131 ip-10-0-142-80 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 18:38:42.842223 ip-10-0-142-80 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 18:38:42.842242 ip-10-0-142-80 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot e660d51da5074540b41c1b583970594e --
Sep 16 18:40:31.650336 ip-10-0-142-80 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:40:32.037328 ip-10-0-142-80 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:32.037328 ip-10-0-142-80 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:40:32.037328 ip-10-0-142-80 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:32.037328 ip-10-0-142-80 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:40:32.037328 ip-10-0-142-80 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:32.039327 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.039255    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:40:32.042432 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042418    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:32.042471 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042433    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:32.042471 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042437    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:32.042471 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042441    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:32.042471 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042444    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:32.042471 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042446    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:32.042471 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042449    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:32.042471 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042452    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:32.042471 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042455    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:32.042471 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042457    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:32.042471 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042460    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:32.042471 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042463    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:32.042471 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042465    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:32.042471 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042468    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:32.042471 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042470    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:32.042471 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042473    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:32.042471 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042476    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:32.042471 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042479    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:32.042896 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042482    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:32.042896 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042485    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:32.042896 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042495    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:32.042896 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042499    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:32.042896 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042501    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:32.042896 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042504    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:32.042896 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042507    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:32.042896 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042510    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:32.042896 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042512    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:32.042896 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042515    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:32.042896 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042517    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:32.042896 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042520    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:32.042896 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042523    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:32.042896 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042525    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:32.042896 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042528    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:32.042896 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042530    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:32.042896 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042533    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:32.042896 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042536    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:32.042896 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042538    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:32.043401 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042541    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:32.043401 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042543    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:32.043401 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042546    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:32.043401 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042548    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:32.043401 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042551    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:32.043401 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042553    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:32.043401 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042557    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:32.043401 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042559    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:32.043401 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042562    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:32.043401 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042565    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:32.043401 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042567    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:32.043401 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042570    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:32.043401 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042572    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:32.043401 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042575    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:32.043401 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042579    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:32.043401 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042582    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:32.043401 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042584    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:32.043401 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042587    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:32.043401 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042589    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:32.043853 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042592    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:32.043853 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042594    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:32.043853 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042597    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:32.043853 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042599    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:32.043853 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042602    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:32.043853 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042605    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:32.043853 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042607    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:32.043853 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042609    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:32.043853 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042612    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:32.043853 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042615    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:32.043853 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042618    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:32.043853 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042620    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:32.043853 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042623    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:32.043853 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042625    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:32.043853 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042627    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:32.043853 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042630    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:32.043853 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042632    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:32.043853 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042635    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:32.043853 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042637    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:32.043853 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042640    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:32.044345 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042642    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:32.044345 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042645    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:32.044345 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042647    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:32.044345 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042650    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:32.044345 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042653    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:32.044345 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042655    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:32.044345 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042660    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:32.044345 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042665    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:32.044345 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042669    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:32.044345 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042672    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:32.044345 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042675    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:32.044345 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042678    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:32.044345 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042682    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:32.044345 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042685    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:32.044345 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042688    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:32.044345 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042691    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:32.044345 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042694    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:32.044345 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042697    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:32.044345 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042699    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:32.044838 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042702    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:32.044838 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042705    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:32.044838 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042708    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:32.044838 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.042711    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:32.044838 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043393    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:32.044838 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043399    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:32.044838 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043403    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:32.044838 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043406    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:32.044838 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043409    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:32.044838 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043412    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:32.044838 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043417    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:32.044838 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043420    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:32.044838 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043423    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:32.044838 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043426    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:32.044838 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043429    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:32.044838 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043432    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:32.044838 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043435    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:32.044838 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043438    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:32.044838 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043441    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:32.045344 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043443    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:32.045344 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043447    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:32.045344 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043449    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:32.045344 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043453    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:32.045344 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043456    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:32.045344 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043458    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:32.045344 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043461    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:32.045344 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043464    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:32.045344 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043466    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:32.045344 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043469    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:32.045344 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043471    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:32.045344 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043474    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:32.045344 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043477    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:32.045344 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043480    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:32.045344 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043482    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:32.045344 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043485    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:32.045344 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043487    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:32.045344 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043489    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:32.045344 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043492    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:32.045344 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043494    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:32.045829 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043497    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:32.045829 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043500    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:32.045829 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043502    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:32.045829 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043505    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:32.045829 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043507    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:32.045829 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043510    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:32.045829 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043512    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:32.045829 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043515    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:32.045829 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043517    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:32.045829 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043520    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:32.045829 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043522    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:32.045829 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043525    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:32.045829 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043527    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:32.045829 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043531    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:32.045829 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043534    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:32.045829 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043536    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:32.045829 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043540    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:32.045829 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043543    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:32.045829 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043545    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:32.045829 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043548    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:32.046349 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043550    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:32.046349 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043553    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:32.046349 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043556    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:32.046349 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043558    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:32.046349 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043561    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:32.046349 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043563    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:32.046349 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043566    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:32.046349 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043569    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:32.046349 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043571    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:32.046349 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043574    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:32.046349 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043576    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:32.046349 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043579    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:32.046349 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043581    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:32.046349 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043584    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:32.046349 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043587    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:32.046349 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043589    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:32.046349 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043594    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:32.046349 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043598    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:32.046349 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043601    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:32.046349 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043603    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:32.046843 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043606    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:32.046843 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043609    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:32.046843 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043611    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:32.046843 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043614    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:32.046843 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043619    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:32.046843 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043622    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:32.046843 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043626    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:32.046843 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043629    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:32.046843 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043631    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:32.046843 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043634    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:32.046843 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043637    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:32.046843 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043639    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:32.046843 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043642    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:32.046843 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043644    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:32.046843 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043647    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:32.046843 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043649    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:32.046843 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043652    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:32.046843 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043655    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:32.046843 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043657    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:32.047321 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043659    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:32.047321 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043662    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:32.047321 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043667    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:32.047321 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043669    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:32.047321 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.043672    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:32.047321 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044006    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:32.047321 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044010    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:32.047321 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044015    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:32.047321 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044019    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:32.047321 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044023    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:32.047321 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044026    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:32.047321 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044028    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:32.047321 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044031    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:32.047321 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044033    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:32.047321 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044052    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:32.047321 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044055    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:32.047321 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044058    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:32.047321 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044061    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:32.047321 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044063    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:32.047779 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044066    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:32.047779 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044069    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:32.047779 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044071    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:32.047779 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044074    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:32.047779 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044077    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:32.047779 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044080    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:32.047779 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044082    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:32.047779 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044085    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:32.047779 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044088    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:32.047779 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044092    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:32.047779 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044094    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:32.047779 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044097    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:32.047779 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044099    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:32.047779 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044103    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:32.047779 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044107    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:32.047779 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044112    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:32.047779 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044115    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:32.047779 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044118    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:32.047779 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044121    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:32.048258 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044123    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:32.048258 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044126    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:32.048258 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044129    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:32.048258 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044132    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:32.048258 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044134    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:32.048258 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044137    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:32.048258 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044140    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:32.048258 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044142    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:32.048258 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044145    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:32.048258 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044153    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:32.048258 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044157    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:32.048258 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044161    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:32.048258 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044164    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:32.048258 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044166    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:32.048258 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044169    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:32.048258 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044171    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:32.048258 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044174    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:32.048258 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044185    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:32.048258 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044188    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:32.048258 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044191    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:32.048743 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044193    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:32.048743 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044196    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:32.048743 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044198    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:32.048743 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044201    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:32.048743 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044204    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:32.048743 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044206    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:32.048743 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044209    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:32.048743 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044211    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:32.048743 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044216    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:32.048743 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044218    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:32.048743 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044220    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:32.048743 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044223    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:32.048743 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044226    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:32.048743 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044228    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:32.048743 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044232    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:32.048743 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044235    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:32.048743 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044237    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:32.048743 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044240    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:32.048743 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044243    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:32.048743 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044246    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:32.049243 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044249    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:32.049243 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044251    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:32.049243 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044254    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:32.049243 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044257    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:32.049243 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044259    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:32.049243 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044262    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:32.049243 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044264    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:32.049243 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044267    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:32.049243 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044269    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:32.049243 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044272    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:32.049243 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044275    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:32.049243 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044277    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:32.049243 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044280    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:32.049243 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044282    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:32.049243 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044285    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:32.049243 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044287    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:32.049243 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044290    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:32.049243 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044292    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:32.049243 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044295    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:32.049243 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044297    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:32.049718 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044301    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:32.049718 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044304    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:32.049718 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044306    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:32.049718 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044309    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:32.049718 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044311    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:32.049718 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.044313    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:32.049718 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044385    2578 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:40:32.049718 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044416    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:40:32.049718 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044422    2578 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:40:32.049718 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044426    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:40:32.049718 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044430    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:40:32.049718 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044433    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:40:32.049718 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044437    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:40:32.049718 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044441    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:40:32.049718 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044444    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:40:32.049718 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044447    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:40:32.049718 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044450    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:40:32.049718 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044453    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:40:32.049718 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044457    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:40:32.049718 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044460    2578 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:40:32.049718 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044463    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:40:32.049718 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044466    2578 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:40:32.049718 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044468    2578 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:40:32.050280 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044472    2578 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:40:32.050280 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044476    2578 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:40:32.050280 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044479    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:40:32.050280 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044482    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:40:32.050280 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044485    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:40:32.050280 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044488    2578 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:40:32.050280 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044492    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:40:32.050280 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044495    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:40:32.050280 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044498    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:40:32.050280 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044502    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:40:32.050280 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044505    2578 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:40:32.050280 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044508    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:40:32.050280 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044511    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:40:32.050280 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044514    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:40:32.050280 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044517    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:40:32.050280 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044521    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:40:32.050280 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044524    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:40:32.050280 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044526    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:40:32.050280 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044529    2578 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:40:32.050280 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044532    2578 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:40:32.050280 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044535    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:40:32.050280 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044539    2578 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:40:32.050280 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044542    2578 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:40:32.050280 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044545    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:40:32.050904 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044547    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:40:32.050904 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044551    2578 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:40:32.050904 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044554    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:40:32.050904 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044557    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:40:32.050904 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044559    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:40:32.050904 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044562    2578 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:40:32.050904 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044565    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:40:32.050904 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044568    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:40:32.050904 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044571    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:40:32.050904 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044574    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:40:32.050904 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044577    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:40:32.050904 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044580    2578 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:40:32.050904 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044582    2578 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:40:32.050904 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044586    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:40:32.050904 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044589    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:40:32.050904 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044592    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:40:32.050904 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044595    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:40:32.050904 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044598    2578 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:40:32.050904 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044601    2578 flags.go:64] FLAG: --help="false"
Sep 16 18:40:32.050904 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044604    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-142-80.ec2.internal"
Sep 16 18:40:32.050904 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044607    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:40:32.050904 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044610    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:40:32.050904 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044613    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:40:32.051494 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044616    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:40:32.051494 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044620    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:40:32.051494 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044623    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:40:32.051494 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044625    2578 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:40:32.051494 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044628    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:40:32.051494 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044631    2578 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:40:32.051494 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044634    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:40:32.051494 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044637    2578 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:40:32.051494 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044640    2578 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:40:32.051494 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044643    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:40:32.051494 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044646    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:40:32.051494 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044649    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:40:32.051494 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044651    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:40:32.051494 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044654    2578 flags.go:64] FLAG: --lock-file=""
Sep 16 18:40:32.051494 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044657    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:40:32.051494 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044660    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:40:32.051494 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044663    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:40:32.051494 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044669    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:40:32.051494 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044672    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:40:32.051494 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044678    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:40:32.051494 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044681    2578 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:40:32.051494 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044684    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:40:32.051494 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044687    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:40:32.051494 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044690    2578 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:40:32.052088 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044693    2578 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:40:32.052088 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044697    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:40:32.052088 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044701    2578 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:40:32.052088 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044704    2578 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:40:32.052088 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044707    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:40:32.052088 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044710    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:40:32.052088 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044713    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:40:32.052088 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044716    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:40:32.052088 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044719    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:40:32.052088 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044722    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:40:32.052088 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044725    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:40:32.052088 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044732    2578 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:40:32.052088 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044735    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:40:32.052088 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044739    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:40:32.052088 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044742    2578 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:40:32.052088 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044745    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:40:32.052088 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044750    2578 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:40:32.052088 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044752    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:40:32.052088 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044755    2578 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:40:32.052088 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044758    2578 flags.go:64] FLAG: --port="10250"
Sep 16 18:40:32.052088 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044761    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:40:32.052088 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044764    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0263710e31c437427"
Sep 16 18:40:32.052088 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044767    2578 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:40:32.052088 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044770    2578 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:40:32.052686 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044773    2578 flags.go:64] FLAG: --register-node="true"
Sep 16 18:40:32.052686 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044776    2578 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:40:32.052686 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044780    2578 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:40:32.052686 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044783    2578 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:40:32.052686 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044786    2578 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:40:32.052686 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044789    2578 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:40:32.052686 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044792    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:40:32.052686 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044795    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:40:32.052686 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044798    2578 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:40:32.052686 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044801    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:40:32.052686 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044804    2578 flags.go:64] FLAG: --runonce="false"
Sep 16 18:40:32.052686 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044807    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:40:32.052686 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044810    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:40:32.052686 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044813    2578 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:40:32.052686 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044816    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:40:32.052686 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044819    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:40:32.052686 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044822    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:40:32.052686 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044825    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:40:32.052686 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044827    2578 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:40:32.052686 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044830    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:40:32.052686 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044833    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:40:32.052686 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044836    2578 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:40:32.052686 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044839    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:40:32.052686 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044842    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:40:32.052686 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044845    2578 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:40:32.053297 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044865    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:40:32.053297 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044874    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:40:32.053297 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044877    2578 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:40:32.053297 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044880    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:40:32.053297 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044884    2578 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:40:32.053297 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044887    2578 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:40:32.053297 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044890    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:40:32.053297 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044893    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:40:32.053297 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044896    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:40:32.053297 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044899    2578 flags.go:64] FLAG: --v="2"
Sep 16 18:40:32.053297 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044902    2578 flags.go:64] FLAG: --version="false"
Sep 16 18:40:32.053297 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044906    2578 flags.go:64] FLAG: --vmodule=""
Sep 16 18:40:32.053297 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044910    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:40:32.053297 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.044913    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:40:32.053297 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045018    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:32.053297 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045023    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:32.053297 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045027    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:32.053297 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045033    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:32.053297 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045049    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:32.053297 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045052    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:32.053297 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045057    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:32.053297 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045059    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:32.053872 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045062    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:32.053872 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045065    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:32.053872 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045068    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:32.053872 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045070    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:32.053872 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045073    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:32.053872 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045075    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:32.053872 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045078    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:32.053872 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045081    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:32.053872 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045083    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:32.053872 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045086    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:32.053872 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045088    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:32.053872 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045091    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:32.053872 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045094    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:32.053872 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045096    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:32.053872 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045099    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:32.053872 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045101    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:32.053872 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045104    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:32.053872 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045106    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:32.053872 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045109    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:32.053872 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045111    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:32.054540 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045114    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:32.054540 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045116    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:32.054540 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045119    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:32.054540 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045122    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:32.054540 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045125    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:32.054540 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045134    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:32.054540 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045137    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:32.054540 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045141    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:32.054540 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045144    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:32.054540 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045147    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:32.054540 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045151    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:32.054540 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045154    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:32.054540 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045157    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:32.054540 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045159    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:32.054540 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045162    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:32.054540 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045164    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:32.054540 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045168    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:32.054540 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045172    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:32.054540 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045175    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:32.054540 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045179    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:32.055265 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045182    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:32.055265 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045186    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:32.055265 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045189    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:32.055265 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045192    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:32.055265 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045195    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:32.055265 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045197    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:32.055265 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045200    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:32.055265 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045203    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:32.055265 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045206    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:32.055265 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045208    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:32.055265 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045211    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:32.055265 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045214    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:32.055265 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045216    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:32.055265 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045219    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:32.055265 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045221    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:32.055265 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045224    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:32.055265 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045227    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:32.055265 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045229    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:32.055265 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045232    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:32.056093 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045235    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:32.056093 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045238    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:32.056093 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045241    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:32.056093 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045245    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:32.056093 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045248    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:32.056093 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045250    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:32.056093 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045253    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:32.056093 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045256    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:32.056093 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045258    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:32.056093 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045261    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:32.056093 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045264    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:32.056093 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045266    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:32.056093 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045269    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:32.056093 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045271    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:32.056093 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045274    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:32.056093 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045277    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:32.056093 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045279    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:32.056093 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045282    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:32.056093 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045284    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:32.056670 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045287    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:32.056670 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045289    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:32.056670 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045292    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:32.056670 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045294    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:32.056670 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045297    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:32.056670 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045299    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:32.056670 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045302    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:32.056670 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045304    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:32.056670 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045307    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:32.056670 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045310    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:32.056670 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045312    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:32.056670 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045315    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:32.056670 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.045318    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:32.056670 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.045327    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:32.056670 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.052191    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:40:32.057090 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.052206    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:40:32.057090 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.053031    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:40:32.057090 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.053054    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:40:32.057090 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053138    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:32.057090 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053143    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:32.057090 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053147    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:32.057090 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053150    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:32.057090 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053153    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:32.057090 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053155    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:32.057090 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053159    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:32.057090 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053163    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:32.057090 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053166    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:32.057090 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053170    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:32.057090 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053173    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:32.057090 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053176    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:32.057090 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053179    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:32.057090 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053181    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:32.057090 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053184    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:32.057090 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053187    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:32.057556 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053189    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:32.057556 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053192    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:32.057556 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053195    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:32.057556 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053198    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:32.057556 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053200    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:32.057556 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053203    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:32.057556 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053206    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:32.057556 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053209    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:32.057556 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053211    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:32.057556 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053221    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:32.057556 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053224    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:32.057556 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053227    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:32.057556 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053229    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:32.057556 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053233    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:32.057556 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053236    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:32.057556 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053238    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:32.057556 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053241    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:32.057556 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053244    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:32.057556 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053247    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:32.057556 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053249    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:32.058063 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053252    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:32.058063 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053255    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:32.058063 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053258    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:32.058063 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053261    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:32.058063 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053263    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:32.058063 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053266    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:32.058063 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053269    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:32.058063 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053271    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:32.058063 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053274    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:32.058063 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053276    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:32.058063 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053279    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:32.058063 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053281    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:32.058063 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053284    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:32.058063 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053286    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:32.058063 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053289    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:32.058063 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053291    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:32.058063 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053294    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:32.058063 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053296    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:32.058063 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053299    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:32.058063 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053302    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:32.058584 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053304    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:32.058584 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053307    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:32.058584 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053316    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:32.058584 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053319    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:32.058584 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053322    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:32.058584 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053325    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:32.058584 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053328    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:32.058584 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053330    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:32.058584 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053333    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:32.058584 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053336    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:32.058584 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053338    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:32.058584 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053341    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:32.058584 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053344    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:32.058584 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053346    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:32.058584 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053348    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:32.058584 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053357    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:32.058584 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053360    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:32.058584 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053362    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:32.058584 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053366    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:32.059061 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053369    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:32.059061 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053372    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:32.059061 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053376    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:32.059061 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053378    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:32.059061 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053381    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:32.059061 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053384    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:32.059061 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053386    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:32.059061 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053389    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:32.059061 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053392    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:32.059061 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053394    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:32.059061 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053397    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:32.059061 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053400    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:32.059061 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053403    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:32.059061 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053405    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:32.059061 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053409    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:32.059061 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053413    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:32.059061 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053422    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:32.059061 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053424    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:32.059061 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053428    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:32.059587 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053431    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:32.059587 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053433    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:32.059587 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053436    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:32.059587 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053439    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:32.059587 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053441    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:32.059587 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.053446    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:32.059587 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053608    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:32.059587 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053613    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:32.059587 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053616    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:32.059587 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053619    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:32.059587 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053622    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:32.059587 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053625    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:32.059587 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053628    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:32.059587 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053631    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:32.059587 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053634    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:32.059957 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053636    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:32.059957 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053640    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:32.059957 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053644    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:32.059957 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053647    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:32.059957 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053649    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:32.059957 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053652    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:32.059957 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053654    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:32.059957 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053657    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:32.059957 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053660    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:32.059957 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053662    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:32.059957 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053665    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:32.059957 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053667    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:32.059957 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053670    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:32.059957 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053672    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:32.059957 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053675    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:32.059957 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053687    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:32.059957 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053691    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:32.059957 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053694    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:32.059957 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053696    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:32.060450 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053699    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:32.060450 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053702    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:32.060450 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053705    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:32.060450 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053707    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:32.060450 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053710    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:32.060450 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053712    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:32.060450 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053715    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:32.060450 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053718    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:32.060450 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053720    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:32.060450 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053724    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:32.060450 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053727    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:32.060450 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053730    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:32.060450 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053733    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:32.060450 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053736    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:32.060450 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053739    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:32.060450 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053742    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:32.060450 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053745    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:32.060450 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053747    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:32.060450 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053750    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:32.060450 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053753    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:32.060938 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053755    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:32.060938 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053758    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:32.060938 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053760    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:32.060938 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053763    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:32.060938 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053765    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:32.060938 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053768    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:32.060938 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053770    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:32.060938 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053773    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:32.060938 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053775    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:32.060938 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053784    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:32.060938 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053788    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:32.060938 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053790    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:32.060938 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053793    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:32.060938 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053796    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:32.060938 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053798    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:32.060938 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053801    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:32.060938 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053803    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:32.060938 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053806    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:32.060938 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053808    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:32.060938 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053811    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:32.061437 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053813    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:32.061437 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053816    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:32.061437 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053818    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:32.061437 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053820    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:32.061437 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053823    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:32.061437 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053825    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:32.061437 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053828    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:32.061437 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053830    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:32.061437 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053833    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:32.061437 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053835    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:32.061437 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053838    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:32.061437 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053840    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:32.061437 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053843    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:32.061437 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053845    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:32.061437 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053848    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:32.061437 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053850    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:32.061437 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053853    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:32.061437 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053855    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:32.061437 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053859    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:32.061437 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053862    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:32.061916 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053865    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:32.061916 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053868    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:32.061916 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053877    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:32.061916 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053880    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:32.061916 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053883    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:32.061916 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053886    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:32.061916 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053889    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:32.061916 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053892    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:32.061916 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053894    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:32.061916 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053897    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:32.061916 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:32.053899    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:32.061916 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.053904    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:32.061916 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.054017    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:40:32.061916 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.057659    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:40:32.061916 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.058469    2578 server.go:1019] "Starting client certificate rotation"
Sep 16 18:40:32.062309 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.058566    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:40:32.062309 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.058606    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:40:32.062309 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.060605    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:40:32.063190 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.063177    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:40:32.074342 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.074327    2578 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:40:32.076856 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.076842    2578 log.go:25] "Validated CRI v1 image API"
Sep 16 18:40:32.077686 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.077666    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:40:32.080610 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.080583    2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 917d95d3-dbf5-47f5-a26e-107e7577bbc2:/dev/nvme0n1p4 e7d27bc5-9f58-46c6-8682-a9e3de5b752f:/dev/nvme0n1p3]
Sep 16 18:40:32.080658 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.080610    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:40:32.087506 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.087412    2578 manager.go:216] Machine: {Timestamp:2026-09-16 18:40:32.086187326 +0000 UTC m=+0.321120762 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105533 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25e58a002f9fb216a03d0a9d798ed0 SystemUUID:ec25e58a-002f-9fb2-16a0-3d0a9d798ed0 BootID:e660d51d-a507-4540-b41c-1b583970594e Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:dd:e4:40:13 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:dd:e4:40:13 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:36:e4:2a:ec:34 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:40:32.087506 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.087503    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:40:32.087610 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.087570    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:40:32.089335 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.089312    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:40:32.089489 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.089339    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-80.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:40:32.090187 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.090177    2578 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:40:32.090219 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.090190    2578 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:40:32.090219 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.090198    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:40:32.090697 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.090687    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:40:32.091366 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.091352    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:40:32.091406 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.091387    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:40:32.095880 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.095868    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:40:32.095981 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.095972    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:40:32.096020 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.096000    2578 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:40:32.096020 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.096009    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:40:32.096020 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.096019    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:40:32.096158 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.096027    2578 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:40:32.096158 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.096050    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:40:32.096446 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.096430    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:40:32.096505 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.096455    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:40:32.100333 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.100308    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:40:32.102624 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.102607    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:40:32.102709 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.102639    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:40:32.103953 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.103940    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:40:32.103992 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.103964    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:40:32.103992 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.103977    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:40:32.103992 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.103984    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:40:32.104089 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.103995    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:40:32.104089 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.104004    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:40:32.104089 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.104016    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:40:32.104089 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.104021    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:40:32.104089 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.104028    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:40:32.104089 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.104033    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:40:32.104089 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.104058    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:40:32.104089 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.104063    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:40:32.104698 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.104689    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:40:32.104728 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.104700    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:40:32.108609 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.108466    2578 server.go:1268] "Started kubelet"
Sep 16 18:40:32.108690 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.108580    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:40:32.108690 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:32.108621    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:40:32.108690 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.108684    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-80.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:40:32.108816 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:32.108689    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-80.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:40:32.108816 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.108717    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:40:32.108816 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.108778    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:40:32.109438 ip-10-0-142-80 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:40:32.110172 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.110140    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:40:32.110385 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.110374    2578 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:40:32.113062 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:32.113032    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:40:32.115117 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.115102    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:40:32.115117 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.115113    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:40:32.115239 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:32.115170    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-80.ec2.internal\" not found"
Sep 16 18:40:32.115937 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.115921    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:40:32.115937 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.115922    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:40:32.116079 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.115945    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:40:32.116130 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.116092    2578 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:40:32.116130 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.116100    2578 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:40:32.117585 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.117471    2578 factory.go:55] Registering systemd factory
Sep 16 18:40:32.117585 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.117484    2578 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:40:32.117813 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.117796    2578 factory.go:153] Registering CRI-O factory
Sep 16 18:40:32.117813 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.117814    2578 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:40:32.117938 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.117861    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:40:32.117938 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.117880    2578 factory.go:103] Registering Raw factory
Sep 16 18:40:32.117938 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.117892    2578 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:40:32.118283 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.118268    2578 manager.go:349] Starting recovery of all containers
Sep 16 18:40:32.123478 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:32.123453    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-80.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:40:32.123616 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:32.123594    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:40:32.124028 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.124004    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6vqtr"
Sep 16 18:40:32.124214 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:32.123329    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-80.ec2.internal.18d5e1321188f5e8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-80.ec2.internal,UID:ip-10-0-142-80.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-80.ec2.internal,},FirstTimestamp:2026-09-16 18:40:32.108443112 +0000 UTC m=+0.343376550,LastTimestamp:2026-09-16 18:40:32.108443112 +0000 UTC m=+0.343376550,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-80.ec2.internal,}"
Sep 16 18:40:32.127565 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.127541    2578 manager.go:354] Recovery completed
Sep 16 18:40:32.133003 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.132990    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:32.134495 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.134478    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6vqtr"
Sep 16 18:40:32.134883 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.134865    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:32.134943 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.134900    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:32.134943 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.134912    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:32.135428 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.135415    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:40:32.135428 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.135426    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:40:32.135520 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.135440    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:40:32.135520 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.135460    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:32.137298 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:32.137235    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-80.ec2.internal.18d5e132131c65c8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-80.ec2.internal,UID:ip-10-0-142-80.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-80.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-80.ec2.internal,},FirstTimestamp:2026-09-16 18:40:32.13488276 +0000 UTC m=+0.369816194,LastTimestamp:2026-09-16 18:40:32.13488276 +0000 UTC m=+0.369816194,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-80.ec2.internal,}"
Sep 16 18:40:32.138933 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.138917    2578 policy_none.go:49] "None policy: Start"
Sep 16 18:40:32.139033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.138937    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:40:32.139033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.138950    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:40:32.139033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.138981    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:32.139033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.139005    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:32.139033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.139018    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:32.140896 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.140882    2578 policy_none.go:47] "Start"
Sep 16 18:40:32.166586 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.166571    2578 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:40:32.193132 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:32.166603    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:40:32.193132 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.166612    2578 server.go:87] "Starting device plugin registration server"
Sep 16 18:40:32.193132 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.166894    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:40:32.193132 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.166906    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:40:32.193132 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.167000    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:40:32.193132 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.167105    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:40:32.193132 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.167115    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:40:32.193132 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:32.167485    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:40:32.193132 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:32.167518    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-80.ec2.internal\" not found"
Sep 16 18:40:32.209818 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.209799    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:40:32.210913 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.210889    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:40:32.210995 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.210935    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:40:32.210995 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.210954    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:40:32.210995 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:32.210991    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:40:32.213006 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.212983    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:32.267688 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.267670    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:32.268397 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.268383    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:32.268471 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.268408    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:32.268471 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.268422    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:32.268471 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.268440    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-80.ec2.internal"
Sep 16 18:40:32.277312 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.277299    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-80.ec2.internal"
Sep 16 18:40:32.277354 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:32.277318    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-80.ec2.internal\": node \"ip-10-0-142-80.ec2.internal\" not found"
Sep 16 18:40:32.293469 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:32.293418    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-80.ec2.internal\" not found"
Sep 16 18:40:32.311881 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.311853    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-80.ec2.internal"]
Sep 16 18:40:32.311942 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.311926    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:32.312652 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.312637    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:32.312726 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.312665    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:32.312726 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.312678    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:32.314882 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.314871    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:32.315012 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.314988    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal"
Sep 16 18:40:32.315071 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.315026    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:32.315631 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.315615    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:32.315695 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.315639    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:32.315695 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.315649    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:32.315754 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.315621    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:32.315754 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.315720    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:32.315754 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.315731    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:32.317267 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.317243    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ff3af002549164d48d170f209bd2fe9c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal\" (UID: \"ff3af002549164d48d170f209bd2fe9c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal"
Sep 16 18:40:32.317359 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.317273    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff3af002549164d48d170f209bd2fe9c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal\" (UID: \"ff3af002549164d48d170f209bd2fe9c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal"
Sep 16 18:40:32.317359 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.317287    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/61224f73c725c86faa0f55f841c5d69e-config\") pod \"kube-apiserver-proxy-ip-10-0-142-80.ec2.internal\" (UID: \"61224f73c725c86faa0f55f841c5d69e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-80.ec2.internal"
Sep 16 18:40:32.317872 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.317860    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-80.ec2.internal"
Sep 16 18:40:32.317910 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.317885    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:32.319144 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.319128    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:32.319233 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.319151    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:32.319233 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.319162    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:32.341590 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:32.341572    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-80.ec2.internal\" not found" node="ip-10-0-142-80.ec2.internal"
Sep 16 18:40:32.345847 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:32.345832    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-80.ec2.internal\" not found" node="ip-10-0-142-80.ec2.internal"
Sep 16 18:40:32.394446 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:32.394426    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-80.ec2.internal\" not found"
Sep 16 18:40:32.417771 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.417752    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ff3af002549164d48d170f209bd2fe9c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal\" (UID: \"ff3af002549164d48d170f209bd2fe9c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal"
Sep 16 18:40:32.417860 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.417782    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff3af002549164d48d170f209bd2fe9c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal\" (UID: \"ff3af002549164d48d170f209bd2fe9c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal"
Sep 16 18:40:32.417860 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.417810    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/61224f73c725c86faa0f55f841c5d69e-config\") pod \"kube-apiserver-proxy-ip-10-0-142-80.ec2.internal\" (UID: \"61224f73c725c86faa0f55f841c5d69e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-80.ec2.internal"
Sep 16 18:40:32.417860 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.417841    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ff3af002549164d48d170f209bd2fe9c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal\" (UID: \"ff3af002549164d48d170f209bd2fe9c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal"
Sep 16 18:40:32.417860 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.417850    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff3af002549164d48d170f209bd2fe9c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal\" (UID: \"ff3af002549164d48d170f209bd2fe9c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal"
Sep 16 18:40:32.417998 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.417846    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/61224f73c725c86faa0f55f841c5d69e-config\") pod \"kube-apiserver-proxy-ip-10-0-142-80.ec2.internal\" (UID: \"61224f73c725c86faa0f55f841c5d69e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-80.ec2.internal"
Sep 16 18:40:32.494530 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:32.494512    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-80.ec2.internal\" not found"
Sep 16 18:40:32.595187 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:32.595146    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-80.ec2.internal\" not found"
Sep 16 18:40:32.643812 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.643786    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal"
Sep 16 18:40:32.648496 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:32.648481    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-80.ec2.internal"
Sep 16 18:40:32.695853 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:32.695833    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-80.ec2.internal\" not found"
Sep 16 18:40:32.796318 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:32.796289    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-80.ec2.internal\" not found"
Sep 16 18:40:32.896874 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:32.896824    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-80.ec2.internal\" not found"
Sep 16 18:40:32.997861 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:32.997836    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-80.ec2.internal\" not found"
Sep 16 18:40:33.058982 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:33.058951    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:40:33.059420 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:33.059151    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:40:33.098452 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:33.098431    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-80.ec2.internal\" not found"
Sep 16 18:40:33.140418 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:33.140386    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:35:32 +0000 UTC" deadline="2028-03-26 19:55:45.933411049 +0000 UTC"
Sep 16 18:40:33.140418 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:33.140415    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13369h15m12.792999491s"
Sep 16 18:40:33.199457 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:33.199407    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-80.ec2.internal\" not found"
Sep 16 18:40:33.214251 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:33.214031    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61224f73c725c86faa0f55f841c5d69e.slice/crio-cac1a22ca37cff34bf32f447e0eaa95cf875872f5e5d2de62286c4c4f56622da WatchSource:0}: Error finding container cac1a22ca37cff34bf32f447e0eaa95cf875872f5e5d2de62286c4c4f56622da: Status 404 returned error can't find the container with id cac1a22ca37cff34bf32f447e0eaa95cf875872f5e5d2de62286c4c4f56622da
Sep 16 18:40:33.215906 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:33.215884    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff3af002549164d48d170f209bd2fe9c.slice/crio-d179fc764850762b8735aeb7e4d65abe717041f7423761b1b29559c09b4f0b7f WatchSource:0}: Error finding container d179fc764850762b8735aeb7e4d65abe717041f7423761b1b29559c09b4f0b7f: Status 404 returned error can't find the container with id d179fc764850762b8735aeb7e4d65abe717041f7423761b1b29559c09b4f0b7f
Sep 16 18:40:33.219891 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:33.219877    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:40:33.299653 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:33.299626    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-80.ec2.internal\" not found"
Sep 16 18:40:33.303111 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:33.303077    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:33.400856 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:33.400830    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-80.ec2.internal\" not found"
Sep 16 18:40:33.444872 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:33.444857    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:33.500938 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:33.500891    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-80.ec2.internal\" not found"
Sep 16 18:40:33.601052 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:33.601016    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-80.ec2.internal\" not found"
Sep 16 18:40:33.603303 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:33.603128    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:33.615568 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:33.615548    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal"
Sep 16 18:40:33.630745 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:33.630725    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:40:33.631822 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:33.631805    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-80.ec2.internal"
Sep 16 18:40:33.641389 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:33.641368    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:40:34.096529 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.096498    2578 apiserver.go:52] "Watching apiserver"
Sep 16 18:40:34.105837 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.105808    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:40:34.106829 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.106804    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-djht2","openshift-network-operator/iptables-alerter-wfvjg","openshift-ovn-kubernetes/ovnkube-node-lp55b","kube-system/global-pull-secret-syncer-qsdt4","kube-system/kube-apiserver-proxy-ip-10-0-142-80.ec2.internal","openshift-cluster-node-tuning-operator/tuned-k7nnn","openshift-image-registry/node-ca-95lgg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal","openshift-multus/multus-4fj62","openshift-multus/multus-additional-cni-plugins-tjsbg","openshift-multus/network-metrics-daemon-xq67c","kube-system/konnectivity-agent-gw4zp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkjtg","openshift-dns/node-resolver-xbtsz"]
Sep 16 18:40:34.109425 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.109399    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-95lgg"
Sep 16 18:40:34.111773 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.111750    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xq67c"
Sep 16 18:40:34.111893 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:34.111847    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xq67c" podUID="733b59e1-b843-4255-bf37-71c81c244989"
Sep 16 18:40:34.112751 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.112720    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-87cgd\""
Sep 16 18:40:34.112864 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.112789    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:40:34.112864 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.112799    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:40:34.112981 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.112792    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:40:34.114201 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.114164    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.116576 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.116397    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:40:34.116948 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.116805    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:40:34.116948 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:34.116877    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsdt4" podUID="6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872"
Sep 16 18:40:34.117129 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.116990    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:40:34.118450 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.118428    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:40:34.118560 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.118461    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:40:34.118560 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.118488    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:40:34.118560 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.118436    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:40:34.118736 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.118715    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:40:34.118794 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.118751    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s6pc4\""
Sep 16 18:40:34.119184 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.119164    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xbtsz"
Sep 16 18:40:34.121506 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.121485    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.121654 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.121635    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:40:34.121863 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.121843    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:40:34.122144 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.122110    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wh7wb\""
Sep 16 18:40:34.123940 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.123925    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jr6mz\""
Sep 16 18:40:34.124227 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.124150    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.124227 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.124188    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:40:34.124558 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.124537    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:40:34.126985 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.126672    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:40:34.126985 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.126726    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkjtg"
Sep 16 18:40:34.127246 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.127223    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:40:34.127246 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.127234    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:40:34.127408 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.127251    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:40:34.127408 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.127347    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-crk2x\""
Sep 16 18:40:34.128270 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.128252    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ab1b46d-5a59-42a7-87c4-13c16da817ad-host\") pod \"node-ca-95lgg\" (UID: \"6ab1b46d-5a59-42a7-87c4-13c16da817ad\") " pod="openshift-image-registry/node-ca-95lgg"
Sep 16 18:40:34.128335 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.128277    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hqcl\" (UniqueName: \"kubernetes.io/projected/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-kube-api-access-5hqcl\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.128335 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.128305    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872-kubelet-config\") pod \"global-pull-secret-syncer-qsdt4\" (UID: \"6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872\") " pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:40:34.128335 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.128330    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-host-slash\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.128505 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.128372    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-var-lib-openvswitch\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.128505 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.128397    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-node-log\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.128505 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.128425    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/733b59e1-b843-4255-bf37-71c81c244989-metrics-certs\") pod \"network-metrics-daemon-xq67c\" (UID: \"733b59e1-b843-4255-bf37-71c81c244989\") " pod="openshift-multus/network-metrics-daemon-xq67c"
Sep 16 18:40:34.128505 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.128441    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-host-cni-bin\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.128505 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.128455    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-host-cni-netd\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.128505 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.128490    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872-dbus\") pod \"global-pull-secret-syncer-qsdt4\" (UID: \"6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872\") " pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:40:34.128863 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.128519    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6ab1b46d-5a59-42a7-87c4-13c16da817ad-serviceca\") pod \"node-ca-95lgg\" (UID: \"6ab1b46d-5a59-42a7-87c4-13c16da817ad\") " pod="openshift-image-registry/node-ca-95lgg"
Sep 16 18:40:34.128863 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.128547    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-run-ovn\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.128863 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.128569    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-host-run-ovn-kubernetes\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.128863 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.128598    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.128863 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.128630    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-ovnkube-config\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.128863 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.128657    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-env-overrides\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.128863 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.128675    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872-original-pull-secret\") pod \"global-pull-secret-syncer-qsdt4\" (UID: \"6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872\") " pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:40:34.128863 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.128697    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-run-systemd\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.128863 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.128729    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-log-socket\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.128863 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.128753    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-ovn-node-metrics-cert\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.128863 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.128795    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-ovnkube-script-lib\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.128863 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.128815    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7qx4\" (UniqueName: \"kubernetes.io/projected/6ab1b46d-5a59-42a7-87c4-13c16da817ad-kube-api-access-w7qx4\") pod \"node-ca-95lgg\" (UID: \"6ab1b46d-5a59-42a7-87c4-13c16da817ad\") " pod="openshift-image-registry/node-ca-95lgg"
Sep 16 18:40:34.128863 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.128840    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnn78\" (UniqueName: \"kubernetes.io/projected/733b59e1-b843-4255-bf37-71c81c244989-kube-api-access-hnn78\") pod \"network-metrics-daemon-xq67c\" (UID: \"733b59e1-b843-4255-bf37-71c81c244989\") " pod="openshift-multus/network-metrics-daemon-xq67c"
Sep 16 18:40:34.128863 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.128865    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-etc-openvswitch\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.129598 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.128891    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-run-openvswitch\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.129598 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.128915    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-host-kubelet\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.129598 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.128949    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-systemd-units\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.129598 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.128989    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-host-run-netns\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.129598 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.129289    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wfvjg"
Sep 16 18:40:34.129598 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.129327    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:40:34.130064 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.130027    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:40:34.130064 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.130048    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p2qjd\""
Sep 16 18:40:34.130267 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.130175    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:40:34.131812 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.131795    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tjsbg"
Sep 16 18:40:34.134183 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.134163    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:40:34.134416 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.134395    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:40:34.134511 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.134399    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djht2"
Sep 16 18:40:34.134511 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.134491    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:40:34.134511 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:34.134499    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djht2" podUID="feaf8529-d8fc-4f8d-8c78-e19f994b6f26"
Sep 16 18:40:34.134739 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.134699    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-92g55\""
Sep 16 18:40:34.138307 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.136211    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:40:34.139048 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.139023    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:40:34.139238 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.139217    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-c7mv8\""
Sep 16 18:40:34.139494 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.139472    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gw4zp"
Sep 16 18:40:34.139919 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.139896    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:40:34.142218 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.142184    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mgr49\""
Sep 16 18:40:34.142333 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.142255    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:40:34.142333 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.142306    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:40:34.160623 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.160605    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-h4lsr"
Sep 16 18:40:34.174452 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.174426    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-h4lsr"
Sep 16 18:40:34.216333 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.216251    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal" event={"ID":"ff3af002549164d48d170f209bd2fe9c","Type":"ContainerStarted","Data":"d179fc764850762b8735aeb7e4d65abe717041f7423761b1b29559c09b4f0b7f"}
Sep 16 18:40:34.217139 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.217105    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:40:34.217712 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.217688    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-80.ec2.internal" event={"ID":"61224f73c725c86faa0f55f841c5d69e","Type":"ContainerStarted","Data":"cac1a22ca37cff34bf32f447e0eaa95cf875872f5e5d2de62286c4c4f56622da"}
Sep 16 18:40:34.230142 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.230118    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3bf00b4-a637-4a14-87f6-a1bb2ae1a276-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tjsbg\" (UID: \"f3bf00b4-a637-4a14-87f6-a1bb2ae1a276\") " pod="openshift-multus/multus-additional-cni-plugins-tjsbg"
Sep 16 18:40:34.230236 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.230147    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fe4ed2ab-567f-4482-8c96-7293f1c82bb9-konnectivity-ca\") pod \"konnectivity-agent-gw4zp\" (UID: \"fe4ed2ab-567f-4482-8c96-7293f1c82bb9\") " pod="kube-system/konnectivity-agent-gw4zp"
Sep 16 18:40:34.230236 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.230170    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/733b59e1-b843-4255-bf37-71c81c244989-metrics-certs\") pod \"network-metrics-daemon-xq67c\" (UID: \"733b59e1-b843-4255-bf37-71c81c244989\") " pod="openshift-multus/network-metrics-daemon-xq67c"
Sep 16 18:40:34.230236 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.230195    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-host-cni-bin\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.230236 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.230218    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872-dbus\") pod \"global-pull-secret-syncer-qsdt4\" (UID: \"6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872\") " pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:40:34.230413 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:34.230310    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:34.230413 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:34.230387    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/733b59e1-b843-4255-bf37-71c81c244989-metrics-certs podName:733b59e1-b843-4255-bf37-71c81c244989 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:34.730367335 +0000 UTC m=+2.965300758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/733b59e1-b843-4255-bf37-71c81c244989-metrics-certs") pod "network-metrics-daemon-xq67c" (UID: "733b59e1-b843-4255-bf37-71c81c244989") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:34.230511 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.230415    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-host-cni-bin\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.230511 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.230422    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-host-run-ovn-kubernetes\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.230511 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.230454    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.230511 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.230463    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872-dbus\") pod \"global-pull-secret-syncer-qsdt4\" (UID: \"6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872\") " pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:40:34.230511 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.230484    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-ovnkube-config\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.230511 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.230486    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-host-run-ovn-kubernetes\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.230801 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.230520    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.230801 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.230556    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99603daa-09aa-4da6-a429-17a33c23586f-host-slash\") pod \"iptables-alerter-wfvjg\" (UID: \"99603daa-09aa-4da6-a429-17a33c23586f\") " pod="openshift-network-operator/iptables-alerter-wfvjg"
Sep 16 18:40:34.230801 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.230587    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-run-systemd\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.230801 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.230611    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-log-socket\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.230801 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.230636    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-etc-sysctl-d\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.230801 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.230658    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-run-systemd\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.230801 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.230662    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-etc-sysctl-conf\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.230801 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.230702    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-log-socket\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.230801 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.230716    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-sys\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.230801 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.230746    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f3bf00b4-a637-4a14-87f6-a1bb2ae1a276-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tjsbg\" (UID: \"f3bf00b4-a637-4a14-87f6-a1bb2ae1a276\") " pod="openshift-multus/multus-additional-cni-plugins-tjsbg"
Sep 16 18:40:34.230801 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.230778    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-etc-openvswitch\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.231343 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.230803    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-host\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.231343 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.230837    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-etc-openvswitch\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.231343 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.230842    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5d250f2-42fe-41a9-91a5-3433fc67a942-tmp\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.231343 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.230890    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-multus-socket-dir-parent\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.231343 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.230920    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-host-kubelet\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.231343 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.230944    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-systemd-units\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.231343 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.230977    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-host-run-netns\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.231343 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.230986    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-systemd-units\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.231343 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.230999    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-etc-kubernetes\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.231343 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231006    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-host-kubelet\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.231343 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231022    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-cni-binary-copy\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.231343 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231034    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-host-run-netns\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.231343 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231055    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-ovnkube-config\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.231343 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231051    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72qjj\" (UniqueName: \"kubernetes.io/projected/feaf8529-d8fc-4f8d-8c78-e19f994b6f26-kube-api-access-72qjj\") pod \"network-check-target-djht2\" (UID: \"feaf8529-d8fc-4f8d-8c78-e19f994b6f26\") " pod="openshift-network-diagnostics/network-check-target-djht2"
Sep 16 18:40:34.231343 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231093    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fe4ed2ab-567f-4482-8c96-7293f1c82bb9-agent-certs\") pod \"konnectivity-agent-gw4zp\" (UID: \"fe4ed2ab-567f-4482-8c96-7293f1c82bb9\") " pod="kube-system/konnectivity-agent-gw4zp"
Sep 16 18:40:34.231343 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231116    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ab1b46d-5a59-42a7-87c4-13c16da817ad-host\") pod \"node-ca-95lgg\" (UID: \"6ab1b46d-5a59-42a7-87c4-13c16da817ad\") " pod="openshift-image-registry/node-ca-95lgg"
Sep 16 18:40:34.231343 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hqcl\" (UniqueName: \"kubernetes.io/projected/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-kube-api-access-5hqcl\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.232130 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231164    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3e865df-22be-4129-bbf3-300285c29e69-tmp-dir\") pod \"node-resolver-xbtsz\" (UID: \"c3e865df-22be-4129-bbf3-300285c29e69\") " pod="openshift-dns/node-resolver-xbtsz"
Sep 16 18:40:34.232130 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231190    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ab1b46d-5a59-42a7-87c4-13c16da817ad-host\") pod \"node-ca-95lgg\" (UID: \"6ab1b46d-5a59-42a7-87c4-13c16da817ad\") " pod="openshift-image-registry/node-ca-95lgg"
Sep 16 18:40:34.232130 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231190    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a5d250f2-42fe-41a9-91a5-3433fc67a942-etc-tuned\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.232130 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231237    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3072b448-5de4-4f9f-b4db-2fb1ff05be1f-device-dir\") pod \"aws-ebs-csi-driver-node-gkjtg\" (UID: \"3072b448-5de4-4f9f-b4db-2fb1ff05be1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkjtg"
Sep 16 18:40:34.232130 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231270    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-host-slash\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.232130 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231292    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-etc-modprobe-d\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.232130 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231314    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-etc-sysconfig\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.232130 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231339    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3072b448-5de4-4f9f-b4db-2fb1ff05be1f-socket-dir\") pod \"aws-ebs-csi-driver-node-gkjtg\" (UID: \"3072b448-5de4-4f9f-b4db-2fb1ff05be1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkjtg"
Sep 16 18:40:34.232130 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231350    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-host-slash\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.232130 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231370    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3072b448-5de4-4f9f-b4db-2fb1ff05be1f-sys-fs\") pod \"aws-ebs-csi-driver-node-gkjtg\" (UID: \"3072b448-5de4-4f9f-b4db-2fb1ff05be1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkjtg"
Sep 16 18:40:34.232130 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231395    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9887\" (UniqueName: \"kubernetes.io/projected/3072b448-5de4-4f9f-b4db-2fb1ff05be1f-kube-api-access-l9887\") pod \"aws-ebs-csi-driver-node-gkjtg\" (UID: \"3072b448-5de4-4f9f-b4db-2fb1ff05be1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkjtg"
Sep 16 18:40:34.232130 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231419    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-host-run-netns\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.232130 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231448    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-host-run-multus-certs\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.232130 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231471    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-var-lib-openvswitch\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.232130 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231494    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3072b448-5de4-4f9f-b4db-2fb1ff05be1f-etc-selinux\") pod \"aws-ebs-csi-driver-node-gkjtg\" (UID: \"3072b448-5de4-4f9f-b4db-2fb1ff05be1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkjtg"
Sep 16 18:40:34.232130 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231519    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-host-run-k8s-cni-cncf-io\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.232130 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231548    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-host-var-lib-kubelet\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.232864 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231560    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-var-lib-openvswitch\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.232864 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231572    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3bf00b4-a637-4a14-87f6-a1bb2ae1a276-system-cni-dir\") pod \"multus-additional-cni-plugins-tjsbg\" (UID: \"f3bf00b4-a637-4a14-87f6-a1bb2ae1a276\") " pod="openshift-multus/multus-additional-cni-plugins-tjsbg"
Sep 16 18:40:34.232864 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231617    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f3bf00b4-a637-4a14-87f6-a1bb2ae1a276-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tjsbg\" (UID: \"f3bf00b4-a637-4a14-87f6-a1bb2ae1a276\") " pod="openshift-multus/multus-additional-cni-plugins-tjsbg"
Sep 16 18:40:34.232864 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231678    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-host-cni-netd\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.232864 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231708    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c3e865df-22be-4129-bbf3-300285c29e69-hosts-file\") pod \"node-resolver-xbtsz\" (UID: \"c3e865df-22be-4129-bbf3-300285c29e69\") " pod="openshift-dns/node-resolver-xbtsz"
Sep 16 18:40:34.232864 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231743    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-host-cni-netd\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.232864 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231733    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-multus-cni-dir\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.232864 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231780    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6ab1b46d-5a59-42a7-87c4-13c16da817ad-serviceca\") pod \"node-ca-95lgg\" (UID: \"6ab1b46d-5a59-42a7-87c4-13c16da817ad\") " pod="openshift-image-registry/node-ca-95lgg"
Sep 16 18:40:34.232864 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231808    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-run-ovn\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.232864 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231833    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-env-overrides\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.232864 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231858    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872-original-pull-secret\") pod \"global-pull-secret-syncer-qsdt4\" (UID: \"6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872\") " pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:40:34.232864 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-multus-daemon-config\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.232864 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231911    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-etc-kubernetes\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.232864 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231942    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5hrp\" (UniqueName: \"kubernetes.io/projected/f3bf00b4-a637-4a14-87f6-a1bb2ae1a276-kube-api-access-k5hrp\") pod \"multus-additional-cni-plugins-tjsbg\" (UID: \"f3bf00b4-a637-4a14-87f6-a1bb2ae1a276\") " pod="openshift-multus/multus-additional-cni-plugins-tjsbg"
Sep 16 18:40:34.232864 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.231992    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-ovn-node-metrics-cert\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.232864 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.232021    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-ovnkube-script-lib\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.232864 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.232047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptzz4\" (UniqueName: \"kubernetes.io/projected/c3e865df-22be-4129-bbf3-300285c29e69-kube-api-access-ptzz4\") pod \"node-resolver-xbtsz\" (UID: \"c3e865df-22be-4129-bbf3-300285c29e69\") " pod="openshift-dns/node-resolver-xbtsz"
Sep 16 18:40:34.233697 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.232103    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-var-lib-kubelet\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.233697 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.232130    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/99603daa-09aa-4da6-a429-17a33c23586f-iptables-alerter-script\") pod \"iptables-alerter-wfvjg\" (UID: \"99603daa-09aa-4da6-a429-17a33c23586f\") " pod="openshift-network-operator/iptables-alerter-wfvjg"
Sep 16 18:40:34.233697 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.232157    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3bf00b4-a637-4a14-87f6-a1bb2ae1a276-cnibin\") pod \"multus-additional-cni-plugins-tjsbg\" (UID: \"f3bf00b4-a637-4a14-87f6-a1bb2ae1a276\") " pod="openshift-multus/multus-additional-cni-plugins-tjsbg"
Sep 16 18:40:34.233697 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.232184    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7qx4\" (UniqueName: \"kubernetes.io/projected/6ab1b46d-5a59-42a7-87c4-13c16da817ad-kube-api-access-w7qx4\") pod \"node-ca-95lgg\" (UID: \"6ab1b46d-5a59-42a7-87c4-13c16da817ad\") " pod="openshift-image-registry/node-ca-95lgg"
Sep 16 18:40:34.233697 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.232212    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnn78\" (UniqueName: \"kubernetes.io/projected/733b59e1-b843-4255-bf37-71c81c244989-kube-api-access-hnn78\") pod \"network-metrics-daemon-xq67c\" (UID: \"733b59e1-b843-4255-bf37-71c81c244989\") " pod="openshift-multus/network-metrics-daemon-xq67c"
Sep 16 18:40:34.233697 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.232242    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-run-openvswitch\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.233697 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.232272    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3072b448-5de4-4f9f-b4db-2fb1ff05be1f-registration-dir\") pod \"aws-ebs-csi-driver-node-gkjtg\" (UID: \"3072b448-5de4-4f9f-b4db-2fb1ff05be1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkjtg"
Sep 16 18:40:34.233697 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.232298    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-env-overrides\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.233697 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.232300    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-host-var-lib-cni-multus\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.233697 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.232350    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4zmj\" (UniqueName: \"kubernetes.io/projected/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-kube-api-access-c4zmj\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.233697 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:34.232361    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:34.233697 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.232377    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mtp2\" (UniqueName: \"kubernetes.io/projected/99603daa-09aa-4da6-a429-17a33c23586f-kube-api-access-9mtp2\") pod \"iptables-alerter-wfvjg\" (UID: \"99603daa-09aa-4da6-a429-17a33c23586f\") " pod="openshift-network-operator/iptables-alerter-wfvjg"
Sep 16 18:40:34.233697 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.232404    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-cnibin\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.233697 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:34.232420    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872-original-pull-secret podName:6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:34.732404153 +0000 UTC m=+2.967337597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872-original-pull-secret") pod "global-pull-secret-syncer-qsdt4" (UID: "6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:34.233697 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.232420    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-run-openvswitch\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.233697 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.232274    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6ab1b46d-5a59-42a7-87c4-13c16da817ad-serviceca\") pod \"node-ca-95lgg\" (UID: \"6ab1b46d-5a59-42a7-87c4-13c16da817ad\") " pod="openshift-image-registry/node-ca-95lgg"
Sep 16 18:40:34.233697 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.232635    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-run-ovn\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.234368 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.232673    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-multus-conf-dir\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.234368 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.232716    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872-kubelet-config\") pod \"global-pull-secret-syncer-qsdt4\" (UID: \"6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872\") " pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:40:34.234368 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.232751    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-lib-modules\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.234368 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.232778    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3072b448-5de4-4f9f-b4db-2fb1ff05be1f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gkjtg\" (UID: \"3072b448-5de4-4f9f-b4db-2fb1ff05be1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkjtg"
Sep 16 18:40:34.234368 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.232810    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-host-var-lib-cni-bin\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.234368 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.232838    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3bf00b4-a637-4a14-87f6-a1bb2ae1a276-cni-binary-copy\") pod \"multus-additional-cni-plugins-tjsbg\" (UID: \"f3bf00b4-a637-4a14-87f6-a1bb2ae1a276\") " pod="openshift-multus/multus-additional-cni-plugins-tjsbg"
Sep 16 18:40:34.234368 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.232864    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-run\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.234368 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.232923    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-hostroot\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.234368 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.232973    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-node-log\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.234368 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.233001    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-etc-systemd\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.234368 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.233003    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872-kubelet-config\") pod \"global-pull-secret-syncer-qsdt4\" (UID: \"6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872\") " pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:40:34.234368 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.233025    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:40:34.234368 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.233036    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjhxd\" (UniqueName: \"kubernetes.io/projected/a5d250f2-42fe-41a9-91a5-3433fc67a942-kube-api-access-kjhxd\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.234368 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.233071    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-node-log\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.234368 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.233124    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-system-cni-dir\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.234368 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.233269    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-ovnkube-script-lib\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.237099 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.237079    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-ovn-node-metrics-cert\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.244553 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.244523    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7qx4\" (UniqueName: \"kubernetes.io/projected/6ab1b46d-5a59-42a7-87c4-13c16da817ad-kube-api-access-w7qx4\") pod \"node-ca-95lgg\" (UID: \"6ab1b46d-5a59-42a7-87c4-13c16da817ad\") " pod="openshift-image-registry/node-ca-95lgg"
Sep 16 18:40:34.244725 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.244669    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnn78\" (UniqueName: \"kubernetes.io/projected/733b59e1-b843-4255-bf37-71c81c244989-kube-api-access-hnn78\") pod \"network-metrics-daemon-xq67c\" (UID: \"733b59e1-b843-4255-bf37-71c81c244989\") " pod="openshift-multus/network-metrics-daemon-xq67c"
Sep 16 18:40:34.245024 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.245002    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hqcl\" (UniqueName: \"kubernetes.io/projected/6bcd73d6-7dcf-4712-b667-7a9fb2018ac2-kube-api-access-5hqcl\") pod \"ovnkube-node-lp55b\" (UID: \"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2\") " pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.333649 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.333616    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-etc-systemd\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.333795 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.333656    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjhxd\" (UniqueName: \"kubernetes.io/projected/a5d250f2-42fe-41a9-91a5-3433fc67a942-kube-api-access-kjhxd\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.333795 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.333682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-system-cni-dir\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.333795 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.333705    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3bf00b4-a637-4a14-87f6-a1bb2ae1a276-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tjsbg\" (UID: \"f3bf00b4-a637-4a14-87f6-a1bb2ae1a276\") " pod="openshift-multus/multus-additional-cni-plugins-tjsbg"
Sep 16 18:40:34.333795 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.333730    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fe4ed2ab-567f-4482-8c96-7293f1c82bb9-konnectivity-ca\") pod \"konnectivity-agent-gw4zp\" (UID: \"fe4ed2ab-567f-4482-8c96-7293f1c82bb9\") " pod="kube-system/konnectivity-agent-gw4zp"
Sep 16 18:40:34.333795 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.333770    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99603daa-09aa-4da6-a429-17a33c23586f-host-slash\") pod \"iptables-alerter-wfvjg\" (UID: \"99603daa-09aa-4da6-a429-17a33c23586f\") " pod="openshift-network-operator/iptables-alerter-wfvjg"
Sep 16 18:40:34.334056 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.333802    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-etc-sysctl-d\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.334056 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.333826    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-etc-sysctl-conf\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.334056 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.333850    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-sys\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.334056 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.333876    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f3bf00b4-a637-4a14-87f6-a1bb2ae1a276-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tjsbg\" (UID: \"f3bf00b4-a637-4a14-87f6-a1bb2ae1a276\") " pod="openshift-multus/multus-additional-cni-plugins-tjsbg"
Sep 16 18:40:34.334056 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.333896    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-system-cni-dir\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.334056 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.333901    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-host\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.334056 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.333948    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99603daa-09aa-4da6-a429-17a33c23586f-host-slash\") pod \"iptables-alerter-wfvjg\" (UID: \"99603daa-09aa-4da6-a429-17a33c23586f\") " pod="openshift-network-operator/iptables-alerter-wfvjg"
Sep 16 18:40:34.334056 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.333953    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5d250f2-42fe-41a9-91a5-3433fc67a942-tmp\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.334056 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.333967    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3bf00b4-a637-4a14-87f6-a1bb2ae1a276-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tjsbg\" (UID: \"f3bf00b4-a637-4a14-87f6-a1bb2ae1a276\") " pod="openshift-multus/multus-additional-cni-plugins-tjsbg"
Sep 16 18:40:34.334056 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.333983    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-multus-socket-dir-parent\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.334056 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334040    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-etc-sysctl-conf\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.334056 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334034    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-etc-kubernetes\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.334677 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334089    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-cni-binary-copy\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.334677 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334105    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-etc-kubernetes\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.334677 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334116    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72qjj\" (UniqueName: \"kubernetes.io/projected/feaf8529-d8fc-4f8d-8c78-e19f994b6f26-kube-api-access-72qjj\") pod \"network-check-target-djht2\" (UID: \"feaf8529-d8fc-4f8d-8c78-e19f994b6f26\") " pod="openshift-network-diagnostics/network-check-target-djht2"
Sep 16 18:40:34.334677 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334152    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fe4ed2ab-567f-4482-8c96-7293f1c82bb9-agent-certs\") pod \"konnectivity-agent-gw4zp\" (UID: \"fe4ed2ab-567f-4482-8c96-7293f1c82bb9\") " pod="kube-system/konnectivity-agent-gw4zp"
Sep 16 18:40:34.334677 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334185    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3e865df-22be-4129-bbf3-300285c29e69-tmp-dir\") pod \"node-resolver-xbtsz\" (UID: \"c3e865df-22be-4129-bbf3-300285c29e69\") " pod="openshift-dns/node-resolver-xbtsz"
Sep 16 18:40:34.334677 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334211    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a5d250f2-42fe-41a9-91a5-3433fc67a942-etc-tuned\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.334677 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334239    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3072b448-5de4-4f9f-b4db-2fb1ff05be1f-device-dir\") pod \"aws-ebs-csi-driver-node-gkjtg\" (UID: \"3072b448-5de4-4f9f-b4db-2fb1ff05be1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkjtg"
Sep 16 18:40:34.334677 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.333957    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-host\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.334677 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334259    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fe4ed2ab-567f-4482-8c96-7293f1c82bb9-konnectivity-ca\") pod \"konnectivity-agent-gw4zp\" (UID: \"fe4ed2ab-567f-4482-8c96-7293f1c82bb9\") " pod="kube-system/konnectivity-agent-gw4zp"
Sep 16 18:40:34.334677 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334268    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-etc-modprobe-d\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.334677 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334333    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-etc-sysconfig\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.334677 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334364    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3072b448-5de4-4f9f-b4db-2fb1ff05be1f-socket-dir\") pod \"aws-ebs-csi-driver-node-gkjtg\" (UID: \"3072b448-5de4-4f9f-b4db-2fb1ff05be1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkjtg"
Sep 16 18:40:34.334677 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334366    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-etc-systemd\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.334677 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334392    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-etc-modprobe-d\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.334677 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334394    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3072b448-5de4-4f9f-b4db-2fb1ff05be1f-sys-fs\") pod \"aws-ebs-csi-driver-node-gkjtg\" (UID: \"3072b448-5de4-4f9f-b4db-2fb1ff05be1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkjtg"
Sep 16 18:40:34.334677 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334432    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9887\" (UniqueName: \"kubernetes.io/projected/3072b448-5de4-4f9f-b4db-2fb1ff05be1f-kube-api-access-l9887\") pod \"aws-ebs-csi-driver-node-gkjtg\" (UID: \"3072b448-5de4-4f9f-b4db-2fb1ff05be1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkjtg"
Sep 16 18:40:34.334677 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334435    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-etc-sysconfig\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.335559 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334443    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3072b448-5de4-4f9f-b4db-2fb1ff05be1f-device-dir\") pod \"aws-ebs-csi-driver-node-gkjtg\" (UID: \"3072b448-5de4-4f9f-b4db-2fb1ff05be1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkjtg"
Sep 16 18:40:34.335559 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334464    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-host-run-netns\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.335559 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334493    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-host-run-netns\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.335559 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334497    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3072b448-5de4-4f9f-b4db-2fb1ff05be1f-socket-dir\") pod \"aws-ebs-csi-driver-node-gkjtg\" (UID: \"3072b448-5de4-4f9f-b4db-2fb1ff05be1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkjtg"
Sep 16 18:40:34.335559 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334494    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-host-run-multus-certs\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.335559 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334394    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-etc-sysctl-d\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.335559 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334538    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3072b448-5de4-4f9f-b4db-2fb1ff05be1f-etc-selinux\") pod \"aws-ebs-csi-driver-node-gkjtg\" (UID: \"3072b448-5de4-4f9f-b4db-2fb1ff05be1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkjtg"
Sep 16 18:40:34.335559 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334542    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-host-run-multus-certs\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.335559 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334558    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f3bf00b4-a637-4a14-87f6-a1bb2ae1a276-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tjsbg\" (UID: \"f3bf00b4-a637-4a14-87f6-a1bb2ae1a276\") " pod="openshift-multus/multus-additional-cni-plugins-tjsbg"
Sep 16 18:40:34.335559 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334567    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-host-run-k8s-cni-cncf-io\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.335559 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334570    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-sys\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.335559 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-host-var-lib-kubelet\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.335559 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334623    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3bf00b4-a637-4a14-87f6-a1bb2ae1a276-system-cni-dir\") pod \"multus-additional-cni-plugins-tjsbg\" (UID: \"f3bf00b4-a637-4a14-87f6-a1bb2ae1a276\") " pod="openshift-multus/multus-additional-cni-plugins-tjsbg"
Sep 16 18:40:34.335559 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334650    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f3bf00b4-a637-4a14-87f6-a1bb2ae1a276-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tjsbg\" (UID: \"f3bf00b4-a637-4a14-87f6-a1bb2ae1a276\") " pod="openshift-multus/multus-additional-cni-plugins-tjsbg"
Sep 16 18:40:34.335559 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334659    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3072b448-5de4-4f9f-b4db-2fb1ff05be1f-etc-selinux\") pod \"aws-ebs-csi-driver-node-gkjtg\" (UID: \"3072b448-5de4-4f9f-b4db-2fb1ff05be1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkjtg"
Sep 16 18:40:34.335559 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334693    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c3e865df-22be-4129-bbf3-300285c29e69-hosts-file\") pod \"node-resolver-xbtsz\" (UID: \"c3e865df-22be-4129-bbf3-300285c29e69\") " pod="openshift-dns/node-resolver-xbtsz"
Sep 16 18:40:34.335559 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334723    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3bf00b4-a637-4a14-87f6-a1bb2ae1a276-system-cni-dir\") pod \"multus-additional-cni-plugins-tjsbg\" (UID: \"f3bf00b4-a637-4a14-87f6-a1bb2ae1a276\") " pod="openshift-multus/multus-additional-cni-plugins-tjsbg"
Sep 16 18:40:34.336417 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334722    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-multus-cni-dir\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.336417 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334728    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-multus-socket-dir-parent\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.336417 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334771    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-host-var-lib-kubelet\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.336417 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334774    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c3e865df-22be-4129-bbf3-300285c29e69-hosts-file\") pod \"node-resolver-xbtsz\" (UID: \"c3e865df-22be-4129-bbf3-300285c29e69\") " pod="openshift-dns/node-resolver-xbtsz"
Sep 16 18:40:34.336417 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-multus-daemon-config\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.336417 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334833    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-multus-cni-dir\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.336417 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334841    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3e865df-22be-4129-bbf3-300285c29e69-tmp-dir\") pod \"node-resolver-xbtsz\" (UID: \"c3e865df-22be-4129-bbf3-300285c29e69\") " pod="openshift-dns/node-resolver-xbtsz"
Sep 16 18:40:34.336417 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334873    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-etc-kubernetes\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.336417 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334901    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5hrp\" (UniqueName: \"kubernetes.io/projected/f3bf00b4-a637-4a14-87f6-a1bb2ae1a276-kube-api-access-k5hrp\") pod \"multus-additional-cni-plugins-tjsbg\" (UID: \"f3bf00b4-a637-4a14-87f6-a1bb2ae1a276\") " pod="openshift-multus/multus-additional-cni-plugins-tjsbg"
Sep 16 18:40:34.336417 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334903    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3072b448-5de4-4f9f-b4db-2fb1ff05be1f-sys-fs\") pod \"aws-ebs-csi-driver-node-gkjtg\" (UID: \"3072b448-5de4-4f9f-b4db-2fb1ff05be1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkjtg"
Sep 16 18:40:34.336417 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.334920    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-etc-kubernetes\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.336417 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.335024    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptzz4\" (UniqueName: \"kubernetes.io/projected/c3e865df-22be-4129-bbf3-300285c29e69-kube-api-access-ptzz4\") pod \"node-resolver-xbtsz\" (UID: \"c3e865df-22be-4129-bbf3-300285c29e69\") " pod="openshift-dns/node-resolver-xbtsz"
Sep 16 18:40:34.336417 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.335071    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-cni-binary-copy\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.336417 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.335136    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-host-run-k8s-cni-cncf-io\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.336417 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.335173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-var-lib-kubelet\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.336417 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.335236    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-var-lib-kubelet\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.336417 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.335250    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/99603daa-09aa-4da6-a429-17a33c23586f-iptables-alerter-script\") pod \"iptables-alerter-wfvjg\" (UID: \"99603daa-09aa-4da6-a429-17a33c23586f\") " pod="openshift-network-operator/iptables-alerter-wfvjg"
Sep 16 18:40:34.336417 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.335271    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f3bf00b4-a637-4a14-87f6-a1bb2ae1a276-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tjsbg\" (UID: \"f3bf00b4-a637-4a14-87f6-a1bb2ae1a276\") " pod="openshift-multus/multus-additional-cni-plugins-tjsbg"
Sep 16 18:40:34.337033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.335284    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3bf00b4-a637-4a14-87f6-a1bb2ae1a276-cnibin\") pod \"multus-additional-cni-plugins-tjsbg\" (UID: \"f3bf00b4-a637-4a14-87f6-a1bb2ae1a276\") " pod="openshift-multus/multus-additional-cni-plugins-tjsbg"
Sep 16 18:40:34.337033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.335322    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3072b448-5de4-4f9f-b4db-2fb1ff05be1f-registration-dir\") pod \"aws-ebs-csi-driver-node-gkjtg\" (UID: \"3072b448-5de4-4f9f-b4db-2fb1ff05be1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkjtg"
Sep 16 18:40:34.337033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.335350    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-host-var-lib-cni-multus\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.337033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.335369    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3072b448-5de4-4f9f-b4db-2fb1ff05be1f-registration-dir\") pod \"aws-ebs-csi-driver-node-gkjtg\" (UID: \"3072b448-5de4-4f9f-b4db-2fb1ff05be1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkjtg"
Sep 16 18:40:34.337033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.335376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4zmj\" (UniqueName: \"kubernetes.io/projected/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-kube-api-access-c4zmj\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.337033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.335325    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3bf00b4-a637-4a14-87f6-a1bb2ae1a276-cnibin\") pod \"multus-additional-cni-plugins-tjsbg\" (UID: \"f3bf00b4-a637-4a14-87f6-a1bb2ae1a276\") " pod="openshift-multus/multus-additional-cni-plugins-tjsbg"
Sep 16 18:40:34.337033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.335404    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mtp2\" (UniqueName: \"kubernetes.io/projected/99603daa-09aa-4da6-a429-17a33c23586f-kube-api-access-9mtp2\") pod \"iptables-alerter-wfvjg\" (UID: \"99603daa-09aa-4da6-a429-17a33c23586f\") " pod="openshift-network-operator/iptables-alerter-wfvjg"
Sep 16 18:40:34.337033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.335414    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-host-var-lib-cni-multus\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.337033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.335429    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-cnibin\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.337033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.335454    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-multus-conf-dir\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.337033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.335460    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-multus-daemon-config\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.337033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.335478    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-lib-modules\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.337033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.335503    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3072b448-5de4-4f9f-b4db-2fb1ff05be1f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gkjtg\" (UID: \"3072b448-5de4-4f9f-b4db-2fb1ff05be1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkjtg"
Sep 16 18:40:34.337033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.335529    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-host-var-lib-cni-bin\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.337033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.335540    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-cnibin\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.337033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.335556    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3bf00b4-a637-4a14-87f6-a1bb2ae1a276-cni-binary-copy\") pod \"multus-additional-cni-plugins-tjsbg\" (UID: \"f3bf00b4-a637-4a14-87f6-a1bb2ae1a276\") " pod="openshift-multus/multus-additional-cni-plugins-tjsbg"
Sep 16 18:40:34.337033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.335582    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-run\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.337033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.335607    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-hostroot\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.337875 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.335614    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-host-var-lib-cni-bin\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.337875 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.335633    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-lib-modules\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.337875 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.335582    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-multus-conf-dir\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.337875 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.335683    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a5d250f2-42fe-41a9-91a5-3433fc67a942-run\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.337875 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.335692    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3072b448-5de4-4f9f-b4db-2fb1ff05be1f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gkjtg\" (UID: \"3072b448-5de4-4f9f-b4db-2fb1ff05be1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkjtg"
Sep 16 18:40:34.337875 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.335720    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-hostroot\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.337875 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.335778    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/99603daa-09aa-4da6-a429-17a33c23586f-iptables-alerter-script\") pod \"iptables-alerter-wfvjg\" (UID: \"99603daa-09aa-4da6-a429-17a33c23586f\") " pod="openshift-network-operator/iptables-alerter-wfvjg"
Sep 16 18:40:34.337875 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.336214    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3bf00b4-a637-4a14-87f6-a1bb2ae1a276-cni-binary-copy\") pod \"multus-additional-cni-plugins-tjsbg\" (UID: \"f3bf00b4-a637-4a14-87f6-a1bb2ae1a276\") " pod="openshift-multus/multus-additional-cni-plugins-tjsbg"
Sep 16 18:40:34.337875 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.336948    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a5d250f2-42fe-41a9-91a5-3433fc67a942-etc-tuned\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.337875 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.337086    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fe4ed2ab-567f-4482-8c96-7293f1c82bb9-agent-certs\") pod \"konnectivity-agent-gw4zp\" (UID: \"fe4ed2ab-567f-4482-8c96-7293f1c82bb9\") " pod="kube-system/konnectivity-agent-gw4zp"
Sep 16 18:40:34.337875 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.337747    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5d250f2-42fe-41a9-91a5-3433fc67a942-tmp\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.344481 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:34.344317    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:34.344481 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:34.344343    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:34.344481 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:34.344358    2578 projected.go:196] Error preparing data for projected volume kube-api-access-72qjj for pod openshift-network-diagnostics/network-check-target-djht2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:34.344481 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:34.344422    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/feaf8529-d8fc-4f8d-8c78-e19f994b6f26-kube-api-access-72qjj podName:feaf8529-d8fc-4f8d-8c78-e19f994b6f26 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:34.844405461 +0000 UTC m=+3.079338907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-72qjj" (UniqueName: "kubernetes.io/projected/feaf8529-d8fc-4f8d-8c78-e19f994b6f26-kube-api-access-72qjj") pod "network-check-target-djht2" (UID: "feaf8529-d8fc-4f8d-8c78-e19f994b6f26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:34.346656 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.346599    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjhxd\" (UniqueName: \"kubernetes.io/projected/a5d250f2-42fe-41a9-91a5-3433fc67a942-kube-api-access-kjhxd\") pod \"tuned-k7nnn\" (UID: \"a5d250f2-42fe-41a9-91a5-3433fc67a942\") " pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.347116 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.347094    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5hrp\" (UniqueName: \"kubernetes.io/projected/f3bf00b4-a637-4a14-87f6-a1bb2ae1a276-kube-api-access-k5hrp\") pod \"multus-additional-cni-plugins-tjsbg\" (UID: \"f3bf00b4-a637-4a14-87f6-a1bb2ae1a276\") " pod="openshift-multus/multus-additional-cni-plugins-tjsbg"
Sep 16 18:40:34.347475 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.347458    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptzz4\" (UniqueName: \"kubernetes.io/projected/c3e865df-22be-4129-bbf3-300285c29e69-kube-api-access-ptzz4\") pod \"node-resolver-xbtsz\" (UID: \"c3e865df-22be-4129-bbf3-300285c29e69\") " pod="openshift-dns/node-resolver-xbtsz"
Sep 16 18:40:34.347605 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.347585    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9887\" (UniqueName: \"kubernetes.io/projected/3072b448-5de4-4f9f-b4db-2fb1ff05be1f-kube-api-access-l9887\") pod \"aws-ebs-csi-driver-node-gkjtg\" (UID: \"3072b448-5de4-4f9f-b4db-2fb1ff05be1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkjtg"
Sep 16 18:40:34.348303 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.348263    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mtp2\" (UniqueName: \"kubernetes.io/projected/99603daa-09aa-4da6-a429-17a33c23586f-kube-api-access-9mtp2\") pod \"iptables-alerter-wfvjg\" (UID: \"99603daa-09aa-4da6-a429-17a33c23586f\") " pod="openshift-network-operator/iptables-alerter-wfvjg"
Sep 16 18:40:34.348490 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.348471    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4zmj\" (UniqueName: \"kubernetes.io/projected/9a8c38dd-ebe0-4ea8-904e-894b478c3e58-kube-api-access-c4zmj\") pod \"multus-4fj62\" (UID: \"9a8c38dd-ebe0-4ea8-904e-894b478c3e58\") " pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.354970 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.354932    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:34.422436 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.422411    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-95lgg"
Sep 16 18:40:34.431507 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.431482    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:40:34.441051 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.441029    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xbtsz"
Sep 16 18:40:34.446955 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.446939    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k7nnn"
Sep 16 18:40:34.454943 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.454926    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4fj62"
Sep 16 18:40:34.461833 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.461812    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkjtg"
Sep 16 18:40:34.469583 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.469562    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wfvjg"
Sep 16 18:40:34.476498 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.476477    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tjsbg"
Sep 16 18:40:34.482220 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.482203    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gw4zp"
Sep 16 18:40:34.737744 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.737652    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872-original-pull-secret\") pod \"global-pull-secret-syncer-qsdt4\" (UID: \"6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872\") " pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:40:34.737744 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.737718    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/733b59e1-b843-4255-bf37-71c81c244989-metrics-certs\") pod \"network-metrics-daemon-xq67c\" (UID: \"733b59e1-b843-4255-bf37-71c81c244989\") " pod="openshift-multus/network-metrics-daemon-xq67c"
Sep 16 18:40:34.737989 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:34.737848    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:34.737989 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:34.737879    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:34.737989 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:34.737929    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872-original-pull-secret podName:6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:35.737906897 +0000 UTC m=+3.972840336 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872-original-pull-secret") pod "global-pull-secret-syncer-qsdt4" (UID: "6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:34.737989 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:34.737950    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/733b59e1-b843-4255-bf37-71c81c244989-metrics-certs podName:733b59e1-b843-4255-bf37-71c81c244989 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:35.737940384 +0000 UTC m=+3.972873819 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/733b59e1-b843-4255-bf37-71c81c244989-metrics-certs") pod "network-metrics-daemon-xq67c" (UID: "733b59e1-b843-4255-bf37-71c81c244989") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:34.939514 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:34.939463    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72qjj\" (UniqueName: \"kubernetes.io/projected/feaf8529-d8fc-4f8d-8c78-e19f994b6f26-kube-api-access-72qjj\") pod \"network-check-target-djht2\" (UID: \"feaf8529-d8fc-4f8d-8c78-e19f994b6f26\") " pod="openshift-network-diagnostics/network-check-target-djht2"
Sep 16 18:40:34.939693 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:34.939632    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:34.939693 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:34.939654    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:34.939693 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:34.939671    2578 projected.go:196] Error preparing data for projected volume kube-api-access-72qjj for pod openshift-network-diagnostics/network-check-target-djht2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:34.939835 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:34.939739    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/feaf8529-d8fc-4f8d-8c78-e19f994b6f26-kube-api-access-72qjj podName:feaf8529-d8fc-4f8d-8c78-e19f994b6f26 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:35.939719259 +0000 UTC m=+4.174652705 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-72qjj" (UniqueName: "kubernetes.io/projected/feaf8529-d8fc-4f8d-8c78-e19f994b6f26-kube-api-access-72qjj") pod "network-check-target-djht2" (UID: "feaf8529-d8fc-4f8d-8c78-e19f994b6f26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:35.177755 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:35.177716    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:35:34 +0000 UTC" deadline="2028-06-16 01:53:50.550994732 +0000 UTC"
Sep 16 18:40:35.177755 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:35.177750    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15319h13m15.373249345s"
Sep 16 18:40:35.203398 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:35.203364    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a8c38dd_ebe0_4ea8_904e_894b478c3e58.slice/crio-51800f1c231beb5b9cc48bdcd23d2688259388a5631c201113562170ca25c9cb WatchSource:0}: Error finding container 51800f1c231beb5b9cc48bdcd23d2688259388a5631c201113562170ca25c9cb: Status 404 returned error can't find the container with id 51800f1c231beb5b9cc48bdcd23d2688259388a5631c201113562170ca25c9cb
Sep 16 18:40:35.204350 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:35.204329    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99603daa_09aa_4da6_a429_17a33c23586f.slice/crio-de85570fba670ae48670c9e49a149186680ec215f350346b552909127f303f74 WatchSource:0}: Error finding container de85570fba670ae48670c9e49a149186680ec215f350346b552909127f303f74: Status 404 returned error can't find the container with id de85570fba670ae48670c9e49a149186680ec215f350346b552909127f303f74
Sep 16 18:40:35.205851 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:35.205821    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe4ed2ab_567f_4482_8c96_7293f1c82bb9.slice/crio-4f9422a70e6e467f85d65f51114acd7cb399b62d00e34989c0bbc293dd85ff98 WatchSource:0}: Error finding container 4f9422a70e6e467f85d65f51114acd7cb399b62d00e34989c0bbc293dd85ff98: Status 404 returned error can't find the container with id 4f9422a70e6e467f85d65f51114acd7cb399b62d00e34989c0bbc293dd85ff98
Sep 16 18:40:35.208447 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:35.208275    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d250f2_42fe_41a9_91a5_3433fc67a942.slice/crio-3487581b317960a48f9ca6d64e189c42cf2bfc1052307a0bca3ef483d7330a98 WatchSource:0}: Error finding container 3487581b317960a48f9ca6d64e189c42cf2bfc1052307a0bca3ef483d7330a98: Status 404 returned error can't find the container with id 3487581b317960a48f9ca6d64e189c42cf2bfc1052307a0bca3ef483d7330a98
Sep 16 18:40:35.211638 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:35.211595    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e865df_22be_4129_bbf3_300285c29e69.slice/crio-3f6facb4a704fba3f6e8a10792dc99983bf5074f9d0d4cb94417aea3be4520a7 WatchSource:0}: Error finding container 3f6facb4a704fba3f6e8a10792dc99983bf5074f9d0d4cb94417aea3be4520a7: Status 404 returned error can't find the container with id 3f6facb4a704fba3f6e8a10792dc99983bf5074f9d0d4cb94417aea3be4520a7
Sep 16 18:40:35.212261 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:35.212243    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3072b448_5de4_4f9f_b4db_2fb1ff05be1f.slice/crio-f3565ec8600e20169954a30b12a7816018150b5256e937f5c5551117884ea9a7 WatchSource:0}: Error finding container f3565ec8600e20169954a30b12a7816018150b5256e937f5c5551117884ea9a7: Status 404 returned error can't find the container with id f3565ec8600e20169954a30b12a7816018150b5256e937f5c5551117884ea9a7
Sep 16 18:40:35.214004 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:35.213939    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3bf00b4_a637_4a14_87f6_a1bb2ae1a276.slice/crio-23c76f77d60b0ddfcb27ce10a6603ef5f84bbea85df9e287076fb4f7c395543f WatchSource:0}: Error finding container 23c76f77d60b0ddfcb27ce10a6603ef5f84bbea85df9e287076fb4f7c395543f: Status 404 returned error can't find the container with id 23c76f77d60b0ddfcb27ce10a6603ef5f84bbea85df9e287076fb4f7c395543f
Sep 16 18:40:35.214921 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:35.214825    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bcd73d6_7dcf_4712_b667_7a9fb2018ac2.slice/crio-7e42770e39c6dc278c75216e3cd61a19531127b2d6e1a31e625e39e5982f01f0 WatchSource:0}: Error finding container 7e42770e39c6dc278c75216e3cd61a19531127b2d6e1a31e625e39e5982f01f0: Status 404 returned error can't find the container with id 7e42770e39c6dc278c75216e3cd61a19531127b2d6e1a31e625e39e5982f01f0
Sep 16 18:40:35.216591 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:40:35.216505    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab1b46d_5a59_42a7_87c4_13c16da817ad.slice/crio-c02fb788c6c216850bb6c64080887f8e14f7ebb2b1ed186d31fc9f7ea5f512ac WatchSource:0}: Error finding container c02fb788c6c216850bb6c64080887f8e14f7ebb2b1ed186d31fc9f7ea5f512ac: Status 404 returned error can't find the container with id c02fb788c6c216850bb6c64080887f8e14f7ebb2b1ed186d31fc9f7ea5f512ac
Sep 16 18:40:35.220556 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:35.220530    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gw4zp" event={"ID":"fe4ed2ab-567f-4482-8c96-7293f1c82bb9","Type":"ContainerStarted","Data":"4f9422a70e6e467f85d65f51114acd7cb399b62d00e34989c0bbc293dd85ff98"}
Sep 16 18:40:35.221875 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:35.221847    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lp55b" event={"ID":"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2","Type":"ContainerStarted","Data":"7e42770e39c6dc278c75216e3cd61a19531127b2d6e1a31e625e39e5982f01f0"}
Sep 16 18:40:35.223035 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:35.223007    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkjtg" event={"ID":"3072b448-5de4-4f9f-b4db-2fb1ff05be1f","Type":"ContainerStarted","Data":"f3565ec8600e20169954a30b12a7816018150b5256e937f5c5551117884ea9a7"}
Sep 16 18:40:35.224986 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:35.224959    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k7nnn" event={"ID":"a5d250f2-42fe-41a9-91a5-3433fc67a942","Type":"ContainerStarted","Data":"3487581b317960a48f9ca6d64e189c42cf2bfc1052307a0bca3ef483d7330a98"}
Sep 16 18:40:35.226456 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:35.226433    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wfvjg" event={"ID":"99603daa-09aa-4da6-a429-17a33c23586f","Type":"ContainerStarted","Data":"de85570fba670ae48670c9e49a149186680ec215f350346b552909127f303f74"}
Sep 16 18:40:35.227969 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:35.227934    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4fj62" event={"ID":"9a8c38dd-ebe0-4ea8-904e-894b478c3e58","Type":"ContainerStarted","Data":"51800f1c231beb5b9cc48bdcd23d2688259388a5631c201113562170ca25c9cb"}
Sep 16 18:40:35.232489 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:35.232460    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-95lgg" event={"ID":"6ab1b46d-5a59-42a7-87c4-13c16da817ad","Type":"ContainerStarted","Data":"c02fb788c6c216850bb6c64080887f8e14f7ebb2b1ed186d31fc9f7ea5f512ac"}
Sep 16 18:40:35.233773 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:35.233747    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tjsbg" event={"ID":"f3bf00b4-a637-4a14-87f6-a1bb2ae1a276","Type":"ContainerStarted","Data":"23c76f77d60b0ddfcb27ce10a6603ef5f84bbea85df9e287076fb4f7c395543f"}
Sep 16 18:40:35.234910 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:35.234890    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xbtsz" event={"ID":"c3e865df-22be-4129-bbf3-300285c29e69","Type":"ContainerStarted","Data":"3f6facb4a704fba3f6e8a10792dc99983bf5074f9d0d4cb94417aea3be4520a7"}
Sep 16 18:40:35.746773 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:35.746689    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/733b59e1-b843-4255-bf37-71c81c244989-metrics-certs\") pod \"network-metrics-daemon-xq67c\" (UID: \"733b59e1-b843-4255-bf37-71c81c244989\") " pod="openshift-multus/network-metrics-daemon-xq67c"
Sep 16 18:40:35.746929 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:35.746783    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872-original-pull-secret\") pod \"global-pull-secret-syncer-qsdt4\" (UID: \"6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872\") " pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:40:35.746929 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:35.746917    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:35.747044 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:35.746977    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872-original-pull-secret podName:6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:37.746958412 +0000 UTC m=+5.981891852 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872-original-pull-secret") pod "global-pull-secret-syncer-qsdt4" (UID: "6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:35.747240 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:35.747220    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:35.747318 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:35.747280    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/733b59e1-b843-4255-bf37-71c81c244989-metrics-certs podName:733b59e1-b843-4255-bf37-71c81c244989 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:37.747262165 +0000 UTC m=+5.982195604 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/733b59e1-b843-4255-bf37-71c81c244989-metrics-certs") pod "network-metrics-daemon-xq67c" (UID: "733b59e1-b843-4255-bf37-71c81c244989") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:35.949153 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:35.949100    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72qjj\" (UniqueName: \"kubernetes.io/projected/feaf8529-d8fc-4f8d-8c78-e19f994b6f26-kube-api-access-72qjj\") pod \"network-check-target-djht2\" (UID: \"feaf8529-d8fc-4f8d-8c78-e19f994b6f26\") " pod="openshift-network-diagnostics/network-check-target-djht2"
Sep 16 18:40:35.949346 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:35.949309    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:35.949346 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:35.949331    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:35.949346 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:35.949344    2578 projected.go:196] Error preparing data for projected volume kube-api-access-72qjj for pod openshift-network-diagnostics/network-check-target-djht2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:35.949531 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:35.949403    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/feaf8529-d8fc-4f8d-8c78-e19f994b6f26-kube-api-access-72qjj podName:feaf8529-d8fc-4f8d-8c78-e19f994b6f26 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:37.949385251 +0000 UTC m=+6.184318692 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-72qjj" (UniqueName: "kubernetes.io/projected/feaf8529-d8fc-4f8d-8c78-e19f994b6f26-kube-api-access-72qjj") pod "network-check-target-djht2" (UID: "feaf8529-d8fc-4f8d-8c78-e19f994b6f26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:36.178793 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:36.178676    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:35:34 +0000 UTC" deadline="2028-06-04 18:04:23.02222421 +0000 UTC"
Sep 16 18:40:36.178793 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:36.178741    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15047h23m46.843487277s"
Sep 16 18:40:36.231086 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:36.231056    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:40:36.231270 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:36.231198    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsdt4" podUID="6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872"
Sep 16 18:40:36.231663 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:36.231599    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djht2"
Sep 16 18:40:36.231756 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:36.231693    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djht2" podUID="feaf8529-d8fc-4f8d-8c78-e19f994b6f26"
Sep 16 18:40:36.231824 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:36.231767    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xq67c"
Sep 16 18:40:36.231881 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:36.231858    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xq67c" podUID="733b59e1-b843-4255-bf37-71c81c244989"
Sep 16 18:40:36.251532 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:36.251501    2578 generic.go:356] "Generic (PLEG): container finished" podID="ff3af002549164d48d170f209bd2fe9c" containerID="8d7a2cbe98ee9bb43f357e04891cfb0f1f1aaa15980fb7c5c0ce957e7719be02" exitCode=0
Sep 16 18:40:36.251643 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:36.251595    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal" event={"ID":"ff3af002549164d48d170f209bd2fe9c","Type":"ContainerDied","Data":"8d7a2cbe98ee9bb43f357e04891cfb0f1f1aaa15980fb7c5c0ce957e7719be02"}
Sep 16 18:40:36.269615 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:36.269537    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-80.ec2.internal" event={"ID":"61224f73c725c86faa0f55f841c5d69e","Type":"ContainerStarted","Data":"30019753a605fbe1db3de6878157b7a5712d74965db274588f1000dfd71a65ed"}
Sep 16 18:40:36.288478 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:36.288424    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-80.ec2.internal" podStartSLOduration=3.288405772 podStartE2EDuration="3.288405772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:36.288254417 +0000 UTC m=+4.523187866" watchObservedRunningTime="2026-09-16 18:40:36.288405772 +0000 UTC m=+4.523339221"
Sep 16 18:40:37.294479 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:37.294183    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal" event={"ID":"ff3af002549164d48d170f209bd2fe9c","Type":"ContainerStarted","Data":"6e5743ef5be98465fef807b2466c37dab0378ad18e6af7de024dee54195c0d88"}
Sep 16 18:40:37.764780 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:37.764740    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872-original-pull-secret\") pod \"global-pull-secret-syncer-qsdt4\" (UID: \"6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872\") " pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:40:37.764973 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:37.764824    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/733b59e1-b843-4255-bf37-71c81c244989-metrics-certs\") pod \"network-metrics-daemon-xq67c\" (UID: \"733b59e1-b843-4255-bf37-71c81c244989\") " pod="openshift-multus/network-metrics-daemon-xq67c"
Sep 16 18:40:37.765063 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:37.765045    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:37.765205 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:37.765141    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/733b59e1-b843-4255-bf37-71c81c244989-metrics-certs podName:733b59e1-b843-4255-bf37-71c81c244989 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:41.765119213 +0000 UTC m=+10.000052654 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/733b59e1-b843-4255-bf37-71c81c244989-metrics-certs") pod "network-metrics-daemon-xq67c" (UID: "733b59e1-b843-4255-bf37-71c81c244989") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:37.765286 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:37.765222    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:37.765286 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:37.765265    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872-original-pull-secret podName:6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:41.765251375 +0000 UTC m=+10.000184815 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872-original-pull-secret") pod "global-pull-secret-syncer-qsdt4" (UID: "6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:37.966627 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:37.966590    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72qjj\" (UniqueName: \"kubernetes.io/projected/feaf8529-d8fc-4f8d-8c78-e19f994b6f26-kube-api-access-72qjj\") pod \"network-check-target-djht2\" (UID: \"feaf8529-d8fc-4f8d-8c78-e19f994b6f26\") " pod="openshift-network-diagnostics/network-check-target-djht2"
Sep 16 18:40:37.966830 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:37.966812    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:37.966900 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:37.966839    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:37.966900 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:37.966852    2578 projected.go:196] Error preparing data for projected volume kube-api-access-72qjj for pod openshift-network-diagnostics/network-check-target-djht2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:37.967010 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:37.966913    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/feaf8529-d8fc-4f8d-8c78-e19f994b6f26-kube-api-access-72qjj podName:feaf8529-d8fc-4f8d-8c78-e19f994b6f26 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:41.966892087 +0000 UTC m=+10.201825528 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-72qjj" (UniqueName: "kubernetes.io/projected/feaf8529-d8fc-4f8d-8c78-e19f994b6f26-kube-api-access-72qjj") pod "network-check-target-djht2" (UID: "feaf8529-d8fc-4f8d-8c78-e19f994b6f26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:38.220519 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:38.219440    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:40:38.220519 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:38.219570    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsdt4" podUID="6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872"
Sep 16 18:40:38.220519 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:38.220145    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djht2"
Sep 16 18:40:38.220519 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:38.220256    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djht2" podUID="feaf8529-d8fc-4f8d-8c78-e19f994b6f26"
Sep 16 18:40:38.220519 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:38.220315    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xq67c"
Sep 16 18:40:38.220519 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:38.220393    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xq67c" podUID="733b59e1-b843-4255-bf37-71c81c244989"
Sep 16 18:40:40.216880 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:40.216851    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xq67c"
Sep 16 18:40:40.217393 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:40.216861    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:40:40.217393 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:40.216979    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xq67c" podUID="733b59e1-b843-4255-bf37-71c81c244989"
Sep 16 18:40:40.217393 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:40.216863    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djht2"
Sep 16 18:40:40.217393 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:40.217059    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsdt4" podUID="6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872"
Sep 16 18:40:40.217393 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:40.217193    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djht2" podUID="feaf8529-d8fc-4f8d-8c78-e19f994b6f26"
Sep 16 18:40:41.798059 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:41.798013    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872-original-pull-secret\") pod \"global-pull-secret-syncer-qsdt4\" (UID: \"6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872\") " pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:40:41.798521 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:41.798088    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/733b59e1-b843-4255-bf37-71c81c244989-metrics-certs\") pod \"network-metrics-daemon-xq67c\" (UID: \"733b59e1-b843-4255-bf37-71c81c244989\") " pod="openshift-multus/network-metrics-daemon-xq67c"
Sep 16 18:40:41.798521 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:41.798163    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:41.798521 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:41.798211    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:41.798521 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:41.798235    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872-original-pull-secret podName:6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:49.798215579 +0000 UTC m=+18.033149015 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872-original-pull-secret") pod "global-pull-secret-syncer-qsdt4" (UID: "6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:41.798521 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:41.798260    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/733b59e1-b843-4255-bf37-71c81c244989-metrics-certs podName:733b59e1-b843-4255-bf37-71c81c244989 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:49.798243749 +0000 UTC m=+18.033177187 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/733b59e1-b843-4255-bf37-71c81c244989-metrics-certs") pod "network-metrics-daemon-xq67c" (UID: "733b59e1-b843-4255-bf37-71c81c244989") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:41.999995 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:41.999957    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72qjj\" (UniqueName: \"kubernetes.io/projected/feaf8529-d8fc-4f8d-8c78-e19f994b6f26-kube-api-access-72qjj\") pod \"network-check-target-djht2\" (UID: \"feaf8529-d8fc-4f8d-8c78-e19f994b6f26\") " pod="openshift-network-diagnostics/network-check-target-djht2"
Sep 16 18:40:42.000207 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:42.000152    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:42.000207 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:42.000174    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:42.000207 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:42.000186    2578 projected.go:196] Error preparing data for projected volume kube-api-access-72qjj for pod openshift-network-diagnostics/network-check-target-djht2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:42.000386 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:42.000243    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/feaf8529-d8fc-4f8d-8c78-e19f994b6f26-kube-api-access-72qjj podName:feaf8529-d8fc-4f8d-8c78-e19f994b6f26 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:50.000226317 +0000 UTC m=+18.235159765 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-72qjj" (UniqueName: "kubernetes.io/projected/feaf8529-d8fc-4f8d-8c78-e19f994b6f26-kube-api-access-72qjj") pod "network-check-target-djht2" (UID: "feaf8529-d8fc-4f8d-8c78-e19f994b6f26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:42.215571 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:42.214743    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:40:42.215571 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:42.214860    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsdt4" podUID="6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872"
Sep 16 18:40:42.215571 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:42.215241    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djht2"
Sep 16 18:40:42.215571 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:42.215340    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djht2" podUID="feaf8529-d8fc-4f8d-8c78-e19f994b6f26"
Sep 16 18:40:42.215571 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:42.215393    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xq67c"
Sep 16 18:40:42.215571 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:42.215471    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xq67c" podUID="733b59e1-b843-4255-bf37-71c81c244989"
Sep 16 18:40:44.211408 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:44.211373    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:40:44.211822 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:44.211381    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xq67c"
Sep 16 18:40:44.211822 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:44.211505    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsdt4" podUID="6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872"
Sep 16 18:40:44.211822 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:44.211522    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djht2"
Sep 16 18:40:44.211822 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:44.211643    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xq67c" podUID="733b59e1-b843-4255-bf37-71c81c244989"
Sep 16 18:40:44.211822 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:44.211727    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djht2" podUID="feaf8529-d8fc-4f8d-8c78-e19f994b6f26"
Sep 16 18:40:46.211894 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:46.211859    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xq67c"
Sep 16 18:40:46.211894 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:46.211887    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djht2"
Sep 16 18:40:46.212408 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:46.211914    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:40:46.212408 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:46.212012    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djht2" podUID="feaf8529-d8fc-4f8d-8c78-e19f994b6f26"
Sep 16 18:40:46.212408 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:46.212083    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xq67c" podUID="733b59e1-b843-4255-bf37-71c81c244989"
Sep 16 18:40:46.212408 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:46.212167    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsdt4" podUID="6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872"
Sep 16 18:40:48.214637 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:48.214603    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:40:48.214637 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:48.214620    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djht2"
Sep 16 18:40:48.215147 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:48.214624    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xq67c"
Sep 16 18:40:48.215147 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:48.214717    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsdt4" podUID="6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872"
Sep 16 18:40:48.215147 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:48.214787    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djht2" podUID="feaf8529-d8fc-4f8d-8c78-e19f994b6f26"
Sep 16 18:40:48.215147 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:48.214884    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xq67c" podUID="733b59e1-b843-4255-bf37-71c81c244989"
Sep 16 18:40:49.862912 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:49.862875    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/733b59e1-b843-4255-bf37-71c81c244989-metrics-certs\") pod \"network-metrics-daemon-xq67c\" (UID: \"733b59e1-b843-4255-bf37-71c81c244989\") " pod="openshift-multus/network-metrics-daemon-xq67c"
Sep 16 18:40:49.863383 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:49.862943    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872-original-pull-secret\") pod \"global-pull-secret-syncer-qsdt4\" (UID: \"6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872\") " pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:40:49.863383 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:49.863029    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:49.863383 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:49.863068    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:49.863383 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:49.863108    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/733b59e1-b843-4255-bf37-71c81c244989-metrics-certs podName:733b59e1-b843-4255-bf37-71c81c244989 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:05.863091061 +0000 UTC m=+34.098024501 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/733b59e1-b843-4255-bf37-71c81c244989-metrics-certs") pod "network-metrics-daemon-xq67c" (UID: "733b59e1-b843-4255-bf37-71c81c244989") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:49.863383 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:49.863125    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872-original-pull-secret podName:6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:05.863117536 +0000 UTC m=+34.098050958 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872-original-pull-secret") pod "global-pull-secret-syncer-qsdt4" (UID: "6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:50.065152 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:50.065124    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72qjj\" (UniqueName: \"kubernetes.io/projected/feaf8529-d8fc-4f8d-8c78-e19f994b6f26-kube-api-access-72qjj\") pod \"network-check-target-djht2\" (UID: \"feaf8529-d8fc-4f8d-8c78-e19f994b6f26\") " pod="openshift-network-diagnostics/network-check-target-djht2"
Sep 16 18:40:50.065325 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:50.065240    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:50.065325 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:50.065259    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:50.065325 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:50.065272    2578 projected.go:196] Error preparing data for projected volume kube-api-access-72qjj for pod openshift-network-diagnostics/network-check-target-djht2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:50.065325 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:50.065320    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/feaf8529-d8fc-4f8d-8c78-e19f994b6f26-kube-api-access-72qjj podName:feaf8529-d8fc-4f8d-8c78-e19f994b6f26 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:06.065307019 +0000 UTC m=+34.300240440 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-72qjj" (UniqueName: "kubernetes.io/projected/feaf8529-d8fc-4f8d-8c78-e19f994b6f26-kube-api-access-72qjj") pod "network-check-target-djht2" (UID: "feaf8529-d8fc-4f8d-8c78-e19f994b6f26") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:50.211905 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:50.211859    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xq67c"
Sep 16 18:40:50.211905 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:50.211890    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djht2"
Sep 16 18:40:50.212135 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:50.211995    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xq67c" podUID="733b59e1-b843-4255-bf37-71c81c244989"
Sep 16 18:40:50.212135 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:50.212063    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:40:50.212258 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:50.212201    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djht2" podUID="feaf8529-d8fc-4f8d-8c78-e19f994b6f26"
Sep 16 18:40:50.212310 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:50.212285    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsdt4" podUID="6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872"
Sep 16 18:40:52.211900 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:52.211867    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xq67c"
Sep 16 18:40:52.212345 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:52.211972    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:40:52.212345 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:52.211973    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xq67c" podUID="733b59e1-b843-4255-bf37-71c81c244989"
Sep 16 18:40:52.212345 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:52.212078    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djht2"
Sep 16 18:40:52.212345 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:52.212078    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsdt4" podUID="6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872"
Sep 16 18:40:52.212345 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:52.212191    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djht2" podUID="feaf8529-d8fc-4f8d-8c78-e19f994b6f26"
Sep 16 18:40:53.328230 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:53.328051    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkjtg" event={"ID":"3072b448-5de4-4f9f-b4db-2fb1ff05be1f","Type":"ContainerStarted","Data":"9fc6afd8253a11d2927a46f06a1acdde96a1f7d04af038f80056ce1a52142076"}
Sep 16 18:40:53.330819 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:53.330764    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k7nnn" event={"ID":"a5d250f2-42fe-41a9-91a5-3433fc67a942","Type":"ContainerStarted","Data":"18bb8af9b3dcde05d0e7fcc32c7d75ddf7da21275ae7002e79d8fcd152fb0e5a"}
Sep 16 18:40:53.333479 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:53.333203    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4fj62" event={"ID":"9a8c38dd-ebe0-4ea8-904e-894b478c3e58","Type":"ContainerStarted","Data":"389e49eb3bd31e89ecb75e904959fa29c4acf663d4586de39ce37331dbee6780"}
Sep 16 18:40:53.335607 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:53.335579    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-95lgg" event={"ID":"6ab1b46d-5a59-42a7-87c4-13c16da817ad","Type":"ContainerStarted","Data":"04ca25f6a63d5be3920c977e0e35c612cd973e9d1a3e4692b007373258210ccc"}
Sep 16 18:40:53.337397 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:53.337375    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tjsbg" event={"ID":"f3bf00b4-a637-4a14-87f6-a1bb2ae1a276","Type":"ContainerStarted","Data":"4acb71a996906e847fd0f2804fb548f5e2890197664fa2880b0d4d8f54bf9c85"}
Sep 16 18:40:53.339956 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:53.339931    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xbtsz" event={"ID":"c3e865df-22be-4129-bbf3-300285c29e69","Type":"ContainerStarted","Data":"0834f81a52b4096d4db39f3b109740cc0b09a96cb85c0d319d1824f6fd925a8f"}
Sep 16 18:40:53.342692 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:53.342661    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gw4zp" event={"ID":"fe4ed2ab-567f-4482-8c96-7293f1c82bb9","Type":"ContainerStarted","Data":"68e8a6ee2c367adc94b0eb78b01c36ca8b17779573d3b48167c2caf75088c874"}
Sep 16 18:40:53.345428 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:53.345407    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lp55b" event={"ID":"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2","Type":"ContainerStarted","Data":"750a8bb2455f4920a1f1bd4c26ec3e2a01cf07faf2d9e06073baa648e62cfa8c"}
Sep 16 18:40:53.345497 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:53.345484    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lp55b" event={"ID":"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2","Type":"ContainerStarted","Data":"2bd90bbcbc3835453d5638a8f46ffa32b1949b4ea62ba0ce4a719a946f04e9e9"}
Sep 16 18:40:53.357867 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:53.357813    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-k7nnn" podStartSLOduration=3.512085585 podStartE2EDuration="21.357797265s" totalImagesPullingTime="17.84571168s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:53.357709933 +0000 UTC m=+21.592643376" watchObservedRunningTime="2026-09-16 18:40:53.357797265 +0000 UTC m=+21.592730710"
Sep 16 18:40:53.358065 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:53.358020    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-80.ec2.internal" podStartSLOduration=20.358013457 podStartE2EDuration="20.358013457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:37.312297006 +0000 UTC m=+5.547230484" watchObservedRunningTime="2026-09-16 18:40:53.358013457 +0000 UTC m=+21.592946902"
Sep 16 18:40:53.413209 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:53.413155    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-95lgg" podStartSLOduration=3.5759640900000003 podStartE2EDuration="21.4131322s" totalImagesPullingTime="17.83716811s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:53.384294378 +0000 UTC m=+21.619227818" watchObservedRunningTime="2026-09-16 18:40:53.4131322 +0000 UTC m=+21.648065664"
Sep 16 18:40:53.432760 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:53.432720    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4fj62" podStartSLOduration=3.459733432 podStartE2EDuration="21.432704129s" totalImagesPullingTime="17.972970697s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:53.413775376 +0000 UTC m=+21.648708820" watchObservedRunningTime="2026-09-16 18:40:53.432704129 +0000 UTC m=+21.667637573"
Sep 16 18:40:53.433021 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:53.433001    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xbtsz" podStartSLOduration=3.590174248 podStartE2EDuration="21.432993571s" totalImagesPullingTime="17.842819323s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:53.432440669 +0000 UTC m=+21.667374149" watchObservedRunningTime="2026-09-16 18:40:53.432993571 +0000 UTC m=+21.667927015"
Sep 16 18:40:53.485946 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:53.485903    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gw4zp" podStartSLOduration=11.1209947 podStartE2EDuration="21.485886529s" totalImagesPullingTime="10.364891829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:53.485232228 +0000 UTC m=+21.720165671" watchObservedRunningTime="2026-09-16 18:40:53.485886529 +0000 UTC m=+21.720819972"
Sep 16 18:40:54.211860 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:54.211828    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xq67c"
Sep 16 18:40:54.212013 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:54.211828    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:40:54.212013 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:54.211969    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djht2"
Sep 16 18:40:54.212013 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:54.211975    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xq67c" podUID="733b59e1-b843-4255-bf37-71c81c244989"
Sep 16 18:40:54.212210 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:54.212061    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsdt4" podUID="6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872"
Sep 16 18:40:54.212210 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:54.212154    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djht2" podUID="feaf8529-d8fc-4f8d-8c78-e19f994b6f26"
Sep 16 18:40:54.349141 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:54.349055    2578 generic.go:356] "Generic (PLEG): container finished" podID="f3bf00b4-a637-4a14-87f6-a1bb2ae1a276" containerID="4acb71a996906e847fd0f2804fb548f5e2890197664fa2880b0d4d8f54bf9c85" exitCode=0
Sep 16 18:40:54.349467 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:54.349150    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tjsbg" event={"ID":"f3bf00b4-a637-4a14-87f6-a1bb2ae1a276","Type":"ContainerDied","Data":"4acb71a996906e847fd0f2804fb548f5e2890197664fa2880b0d4d8f54bf9c85"}
Sep 16 18:40:54.352811 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:54.352785    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lp55b" event={"ID":"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2","Type":"ContainerStarted","Data":"ce3f9778a40fd44f089a5a11cf6722c8e739f7e639b81262ee0c2b317402b369"}
Sep 16 18:40:54.352915 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:54.352849    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lp55b" event={"ID":"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2","Type":"ContainerStarted","Data":"cc3bd820389eb8233957a88d3b4c38ab84dc071075e3db1dbc13829ca80c1580"}
Sep 16 18:40:54.352915 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:54.352893    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lp55b" event={"ID":"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2","Type":"ContainerStarted","Data":"9d2feda3bd25597577d9f714bea2aee9048ed9e39c00994512383bc43f9acd48"}
Sep 16 18:40:54.353028 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:54.352937    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lp55b" event={"ID":"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2","Type":"ContainerStarted","Data":"dbb0e8ff8c8a98c24aae111c467ab47246f5b1b8c320d01270f1904fc2f81f44"}
Sep 16 18:40:54.621365 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:54.621301    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:40:54.840620 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:54.840587    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gw4zp"
Sep 16 18:40:54.841146 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:54.841125    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gw4zp"
Sep 16 18:40:55.181891 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:55.181782    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:40:54.621332523Z","UUID":"6d552de9-fdf3-432b-984a-8e286bece7ff","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:40:55.183463 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:55.183433    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:40:55.183587 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:55.183476    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:40:55.359985 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:55.359948    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkjtg" event={"ID":"3072b448-5de4-4f9f-b4db-2fb1ff05be1f","Type":"ContainerStarted","Data":"4c982d4adcff6b7445b1a28e7bdac3d389455ce55fe5c7db3cca18d24578de94"}
Sep 16 18:40:55.361699 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:55.361644    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wfvjg" event={"ID":"99603daa-09aa-4da6-a429-17a33c23586f","Type":"ContainerStarted","Data":"f4764911f484f670f10b5079184b4e65f4ccaeb3bdc1aed2296684a838f8fa34"}
Sep 16 18:40:56.211508 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:56.211478    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xq67c"
Sep 16 18:40:56.211840 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:56.211535    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djht2"
Sep 16 18:40:56.211840 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:56.211550    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:40:56.211840 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:56.211640    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsdt4" podUID="6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872"
Sep 16 18:40:56.211840 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:56.211722    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xq67c" podUID="733b59e1-b843-4255-bf37-71c81c244989"
Sep 16 18:40:56.211840 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:56.211788    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djht2" podUID="feaf8529-d8fc-4f8d-8c78-e19f994b6f26"
Sep 16 18:40:56.364364 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:56.364337    2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 18:40:57.394130 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:57.394087    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wfvjg" podStartSLOduration=7.544880604 podStartE2EDuration="25.394065736s" totalImagesPullingTime="17.849185132s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:55.379745125 +0000 UTC m=+23.614678570" watchObservedRunningTime="2026-09-16 18:40:57.394065736 +0000 UTC m=+25.628999180"
Sep 16 18:40:57.394669 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:57.394628    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkjtg" podStartSLOduration=5.271278365 podStartE2EDuration="25.394620494s" totalImagesPullingTime="20.123342129s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:57.394205758 +0000 UTC m=+25.629139194" watchObservedRunningTime="2026-09-16 18:40:57.394620494 +0000 UTC m=+25.629553938"
Sep 16 18:40:58.211676 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:58.211531    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xq67c"
Sep 16 18:40:58.211830 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:58.211533    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djht2"
Sep 16 18:40:58.211830 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:58.211750    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xq67c" podUID="733b59e1-b843-4255-bf37-71c81c244989"
Sep 16 18:40:58.211830 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:58.211818    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djht2" podUID="feaf8529-d8fc-4f8d-8c78-e19f994b6f26"
Sep 16 18:40:58.211953 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:58.211610    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:40:58.211953 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:40:58.211892    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsdt4" podUID="6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872"
Sep 16 18:40:58.378702 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:58.378674    2578 generic.go:356] "Generic (PLEG): container finished" podID="f3bf00b4-a637-4a14-87f6-a1bb2ae1a276" containerID="4e669ea8ff5f5ab9376c4883894939d9387986d7cc191b81f565a882b25fadbd" exitCode=0
Sep 16 18:40:58.378812 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:58.378753    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tjsbg" event={"ID":"f3bf00b4-a637-4a14-87f6-a1bb2ae1a276","Type":"ContainerDied","Data":"4e669ea8ff5f5ab9376c4883894939d9387986d7cc191b81f565a882b25fadbd"}
Sep 16 18:40:58.382315 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:58.382297    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lp55b" event={"ID":"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2","Type":"ContainerStarted","Data":"d1813887c49ccdd37544fdffd0c7bff3662cd930f14ebd5a6721012b8c80d3b1"}
Sep 16 18:40:58.387732 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:58.387706    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gkjtg" event={"ID":"3072b448-5de4-4f9f-b4db-2fb1ff05be1f","Type":"ContainerStarted","Data":"45a9d7012bb7a27ae069ada40cf629c315629972f01130435fdb95b29134bcc3"}
Sep 16 18:40:58.782164 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:58.782138    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gw4zp"
Sep 16 18:40:58.782694 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:58.782258    2578 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 18:40:58.782764 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:40:58.782720    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gw4zp"
Sep 16 18:41:00.211678 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:00.211646    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xq67c"
Sep 16 18:41:00.211678 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:00.211671    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djht2"
Sep 16 18:41:00.212343 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:00.211646    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:41:00.212343 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:00.211762    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xq67c" podUID="733b59e1-b843-4255-bf37-71c81c244989"
Sep 16 18:41:00.212343 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:00.211813    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsdt4" podUID="6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872"
Sep 16 18:41:00.212343 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:00.211888    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djht2" podUID="feaf8529-d8fc-4f8d-8c78-e19f994b6f26"
Sep 16 18:41:00.393707 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:00.393677    2578 generic.go:356] "Generic (PLEG): container finished" podID="f3bf00b4-a637-4a14-87f6-a1bb2ae1a276" containerID="94927221e109f1572cbd4012dd96f3df5f7dec4b68dbe0b0de15fe26f2c2773c" exitCode=0
Sep 16 18:41:00.393866 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:00.393756    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tjsbg" event={"ID":"f3bf00b4-a637-4a14-87f6-a1bb2ae1a276","Type":"ContainerDied","Data":"94927221e109f1572cbd4012dd96f3df5f7dec4b68dbe0b0de15fe26f2c2773c"}
Sep 16 18:41:00.397629 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:00.397602    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lp55b" event={"ID":"6bcd73d6-7dcf-4712-b667-7a9fb2018ac2","Type":"ContainerStarted","Data":"a5d1251589abe9fff07afebd7ab51c23143cb9e2867f6918d6babd267acbf8f3"}
Sep 16 18:41:00.397848 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:00.397829    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:41:00.397931 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:00.397888    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:41:00.414137 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:00.414118    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:41:00.447461 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:00.447421    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lp55b" podStartSLOduration=10.561405567 podStartE2EDuration="28.447408774s" totalImagesPullingTime="17.886003207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:00.447275631 +0000 UTC m=+28.682209074" watchObservedRunningTime="2026-09-16 18:41:00.447408774 +0000 UTC m=+28.682342233"
Sep 16 18:41:01.404136 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:01.404018    2578 generic.go:356] "Generic (PLEG): container finished" podID="f3bf00b4-a637-4a14-87f6-a1bb2ae1a276" containerID="689f2c84a5d0ba703ad234e34de7d2b8ce05a5641b62126815d1a3a075a71b37" exitCode=0
Sep 16 18:41:01.404136 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:01.404075    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tjsbg" event={"ID":"f3bf00b4-a637-4a14-87f6-a1bb2ae1a276","Type":"ContainerDied","Data":"689f2c84a5d0ba703ad234e34de7d2b8ce05a5641b62126815d1a3a075a71b37"}
Sep 16 18:41:01.404878 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:01.404861    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:41:01.421640 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:01.421610    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:41:01.657465 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:01.657434    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qsdt4"]
Sep 16 18:41:01.657618 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:01.657566    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:41:01.657724 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:01.657680    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsdt4" podUID="6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872"
Sep 16 18:41:01.660280 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:01.660246    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xq67c"]
Sep 16 18:41:01.660402 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:01.660365    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xq67c"
Sep 16 18:41:01.660508 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:01.660483    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xq67c" podUID="733b59e1-b843-4255-bf37-71c81c244989"
Sep 16 18:41:01.672802 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:01.672780    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-djht2"]
Sep 16 18:41:01.672909 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:01.672860    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djht2"
Sep 16 18:41:01.672962 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:01.672933    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djht2" podUID="feaf8529-d8fc-4f8d-8c78-e19f994b6f26"
Sep 16 18:41:03.211900 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:03.211865    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:41:03.212424 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:03.211978    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djht2"
Sep 16 18:41:03.212424 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:03.211987    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qsdt4" podUID="6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872"
Sep 16 18:41:03.212424 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:03.212016    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xq67c"
Sep 16 18:41:03.212424 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:03.212131    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-djht2" podUID="feaf8529-d8fc-4f8d-8c78-e19f994b6f26"
Sep 16 18:41:03.212424 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:03.212228    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xq67c" podUID="733b59e1-b843-4255-bf37-71c81c244989"
Sep 16 18:41:05.045992 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.045803    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-80.ec2.internal" event="NodeReady"
Sep 16 18:41:05.046396 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.046101    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:41:05.116812 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.116776    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-scmml"]
Sep 16 18:41:05.119282 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.119261    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-86bz5"]
Sep 16 18:41:05.119414 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.119362    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-scmml"
Sep 16 18:41:05.121257 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.121239    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-86bz5"
Sep 16 18:41:05.123408 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.123375    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:41:05.123532 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.123515    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:41:05.123666 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.123650    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:41:05.124664 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.123650    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hq9jm\""
Sep 16 18:41:05.125695 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.125060    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:41:05.125695 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.125255    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:41:05.125853 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.125750    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-67vjb\""
Sep 16 18:41:05.135014 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.134996    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-scmml"]
Sep 16 18:41:05.136256 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.136236    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-86bz5"]
Sep 16 18:41:05.211554 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.211525    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djht2"
Sep 16 18:41:05.211677 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.211628    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xq67c"
Sep 16 18:41:05.211874 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.211768    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:41:05.214373 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.214351    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:41:05.214491 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.214354    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:41:05.214631 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.214614    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:41:05.214713 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.214682    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-g58c8\""
Sep 16 18:41:05.214786 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.214769    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s9z5g\""
Sep 16 18:41:05.214854 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.214812    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:41:05.277886 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.277856    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbhcw\" (UniqueName: \"kubernetes.io/projected/5dc0ed37-07b5-439b-94cf-1cf3104ad233-kube-api-access-wbhcw\") pod \"ingress-canary-scmml\" (UID: \"5dc0ed37-07b5-439b-94cf-1cf3104ad233\") " pod="openshift-ingress-canary/ingress-canary-scmml"
Sep 16 18:41:05.278006 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.277927    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/309dbcdf-d468-4001-805b-e71bb5bdd775-config-volume\") pod \"dns-default-86bz5\" (UID: \"309dbcdf-d468-4001-805b-e71bb5bdd775\") " pod="openshift-dns/dns-default-86bz5"
Sep 16 18:41:05.278006 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.277967    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5dc0ed37-07b5-439b-94cf-1cf3104ad233-cert\") pod \"ingress-canary-scmml\" (UID: \"5dc0ed37-07b5-439b-94cf-1cf3104ad233\") " pod="openshift-ingress-canary/ingress-canary-scmml"
Sep 16 18:41:05.278006 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.277990    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6nsm\" (UniqueName: \"kubernetes.io/projected/309dbcdf-d468-4001-805b-e71bb5bdd775-kube-api-access-k6nsm\") pod \"dns-default-86bz5\" (UID: \"309dbcdf-d468-4001-805b-e71bb5bdd775\") " pod="openshift-dns/dns-default-86bz5"
Sep 16 18:41:05.278205 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.278029    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/309dbcdf-d468-4001-805b-e71bb5bdd775-tmp-dir\") pod \"dns-default-86bz5\" (UID: \"309dbcdf-d468-4001-805b-e71bb5bdd775\") " pod="openshift-dns/dns-default-86bz5"
Sep 16 18:41:05.278205 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.278147    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/309dbcdf-d468-4001-805b-e71bb5bdd775-metrics-tls\") pod \"dns-default-86bz5\" (UID: \"309dbcdf-d468-4001-805b-e71bb5bdd775\") " pod="openshift-dns/dns-default-86bz5"
Sep 16 18:41:05.379280 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.379199    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/309dbcdf-d468-4001-805b-e71bb5bdd775-metrics-tls\") pod \"dns-default-86bz5\" (UID: \"309dbcdf-d468-4001-805b-e71bb5bdd775\") " pod="openshift-dns/dns-default-86bz5"
Sep 16 18:41:05.379280 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.379265    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbhcw\" (UniqueName: \"kubernetes.io/projected/5dc0ed37-07b5-439b-94cf-1cf3104ad233-kube-api-access-wbhcw\") pod \"ingress-canary-scmml\" (UID: \"5dc0ed37-07b5-439b-94cf-1cf3104ad233\") " pod="openshift-ingress-canary/ingress-canary-scmml"
Sep 16 18:41:05.379474 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.379307    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/309dbcdf-d468-4001-805b-e71bb5bdd775-config-volume\") pod \"dns-default-86bz5\" (UID: \"309dbcdf-d468-4001-805b-e71bb5bdd775\") " pod="openshift-dns/dns-default-86bz5"
Sep 16 18:41:05.379474 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:05.379351    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:05.379474 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:05.379430    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/309dbcdf-d468-4001-805b-e71bb5bdd775-metrics-tls podName:309dbcdf-d468-4001-805b-e71bb5bdd775 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:05.879409744 +0000 UTC m=+34.114343180 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/309dbcdf-d468-4001-805b-e71bb5bdd775-metrics-tls") pod "dns-default-86bz5" (UID: "309dbcdf-d468-4001-805b-e71bb5bdd775") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:05.379622 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.379356    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5dc0ed37-07b5-439b-94cf-1cf3104ad233-cert\") pod \"ingress-canary-scmml\" (UID: \"5dc0ed37-07b5-439b-94cf-1cf3104ad233\") " pod="openshift-ingress-canary/ingress-canary-scmml"
Sep 16 18:41:05.379622 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.379569    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6nsm\" (UniqueName: \"kubernetes.io/projected/309dbcdf-d468-4001-805b-e71bb5bdd775-kube-api-access-k6nsm\") pod \"dns-default-86bz5\" (UID: \"309dbcdf-d468-4001-805b-e71bb5bdd775\") " pod="openshift-dns/dns-default-86bz5"
Sep 16 18:41:05.379712 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.379626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/309dbcdf-d468-4001-805b-e71bb5bdd775-tmp-dir\") pod \"dns-default-86bz5\" (UID: \"309dbcdf-d468-4001-805b-e71bb5bdd775\") " pod="openshift-dns/dns-default-86bz5"
Sep 16 18:41:05.380072 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.379954    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/309dbcdf-d468-4001-805b-e71bb5bdd775-tmp-dir\") pod \"dns-default-86bz5\" (UID: \"309dbcdf-d468-4001-805b-e71bb5bdd775\") " pod="openshift-dns/dns-default-86bz5"
Sep 16 18:41:05.380238 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.380211    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/309dbcdf-d468-4001-805b-e71bb5bdd775-config-volume\") pod \"dns-default-86bz5\" (UID: \"309dbcdf-d468-4001-805b-e71bb5bdd775\") " pod="openshift-dns/dns-default-86bz5"
Sep 16 18:41:05.383719 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.383698    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5dc0ed37-07b5-439b-94cf-1cf3104ad233-cert\") pod \"ingress-canary-scmml\" (UID: \"5dc0ed37-07b5-439b-94cf-1cf3104ad233\") " pod="openshift-ingress-canary/ingress-canary-scmml"
Sep 16 18:41:05.400438 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.400415    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbhcw\" (UniqueName: \"kubernetes.io/projected/5dc0ed37-07b5-439b-94cf-1cf3104ad233-kube-api-access-wbhcw\") pod \"ingress-canary-scmml\" (UID: \"5dc0ed37-07b5-439b-94cf-1cf3104ad233\") " pod="openshift-ingress-canary/ingress-canary-scmml"
Sep 16 18:41:05.401318 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.401295    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6nsm\" (UniqueName: \"kubernetes.io/projected/309dbcdf-d468-4001-805b-e71bb5bdd775-kube-api-access-k6nsm\") pod \"dns-default-86bz5\" (UID: \"309dbcdf-d468-4001-805b-e71bb5bdd775\") " pod="openshift-dns/dns-default-86bz5"
Sep 16 18:41:05.433351 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.433327    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-scmml"
Sep 16 18:41:05.856239 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.856213    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-scmml"]
Sep 16 18:41:05.860483 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:41:05.860456    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc0ed37_07b5_439b_94cf_1cf3104ad233.slice/crio-21656f9d0aa153833c91eba028114084dfd4ca1ee61a0a9118b9eb0fe1781e51 WatchSource:0}: Error finding container 21656f9d0aa153833c91eba028114084dfd4ca1ee61a0a9118b9eb0fe1781e51: Status 404 returned error can't find the container with id 21656f9d0aa153833c91eba028114084dfd4ca1ee61a0a9118b9eb0fe1781e51
Sep 16 18:41:05.884140 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.884089    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/733b59e1-b843-4255-bf37-71c81c244989-metrics-certs\") pod \"network-metrics-daemon-xq67c\" (UID: \"733b59e1-b843-4255-bf37-71c81c244989\") " pod="openshift-multus/network-metrics-daemon-xq67c"
Sep 16 18:41:05.884219 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.884143    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/309dbcdf-d468-4001-805b-e71bb5bdd775-metrics-tls\") pod \"dns-default-86bz5\" (UID: \"309dbcdf-d468-4001-805b-e71bb5bdd775\") " pod="openshift-dns/dns-default-86bz5"
Sep 16 18:41:05.884219 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.884184    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872-original-pull-secret\") pod \"global-pull-secret-syncer-qsdt4\" (UID: \"6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872\") " pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:41:05.884322 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:05.884232    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:41:05.884322 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:05.884288    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/733b59e1-b843-4255-bf37-71c81c244989-metrics-certs podName:733b59e1-b843-4255-bf37-71c81c244989 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:37.884270216 +0000 UTC m=+66.119203650 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/733b59e1-b843-4255-bf37-71c81c244989-metrics-certs") pod "network-metrics-daemon-xq67c" (UID: "733b59e1-b843-4255-bf37-71c81c244989") : secret "metrics-daemon-secret" not found
Sep 16 18:41:05.884419 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:05.884326    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:05.884419 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:05.884389    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/309dbcdf-d468-4001-805b-e71bb5bdd775-metrics-tls podName:309dbcdf-d468-4001-805b-e71bb5bdd775 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:06.884371152 +0000 UTC m=+35.119304592 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/309dbcdf-d468-4001-805b-e71bb5bdd775-metrics-tls") pod "dns-default-86bz5" (UID: "309dbcdf-d468-4001-805b-e71bb5bdd775") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:05.886441 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:05.886419    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872-original-pull-secret\") pod \"global-pull-secret-syncer-qsdt4\" (UID: \"6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872\") " pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:41:06.086169 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:06.086142    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72qjj\" (UniqueName: \"kubernetes.io/projected/feaf8529-d8fc-4f8d-8c78-e19f994b6f26-kube-api-access-72qjj\") pod \"network-check-target-djht2\" (UID: \"feaf8529-d8fc-4f8d-8c78-e19f994b6f26\") " pod="openshift-network-diagnostics/network-check-target-djht2"
Sep 16 18:41:06.088343 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:06.088326    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72qjj\" (UniqueName: \"kubernetes.io/projected/feaf8529-d8fc-4f8d-8c78-e19f994b6f26-kube-api-access-72qjj\") pod \"network-check-target-djht2\" (UID: \"feaf8529-d8fc-4f8d-8c78-e19f994b6f26\") " pod="openshift-network-diagnostics/network-check-target-djht2"
Sep 16 18:41:06.122400 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:06.122381    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-djht2"
Sep 16 18:41:06.135294 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:06.135232    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qsdt4"
Sep 16 18:41:06.399167 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:06.399141    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-djht2"]
Sep 16 18:41:06.401670 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:41:06.401641    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeaf8529_d8fc_4f8d_8c78_e19f994b6f26.slice/crio-4f9992edf31f36705ff372d10db179f750fbfa834ac6e0051d7d2d0b1fba5f80 WatchSource:0}: Error finding container 4f9992edf31f36705ff372d10db179f750fbfa834ac6e0051d7d2d0b1fba5f80: Status 404 returned error can't find the container with id 4f9992edf31f36705ff372d10db179f750fbfa834ac6e0051d7d2d0b1fba5f80
Sep 16 18:41:06.413107 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:06.413080    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qsdt4"]
Sep 16 18:41:06.416582 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:41:06.416558    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b56f6c6_d7a4_4ca4_a9f1_d7670c1ee872.slice/crio-00ad5305ea18a682f77d372b69aa2b1915bb15c3fd9960fd6f282bbf6b8a351f WatchSource:0}: Error finding container 00ad5305ea18a682f77d372b69aa2b1915bb15c3fd9960fd6f282bbf6b8a351f: Status 404 returned error can't find the container with id 00ad5305ea18a682f77d372b69aa2b1915bb15c3fd9960fd6f282bbf6b8a351f
Sep 16 18:41:06.419866 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:06.419839    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-djht2" event={"ID":"feaf8529-d8fc-4f8d-8c78-e19f994b6f26","Type":"ContainerStarted","Data":"4f9992edf31f36705ff372d10db179f750fbfa834ac6e0051d7d2d0b1fba5f80"}
Sep 16 18:41:06.423643 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:06.423545    2578 generic.go:356] "Generic (PLEG): container finished" podID="f3bf00b4-a637-4a14-87f6-a1bb2ae1a276" containerID="58cd069b2b30db3d917245c5ee03ce2f60ffc2e8b3b0b344d9c406fb29f1d26e" exitCode=0
Sep 16 18:41:06.423643 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:06.423590    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tjsbg" event={"ID":"f3bf00b4-a637-4a14-87f6-a1bb2ae1a276","Type":"ContainerDied","Data":"58cd069b2b30db3d917245c5ee03ce2f60ffc2e8b3b0b344d9c406fb29f1d26e"}
Sep 16 18:41:06.424934 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:06.424903    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-scmml" event={"ID":"5dc0ed37-07b5-439b-94cf-1cf3104ad233","Type":"ContainerStarted","Data":"21656f9d0aa153833c91eba028114084dfd4ca1ee61a0a9118b9eb0fe1781e51"}
Sep 16 18:41:06.892266 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:06.891998    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/309dbcdf-d468-4001-805b-e71bb5bdd775-metrics-tls\") pod \"dns-default-86bz5\" (UID: \"309dbcdf-d468-4001-805b-e71bb5bdd775\") " pod="openshift-dns/dns-default-86bz5"
Sep 16 18:41:06.892266 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:06.892195    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:06.892266 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:06.892258    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/309dbcdf-d468-4001-805b-e71bb5bdd775-metrics-tls podName:309dbcdf-d468-4001-805b-e71bb5bdd775 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:08.892240058 +0000 UTC m=+37.127173509 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/309dbcdf-d468-4001-805b-e71bb5bdd775-metrics-tls") pod "dns-default-86bz5" (UID: "309dbcdf-d468-4001-805b-e71bb5bdd775") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:07.432261 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:07.432102    2578 generic.go:356] "Generic (PLEG): container finished" podID="f3bf00b4-a637-4a14-87f6-a1bb2ae1a276" containerID="14429f977e0505c0c9f2318e4c0b6a49ed9c5cb8f20d4b9b409e93aaa001dad7" exitCode=0
Sep 16 18:41:07.432261 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:07.432144    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tjsbg" event={"ID":"f3bf00b4-a637-4a14-87f6-a1bb2ae1a276","Type":"ContainerDied","Data":"14429f977e0505c0c9f2318e4c0b6a49ed9c5cb8f20d4b9b409e93aaa001dad7"}
Sep 16 18:41:07.434772 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:07.434711    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qsdt4" event={"ID":"6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872","Type":"ContainerStarted","Data":"00ad5305ea18a682f77d372b69aa2b1915bb15c3fd9960fd6f282bbf6b8a351f"}
Sep 16 18:41:08.906323 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:08.906240    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/309dbcdf-d468-4001-805b-e71bb5bdd775-metrics-tls\") pod \"dns-default-86bz5\" (UID: \"309dbcdf-d468-4001-805b-e71bb5bdd775\") " pod="openshift-dns/dns-default-86bz5"
Sep 16 18:41:08.906768 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:08.906393    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:08.906768 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:08.906479    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/309dbcdf-d468-4001-805b-e71bb5bdd775-metrics-tls podName:309dbcdf-d468-4001-805b-e71bb5bdd775 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:12.906456039 +0000 UTC m=+41.141389476 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/309dbcdf-d468-4001-805b-e71bb5bdd775-metrics-tls") pod "dns-default-86bz5" (UID: "309dbcdf-d468-4001-805b-e71bb5bdd775") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:09.444187 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:09.444147    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tjsbg" event={"ID":"f3bf00b4-a637-4a14-87f6-a1bb2ae1a276","Type":"ContainerStarted","Data":"5b8cb6a64b1fd087fb3f8127ad9080b43b13e514418ce531acc68b14aa6073e0"}
Sep 16 18:41:09.477313 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:09.477255    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tjsbg" podStartSLOduration=9.498297092 podStartE2EDuration="37.477234883s" totalImagesPullingTime="27.978937791s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:09.475518803 +0000 UTC m=+37.710452248" watchObservedRunningTime="2026-09-16 18:41:09.477234883 +0000 UTC m=+37.712168328"
Sep 16 18:41:12.454961 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:12.454925    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-djht2" event={"ID":"feaf8529-d8fc-4f8d-8c78-e19f994b6f26","Type":"ContainerStarted","Data":"c0c0d18fe0ed04bed8791d38fbefebe1e41d3839b591fa1d39b8913d781920b8"}
Sep 16 18:41:12.455450 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:12.454981    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-djht2"
Sep 16 18:41:12.456427 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:12.456405    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-scmml" event={"ID":"5dc0ed37-07b5-439b-94cf-1cf3104ad233","Type":"ContainerStarted","Data":"9b9f10680aa9ec2cd3c7f6eb2dceeab4d1e0736d81e41141135fd30cfaf49ccc"}
Sep 16 18:41:12.457843 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:12.457822    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qsdt4" event={"ID":"6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872","Type":"ContainerStarted","Data":"3bdfa78093e02ebd5b0a05897d46d9f217349d6975130942a32e58a3f3516532"}
Sep 16 18:41:12.472583 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:12.472542    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-djht2" podStartSLOduration=35.522390826 podStartE2EDuration="40.472529377s" totalImagesPullingTime="4.950138551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:12.471876002 +0000 UTC m=+40.706809448" watchObservedRunningTime="2026-09-16 18:41:12.472529377 +0000 UTC m=+40.707462822"
Sep 16 18:41:12.491441 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:12.491403    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qsdt4" podStartSLOduration=35.544452592 podStartE2EDuration="40.491389765s" totalImagesPullingTime="4.946937173s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:12.491216157 +0000 UTC m=+40.726149600" watchObservedRunningTime="2026-09-16 18:41:12.491389765 +0000 UTC m=+40.726323237"
Sep 16 18:41:12.521266 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:12.521227    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-scmml" podStartSLOduration=2.045098195 podStartE2EDuration="7.521215364s" totalImagesPullingTime="5.476117169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:12.520138707 +0000 UTC m=+40.755072164" watchObservedRunningTime="2026-09-16 18:41:12.521215364 +0000 UTC m=+40.756148808"
Sep 16 18:41:12.932756 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:12.932732    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/309dbcdf-d468-4001-805b-e71bb5bdd775-metrics-tls\") pod \"dns-default-86bz5\" (UID: \"309dbcdf-d468-4001-805b-e71bb5bdd775\") " pod="openshift-dns/dns-default-86bz5"
Sep 16 18:41:12.932890 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:12.932818    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:12.932890 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:12.932867    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/309dbcdf-d468-4001-805b-e71bb5bdd775-metrics-tls podName:309dbcdf-d468-4001-805b-e71bb5bdd775 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:20.932854045 +0000 UTC m=+49.167787467 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/309dbcdf-d468-4001-805b-e71bb5bdd775-metrics-tls") pod "dns-default-86bz5" (UID: "309dbcdf-d468-4001-805b-e71bb5bdd775") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:20.979431 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:20.979396    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/309dbcdf-d468-4001-805b-e71bb5bdd775-metrics-tls\") pod \"dns-default-86bz5\" (UID: \"309dbcdf-d468-4001-805b-e71bb5bdd775\") " pod="openshift-dns/dns-default-86bz5"
Sep 16 18:41:20.979781 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:20.979503    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:20.979781 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:20.979559    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/309dbcdf-d468-4001-805b-e71bb5bdd775-metrics-tls podName:309dbcdf-d468-4001-805b-e71bb5bdd775 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:36.979542004 +0000 UTC m=+65.214475427 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/309dbcdf-d468-4001-805b-e71bb5bdd775-metrics-tls") pod "dns-default-86bz5" (UID: "309dbcdf-d468-4001-805b-e71bb5bdd775") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:29.952404 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:29.952369    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-2n2dw"]
Sep 16 18:41:29.972475 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:29.972454    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-2n2dw"]
Sep 16 18:41:29.972617 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:29.972553    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2n2dw"
Sep 16 18:41:29.975343 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:29.975288    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:41:29.975343 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:29.975286    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 16 18:41:29.975343 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:29.975328    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 16 18:41:29.975623 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:29.975288    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 16 18:41:29.976481 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:29.976464    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-tgwgz\""
Sep 16 18:41:30.062126 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.062104    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-cjpnz"]
Sep 16 18:41:30.075507 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.075485    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8fc56db65-5rbz9"]
Sep 16 18:41:30.075609 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.075597    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cjpnz"
Sep 16 18:41:30.077913 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.077893    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:41:30.078014 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.077895    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 16 18:41:30.079477 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.079463    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 16 18:41:30.079564 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.079465    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:41:30.079564 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.079493    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-5dnd8\""
Sep 16 18:41:30.101426 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.101407    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-8zxkm"]
Sep 16 18:41:30.101528 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.101513    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:41:30.104095 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.104075    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:41:30.104198 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.104115    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:41:30.104353 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.104337    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:41:30.104496 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.104476    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xjxjk\""
Sep 16 18:41:30.112679 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.112655    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:41:30.120321 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.120304    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xfz8z"]
Sep 16 18:41:30.120422 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.120407    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8zxkm"
Sep 16 18:41:30.122873 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.122858    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 16 18:41:30.122954 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.122938    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-qb52x\""
Sep 16 18:41:30.123004 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.122964    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 16 18:41:30.133776 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.133753    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f49e184-c6c2-4ffe-bccb-681321fa052a-serving-cert\") pod \"service-ca-operator-cf7c776b7-2n2dw\" (UID: \"6f49e184-c6c2-4ffe-bccb-681321fa052a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2n2dw"
Sep 16 18:41:30.133889 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.133794    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f49e184-c6c2-4ffe-bccb-681321fa052a-config\") pod \"service-ca-operator-cf7c776b7-2n2dw\" (UID: \"6f49e184-c6c2-4ffe-bccb-681321fa052a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2n2dw"
Sep 16 18:41:30.133889 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.133847    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d2fb\" (UniqueName: \"kubernetes.io/projected/6f49e184-c6c2-4ffe-bccb-681321fa052a-kube-api-access-5d2fb\") pod \"service-ca-operator-cf7c776b7-2n2dw\" (UID: \"6f49e184-c6c2-4ffe-bccb-681321fa052a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2n2dw"
Sep 16 18:41:30.145028 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.145002    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8fc56db65-5rbz9"]
Sep 16 18:41:30.145135 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.145071    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-8zxkm"]
Sep 16 18:41:30.145135 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.145109    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xfz8z"
Sep 16 18:41:30.145135 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.145129    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-cjpnz"]
Sep 16 18:41:30.145293 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.145159    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xfz8z"]
Sep 16 18:41:30.147624 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.147604    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 16 18:41:30.147718 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.147668    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:41:30.147718 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.147680    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 16 18:41:30.147718 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.147698    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 16 18:41:30.147930 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.147808    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-zdbzm\""
Sep 16 18:41:30.234219 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.234170    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f1abe079-27f1-4078-a08c-406b881a7bde-image-registry-private-configuration\") pod \"image-registry-8fc56db65-5rbz9\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") " pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:41:30.234219 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.234196    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f1abe079-27f1-4078-a08c-406b881a7bde-ca-trust-extracted\") pod \"image-registry-8fc56db65-5rbz9\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") " pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:41:30.234219 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.234217    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f49e184-c6c2-4ffe-bccb-681321fa052a-serving-cert\") pod \"service-ca-operator-cf7c776b7-2n2dw\" (UID: \"6f49e184-c6c2-4ffe-bccb-681321fa052a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2n2dw"
Sep 16 18:41:30.234379 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.234238    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f49e184-c6c2-4ffe-bccb-681321fa052a-config\") pod \"service-ca-operator-cf7c776b7-2n2dw\" (UID: \"6f49e184-c6c2-4ffe-bccb-681321fa052a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2n2dw"
Sep 16 18:41:30.234379 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.234253    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f1abe079-27f1-4078-a08c-406b881a7bde-registry-certificates\") pod \"image-registry-8fc56db65-5rbz9\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") " pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:41:30.234379 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.234272    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f1abe079-27f1-4078-a08c-406b881a7bde-trusted-ca\") pod \"image-registry-8fc56db65-5rbz9\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") " pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:41:30.234379 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.234313    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjp2d\" (UniqueName: \"kubernetes.io/projected/aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b-kube-api-access-bjp2d\") pod \"cluster-monitoring-operator-6cd954b897-cjpnz\" (UID: \"aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cjpnz"
Sep 16 18:41:30.234379 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.234342    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2260af00-cdea-4591-9ef1-cf8631deb0fc-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xfz8z\" (UID: \"2260af00-cdea-4591-9ef1-cf8631deb0fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xfz8z"
Sep 16 18:41:30.234379 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.234368    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f1abe079-27f1-4078-a08c-406b881a7bde-installation-pull-secrets\") pod \"image-registry-8fc56db65-5rbz9\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") " pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:41:30.234619 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.234383    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-cjpnz\" (UID: \"aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cjpnz"
Sep 16 18:41:30.234619 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.234404    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2260af00-cdea-4591-9ef1-cf8631deb0fc-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xfz8z\" (UID: \"2260af00-cdea-4591-9ef1-cf8631deb0fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xfz8z"
Sep 16 18:41:30.234619 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.234462    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7v65\" (UniqueName: \"kubernetes.io/projected/2260af00-cdea-4591-9ef1-cf8631deb0fc-kube-api-access-g7v65\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xfz8z\" (UID: \"2260af00-cdea-4591-9ef1-cf8631deb0fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xfz8z"
Sep 16 18:41:30.234619 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.234485    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-registry-tls\") pod \"image-registry-8fc56db65-5rbz9\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") " pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:41:30.234619 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.234501    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-cjpnz\" (UID: \"aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cjpnz"
Sep 16 18:41:30.234619 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.234532    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d2fb\" (UniqueName: \"kubernetes.io/projected/6f49e184-c6c2-4ffe-bccb-681321fa052a-kube-api-access-5d2fb\") pod \"service-ca-operator-cf7c776b7-2n2dw\" (UID: \"6f49e184-c6c2-4ffe-bccb-681321fa052a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2n2dw"
Sep 16 18:41:30.234619 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.234601    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-bound-sa-token\") pod \"image-registry-8fc56db65-5rbz9\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") " pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:41:30.234888 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.234636    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7chl\" (UniqueName: \"kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-kube-api-access-b7chl\") pod \"image-registry-8fc56db65-5rbz9\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") " pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:41:30.234888 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.234694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-8zxkm\" (UID: \"7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8zxkm"
Sep 16 18:41:30.234888 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.234721    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-8zxkm\" (UID: \"7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8zxkm"
Sep 16 18:41:30.234888 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.234775    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f49e184-c6c2-4ffe-bccb-681321fa052a-config\") pod \"service-ca-operator-cf7c776b7-2n2dw\" (UID: \"6f49e184-c6c2-4ffe-bccb-681321fa052a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2n2dw"
Sep 16 18:41:30.237402 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.237383    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f49e184-c6c2-4ffe-bccb-681321fa052a-serving-cert\") pod \"service-ca-operator-cf7c776b7-2n2dw\" (UID: \"6f49e184-c6c2-4ffe-bccb-681321fa052a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2n2dw"
Sep 16 18:41:30.243274 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.243254    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d2fb\" (UniqueName: \"kubernetes.io/projected/6f49e184-c6c2-4ffe-bccb-681321fa052a-kube-api-access-5d2fb\") pod \"service-ca-operator-cf7c776b7-2n2dw\" (UID: \"6f49e184-c6c2-4ffe-bccb-681321fa052a\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2n2dw"
Sep 16 18:41:30.282264 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.282232    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2n2dw"
Sep 16 18:41:30.335309 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.335287    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2260af00-cdea-4591-9ef1-cf8631deb0fc-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xfz8z\" (UID: \"2260af00-cdea-4591-9ef1-cf8631deb0fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xfz8z"
Sep 16 18:41:30.335412 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.335322    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7v65\" (UniqueName: \"kubernetes.io/projected/2260af00-cdea-4591-9ef1-cf8631deb0fc-kube-api-access-g7v65\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xfz8z\" (UID: \"2260af00-cdea-4591-9ef1-cf8631deb0fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xfz8z"
Sep 16 18:41:30.335412 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.335342    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-registry-tls\") pod \"image-registry-8fc56db65-5rbz9\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") " pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:41:30.335412 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.335359    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-cjpnz\" (UID: \"aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cjpnz"
Sep 16 18:41:30.335538 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:30.335460    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:30.335538 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:30.335481    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8fc56db65-5rbz9: secret "image-registry-tls" not found
Sep 16 18:41:30.335538 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.335482    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-bound-sa-token\") pod \"image-registry-8fc56db65-5rbz9\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") " pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:41:30.335538 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.335514    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7chl\" (UniqueName: \"kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-kube-api-access-b7chl\") pod \"image-registry-8fc56db65-5rbz9\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") " pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:41:30.335848 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:30.335543    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-registry-tls podName:f1abe079-27f1-4078-a08c-406b881a7bde nodeName:}" failed. No retries permitted until 2026-09-16 18:41:30.835524137 +0000 UTC m=+59.070457572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-registry-tls") pod "image-registry-8fc56db65-5rbz9" (UID: "f1abe079-27f1-4078-a08c-406b881a7bde") : secret "image-registry-tls" not found
Sep 16 18:41:30.335848 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.335594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-8zxkm\" (UID: \"7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8zxkm"
Sep 16 18:41:30.335848 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.335636    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-8zxkm\" (UID: \"7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8zxkm"
Sep 16 18:41:30.335848 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.335679    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f1abe079-27f1-4078-a08c-406b881a7bde-image-registry-private-configuration\") pod \"image-registry-8fc56db65-5rbz9\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") " pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:41:30.335848 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.335703    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f1abe079-27f1-4078-a08c-406b881a7bde-ca-trust-extracted\") pod \"image-registry-8fc56db65-5rbz9\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") " pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:41:30.335848 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.335747    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f1abe079-27f1-4078-a08c-406b881a7bde-registry-certificates\") pod \"image-registry-8fc56db65-5rbz9\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") " pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:41:30.335848 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.335782    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f1abe079-27f1-4078-a08c-406b881a7bde-trusted-ca\") pod \"image-registry-8fc56db65-5rbz9\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") " pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:41:30.335848 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.335805    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjp2d\" (UniqueName: \"kubernetes.io/projected/aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b-kube-api-access-bjp2d\") pod \"cluster-monitoring-operator-6cd954b897-cjpnz\" (UID: \"aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cjpnz"
Sep 16 18:41:30.335848 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.335832    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2260af00-cdea-4591-9ef1-cf8631deb0fc-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xfz8z\" (UID: \"2260af00-cdea-4591-9ef1-cf8631deb0fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xfz8z"
Sep 16 18:41:30.338023 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.335860    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f1abe079-27f1-4078-a08c-406b881a7bde-installation-pull-secrets\") pod \"image-registry-8fc56db65-5rbz9\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") " pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:41:30.338023 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.335883    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-cjpnz\" (UID: \"aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cjpnz"
Sep 16 18:41:30.338023 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.335933    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2260af00-cdea-4591-9ef1-cf8631deb0fc-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xfz8z\" (UID: \"2260af00-cdea-4591-9ef1-cf8631deb0fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xfz8z"
Sep 16 18:41:30.338023 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:30.335983    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:41:30.338023 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:30.336026    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:41:30.338023 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:30.336033    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b-cluster-monitoring-operator-tls podName:aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b nodeName:}" failed. No retries permitted until 2026-09-16 18:41:30.83601615 +0000 UTC m=+59.070949575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-cjpnz" (UID: "aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:41:30.338023 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:30.336136    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4-networking-console-plugin-cert podName:7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:30.836119207 +0000 UTC m=+59.071052637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-8zxkm" (UID: "7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4") : secret "networking-console-plugin-cert" not found
Sep 16 18:41:30.338023 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.336104    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-cjpnz\" (UID: \"aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cjpnz"
Sep 16 18:41:30.338023 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.336401    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f1abe079-27f1-4078-a08c-406b881a7bde-registry-certificates\") pod \"image-registry-8fc56db65-5rbz9\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") " pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:41:30.338023 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.336405    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-8zxkm\" (UID: \"7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8zxkm"
Sep 16 18:41:30.338023 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.336607    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f1abe079-27f1-4078-a08c-406b881a7bde-ca-trust-extracted\") pod \"image-registry-8fc56db65-5rbz9\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") " pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:41:30.338023 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.336943    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f1abe079-27f1-4078-a08c-406b881a7bde-trusted-ca\") pod \"image-registry-8fc56db65-5rbz9\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") " pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:41:30.338946 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.338924    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2260af00-cdea-4591-9ef1-cf8631deb0fc-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xfz8z\" (UID: \"2260af00-cdea-4591-9ef1-cf8631deb0fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xfz8z"
Sep 16 18:41:30.339325 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.339305    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f1abe079-27f1-4078-a08c-406b881a7bde-image-registry-private-configuration\") pod \"image-registry-8fc56db65-5rbz9\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") " pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:41:30.339382 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.339357    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f1abe079-27f1-4078-a08c-406b881a7bde-installation-pull-secrets\") pod \"image-registry-8fc56db65-5rbz9\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") " pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:41:30.357984 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.357958    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjp2d\" (UniqueName: \"kubernetes.io/projected/aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b-kube-api-access-bjp2d\") pod \"cluster-monitoring-operator-6cd954b897-cjpnz\" (UID: \"aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cjpnz"
Sep 16 18:41:30.358309 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.358263    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7chl\" (UniqueName: \"kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-kube-api-access-b7chl\") pod \"image-registry-8fc56db65-5rbz9\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") " pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:41:30.358799 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.358771    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7v65\" (UniqueName: \"kubernetes.io/projected/2260af00-cdea-4591-9ef1-cf8631deb0fc-kube-api-access-g7v65\") pod \"kube-storage-version-migrator-operator-6c8f84f897-xfz8z\" (UID: \"2260af00-cdea-4591-9ef1-cf8631deb0fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xfz8z"
Sep 16 18:41:30.359409 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.359376    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-bound-sa-token\") pod \"image-registry-8fc56db65-5rbz9\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") " pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:41:30.452672 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.452648    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xfz8z"
Sep 16 18:41:30.576015 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.575986    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-2n2dw"]
Sep 16 18:41:30.578782 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:41:30.578757    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f49e184_c6c2_4ffe_bccb_681321fa052a.slice/crio-afdef9f951555a53ea05e5f46f693ff24cb6987451bc047690059241cbb4c507 WatchSource:0}: Error finding container afdef9f951555a53ea05e5f46f693ff24cb6987451bc047690059241cbb4c507: Status 404 returned error can't find the container with id afdef9f951555a53ea05e5f46f693ff24cb6987451bc047690059241cbb4c507
Sep 16 18:41:30.758170 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.758117    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xfz8z"]
Sep 16 18:41:30.761182 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:41:30.761158    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2260af00_cdea_4591_9ef1_cf8631deb0fc.slice/crio-3ad545cd846228edd4cb9cbca793812f8d8dba9866aaddae01da3398ea75c94b WatchSource:0}: Error finding container 3ad545cd846228edd4cb9cbca793812f8d8dba9866aaddae01da3398ea75c94b: Status 404 returned error can't find the container with id 3ad545cd846228edd4cb9cbca793812f8d8dba9866aaddae01da3398ea75c94b
Sep 16 18:41:30.840297 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.840269    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-registry-tls\") pod \"image-registry-8fc56db65-5rbz9\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") " pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:41:30.840399 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.840328    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-8zxkm\" (UID: \"7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8zxkm"
Sep 16 18:41:30.840399 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:30.840391    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-cjpnz\" (UID: \"aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cjpnz"
Sep 16 18:41:30.840512 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:30.840412    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:30.840512 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:30.840429    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8fc56db65-5rbz9: secret "image-registry-tls" not found
Sep 16 18:41:30.840512 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:30.840440    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:41:30.840512 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:30.840488    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-registry-tls podName:f1abe079-27f1-4078-a08c-406b881a7bde nodeName:}" failed. No retries permitted until 2026-09-16 18:41:31.840473592 +0000 UTC m=+60.075407023 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-registry-tls") pod "image-registry-8fc56db65-5rbz9" (UID: "f1abe079-27f1-4078-a08c-406b881a7bde") : secret "image-registry-tls" not found
Sep 16 18:41:30.840512 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:30.840502    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4-networking-console-plugin-cert podName:7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:31.840495845 +0000 UTC m=+60.075429267 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-8zxkm" (UID: "7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4") : secret "networking-console-plugin-cert" not found
Sep 16 18:41:30.840512 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:30.840502    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:41:30.840732 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:30.840557    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b-cluster-monitoring-operator-tls podName:aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b nodeName:}" failed. No retries permitted until 2026-09-16 18:41:31.84053827 +0000 UTC m=+60.075471695 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-cjpnz" (UID: "aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:41:31.502140 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:31.502089    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xfz8z" event={"ID":"2260af00-cdea-4591-9ef1-cf8631deb0fc","Type":"ContainerStarted","Data":"3ad545cd846228edd4cb9cbca793812f8d8dba9866aaddae01da3398ea75c94b"}
Sep 16 18:41:31.503994 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:31.503970    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2n2dw" event={"ID":"6f49e184-c6c2-4ffe-bccb-681321fa052a","Type":"ContainerStarted","Data":"afdef9f951555a53ea05e5f46f693ff24cb6987451bc047690059241cbb4c507"}
Sep 16 18:41:31.846520 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:31.846428    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-cjpnz\" (UID: \"aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cjpnz"
Sep 16 18:41:31.846801 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:31.846585    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:41:31.846801 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:31.846603    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-registry-tls\") pod \"image-registry-8fc56db65-5rbz9\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") " pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:41:31.846801 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:31.846644    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b-cluster-monitoring-operator-tls podName:aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b nodeName:}" failed. No retries permitted until 2026-09-16 18:41:33.846630777 +0000 UTC m=+62.081564204 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-cjpnz" (UID: "aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:41:31.846801 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:31.846690    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-8zxkm\" (UID: \"7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8zxkm"
Sep 16 18:41:31.846801 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:31.846710    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:31.846801 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:31.846725    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8fc56db65-5rbz9: secret "image-registry-tls" not found
Sep 16 18:41:31.846801 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:31.846772    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:41:31.847383 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:31.846776    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-registry-tls podName:f1abe079-27f1-4078-a08c-406b881a7bde nodeName:}" failed. No retries permitted until 2026-09-16 18:41:33.846762018 +0000 UTC m=+62.081695439 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-registry-tls") pod "image-registry-8fc56db65-5rbz9" (UID: "f1abe079-27f1-4078-a08c-406b881a7bde") : secret "image-registry-tls" not found
Sep 16 18:41:31.847383 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:31.846841    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4-networking-console-plugin-cert podName:7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:33.846826932 +0000 UTC m=+62.081760361 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-8zxkm" (UID: "7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4") : secret "networking-console-plugin-cert" not found
Sep 16 18:41:33.419796 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:33.419723    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lp55b"
Sep 16 18:41:33.511269 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:33.511226    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2n2dw" event={"ID":"6f49e184-c6c2-4ffe-bccb-681321fa052a","Type":"ContainerStarted","Data":"d7fa6774515ed01591a0fd53edb1835ccae5128277ad96335d63b7cc4ff1871a"}
Sep 16 18:41:33.513013 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:33.512987    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xfz8z" event={"ID":"2260af00-cdea-4591-9ef1-cf8631deb0fc","Type":"ContainerStarted","Data":"d09b66b1a20056fdfbd401f9566f9a8addc372fb14b42f25665074a167e579f3"}
Sep 16 18:41:33.527916 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:33.527870    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2n2dw" podStartSLOduration=2.1524242080000002 podStartE2EDuration="4.527853148s" totalImagesPullingTime="2.37542894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:33.527271968 +0000 UTC m=+61.762205390" watchObservedRunningTime="2026-09-16 18:41:33.527853148 +0000 UTC m=+61.762786593"
Sep 16 18:41:33.553783 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:33.553731    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xfz8z" podStartSLOduration=1.35764039 podStartE2EDuration="3.55371358s" totalImagesPullingTime="2.19607319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:33.553202376 +0000 UTC m=+61.788135821" watchObservedRunningTime="2026-09-16 18:41:33.55371358 +0000 UTC m=+61.788647025"
Sep 16 18:41:33.861210 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:33.861179    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-8zxkm\" (UID: \"7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8zxkm"
Sep 16 18:41:33.861362 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:33.861230    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-cjpnz\" (UID: \"aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cjpnz"
Sep 16 18:41:33.861362 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:33.861302    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:41:33.861362 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:33.861323    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:41:33.861362 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:33.861354    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-registry-tls\") pod \"image-registry-8fc56db65-5rbz9\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") " pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:41:33.861514 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:33.861364    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b-cluster-monitoring-operator-tls podName:aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b nodeName:}" failed. No retries permitted until 2026-09-16 18:41:37.861346174 +0000 UTC m=+66.096279617 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-cjpnz" (UID: "aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:41:33.861514 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:33.861403    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4-networking-console-plugin-cert podName:7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:37.861392106 +0000 UTC m=+66.096325543 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-8zxkm" (UID: "7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4") : secret "networking-console-plugin-cert" not found
Sep 16 18:41:33.861514 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:33.861435    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:33.861514 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:33.861447    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8fc56db65-5rbz9: secret "image-registry-tls" not found
Sep 16 18:41:33.861514 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:33.861494    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-registry-tls podName:f1abe079-27f1-4078-a08c-406b881a7bde nodeName:}" failed. No retries permitted until 2026-09-16 18:41:37.861475396 +0000 UTC m=+66.096408825 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-registry-tls") pod "image-registry-8fc56db65-5rbz9" (UID: "f1abe079-27f1-4078-a08c-406b881a7bde") : secret "image-registry-tls" not found
Sep 16 18:41:33.986590 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:33.986556    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sj4ck"]
Sep 16 18:41:33.988970 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:33.988951    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sj4ck"
Sep 16 18:41:33.991652 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:33.991633    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kxlhk\""
Sep 16 18:41:33.991771 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:33.991663    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:41:33.991771 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:33.991677    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:41:33.991884 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:33.991801    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:41:33.992034 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:33.992014    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:41:34.016761 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:34.016737    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sj4ck"]
Sep 16 18:41:34.163867 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:34.163807    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f1156b17-5b7a-48e4-aac5-d0a527142061-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sj4ck\" (UID: \"f1156b17-5b7a-48e4-aac5-d0a527142061\") " pod="openshift-insights/insights-runtime-extractor-sj4ck"
Sep 16 18:41:34.163867 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:34.163861    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f1156b17-5b7a-48e4-aac5-d0a527142061-crio-socket\") pod \"insights-runtime-extractor-sj4ck\" (UID: \"f1156b17-5b7a-48e4-aac5-d0a527142061\") " pod="openshift-insights/insights-runtime-extractor-sj4ck"
Sep 16 18:41:34.163991 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:34.163895    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64x4f\" (UniqueName: \"kubernetes.io/projected/f1156b17-5b7a-48e4-aac5-d0a527142061-kube-api-access-64x4f\") pod \"insights-runtime-extractor-sj4ck\" (UID: \"f1156b17-5b7a-48e4-aac5-d0a527142061\") " pod="openshift-insights/insights-runtime-extractor-sj4ck"
Sep 16 18:41:34.163991 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:34.163936    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f1156b17-5b7a-48e4-aac5-d0a527142061-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sj4ck\" (UID: \"f1156b17-5b7a-48e4-aac5-d0a527142061\") " pod="openshift-insights/insights-runtime-extractor-sj4ck"
Sep 16 18:41:34.164094 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:34.164070    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f1156b17-5b7a-48e4-aac5-d0a527142061-data-volume\") pod \"insights-runtime-extractor-sj4ck\" (UID: \"f1156b17-5b7a-48e4-aac5-d0a527142061\") " pod="openshift-insights/insights-runtime-extractor-sj4ck"
Sep 16 18:41:34.265357 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:34.265332    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f1156b17-5b7a-48e4-aac5-d0a527142061-crio-socket\") pod \"insights-runtime-extractor-sj4ck\" (UID: \"f1156b17-5b7a-48e4-aac5-d0a527142061\") " pod="openshift-insights/insights-runtime-extractor-sj4ck"
Sep 16 18:41:34.265453 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:34.265380    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64x4f\" (UniqueName: \"kubernetes.io/projected/f1156b17-5b7a-48e4-aac5-d0a527142061-kube-api-access-64x4f\") pod \"insights-runtime-extractor-sj4ck\" (UID: \"f1156b17-5b7a-48e4-aac5-d0a527142061\") " pod="openshift-insights/insights-runtime-extractor-sj4ck"
Sep 16 18:41:34.265453 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:34.265425    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f1156b17-5b7a-48e4-aac5-d0a527142061-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sj4ck\" (UID: \"f1156b17-5b7a-48e4-aac5-d0a527142061\") " pod="openshift-insights/insights-runtime-extractor-sj4ck"
Sep 16 18:41:34.265560 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:34.265458    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f1156b17-5b7a-48e4-aac5-d0a527142061-crio-socket\") pod \"insights-runtime-extractor-sj4ck\" (UID: \"f1156b17-5b7a-48e4-aac5-d0a527142061\") " pod="openshift-insights/insights-runtime-extractor-sj4ck"
Sep 16 18:41:34.265560 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:34.265490    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f1156b17-5b7a-48e4-aac5-d0a527142061-data-volume\") pod \"insights-runtime-extractor-sj4ck\" (UID: \"f1156b17-5b7a-48e4-aac5-d0a527142061\") " pod="openshift-insights/insights-runtime-extractor-sj4ck"
Sep 16 18:41:34.265560 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:34.265535    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:34.265560 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:34.265548    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f1156b17-5b7a-48e4-aac5-d0a527142061-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sj4ck\" (UID: \"f1156b17-5b7a-48e4-aac5-d0a527142061\") " pod="openshift-insights/insights-runtime-extractor-sj4ck"
Sep 16 18:41:34.265757 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:34.265600    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f1156b17-5b7a-48e4-aac5-d0a527142061-kube-rbac-proxy-cm podName:f1156b17-5b7a-48e4-aac5-d0a527142061 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:34.765579954 +0000 UTC m=+63.000513381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f1156b17-5b7a-48e4-aac5-d0a527142061-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sj4ck" (UID: "f1156b17-5b7a-48e4-aac5-d0a527142061") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:34.265757 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:34.265656    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:34.265757 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:34.265704    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1156b17-5b7a-48e4-aac5-d0a527142061-insights-runtime-extractor-tls podName:f1156b17-5b7a-48e4-aac5-d0a527142061 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:34.76568892 +0000 UTC m=+63.000622348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f1156b17-5b7a-48e4-aac5-d0a527142061-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sj4ck" (UID: "f1156b17-5b7a-48e4-aac5-d0a527142061") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:34.265862 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:34.265822    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f1156b17-5b7a-48e4-aac5-d0a527142061-data-volume\") pod \"insights-runtime-extractor-sj4ck\" (UID: \"f1156b17-5b7a-48e4-aac5-d0a527142061\") " pod="openshift-insights/insights-runtime-extractor-sj4ck"
Sep 16 18:41:34.275586 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:34.275562    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64x4f\" (UniqueName: \"kubernetes.io/projected/f1156b17-5b7a-48e4-aac5-d0a527142061-kube-api-access-64x4f\") pod \"insights-runtime-extractor-sj4ck\" (UID: \"f1156b17-5b7a-48e4-aac5-d0a527142061\") " pod="openshift-insights/insights-runtime-extractor-sj4ck"
Sep 16 18:41:34.769748 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:34.769715    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f1156b17-5b7a-48e4-aac5-d0a527142061-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sj4ck\" (UID: \"f1156b17-5b7a-48e4-aac5-d0a527142061\") " pod="openshift-insights/insights-runtime-extractor-sj4ck"
Sep 16 18:41:34.770184 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:34.769774    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f1156b17-5b7a-48e4-aac5-d0a527142061-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sj4ck\" (UID: \"f1156b17-5b7a-48e4-aac5-d0a527142061\") " pod="openshift-insights/insights-runtime-extractor-sj4ck"
Sep 16 18:41:34.770184 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:34.769842    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:34.770184 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:34.769860    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:34.770184 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:34.769907    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1156b17-5b7a-48e4-aac5-d0a527142061-insights-runtime-extractor-tls podName:f1156b17-5b7a-48e4-aac5-d0a527142061 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:35.769892146 +0000 UTC m=+64.004825568 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f1156b17-5b7a-48e4-aac5-d0a527142061-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sj4ck" (UID: "f1156b17-5b7a-48e4-aac5-d0a527142061") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:34.770184 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:34.769921    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f1156b17-5b7a-48e4-aac5-d0a527142061-kube-rbac-proxy-cm podName:f1156b17-5b7a-48e4-aac5-d0a527142061 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:35.769915086 +0000 UTC m=+64.004848508 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f1156b17-5b7a-48e4-aac5-d0a527142061-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sj4ck" (UID: "f1156b17-5b7a-48e4-aac5-d0a527142061") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:35.777456 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:35.777416    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f1156b17-5b7a-48e4-aac5-d0a527142061-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sj4ck\" (UID: \"f1156b17-5b7a-48e4-aac5-d0a527142061\") " pod="openshift-insights/insights-runtime-extractor-sj4ck"
Sep 16 18:41:35.777872 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:35.777494    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f1156b17-5b7a-48e4-aac5-d0a527142061-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sj4ck\" (UID: \"f1156b17-5b7a-48e4-aac5-d0a527142061\") " pod="openshift-insights/insights-runtime-extractor-sj4ck"
Sep 16 18:41:35.777872 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:35.777533    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:35.777872 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:35.777590    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f1156b17-5b7a-48e4-aac5-d0a527142061-kube-rbac-proxy-cm podName:f1156b17-5b7a-48e4-aac5-d0a527142061 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:37.777576845 +0000 UTC m=+66.012510271 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f1156b17-5b7a-48e4-aac5-d0a527142061-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sj4ck" (UID: "f1156b17-5b7a-48e4-aac5-d0a527142061") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:35.777872 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:35.777598    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:35.777872 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:35.777636    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1156b17-5b7a-48e4-aac5-d0a527142061-insights-runtime-extractor-tls podName:f1156b17-5b7a-48e4-aac5-d0a527142061 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:37.777625256 +0000 UTC m=+66.012558678 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f1156b17-5b7a-48e4-aac5-d0a527142061-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sj4ck" (UID: "f1156b17-5b7a-48e4-aac5-d0a527142061") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:36.069465 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:36.069390    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xbtsz_c3e865df-22be-4129-bbf3-300285c29e69/dns-node-resolver/0.log"
Sep 16 18:41:36.870598 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:36.870569    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-95lgg_6ab1b46d-5a59-42a7-87c4-13c16da817ad/node-ca/0.log"
Sep 16 18:41:36.986918 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:36.986887    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/309dbcdf-d468-4001-805b-e71bb5bdd775-metrics-tls\") pod \"dns-default-86bz5\" (UID: \"309dbcdf-d468-4001-805b-e71bb5bdd775\") " pod="openshift-dns/dns-default-86bz5"
Sep 16 18:41:36.987066 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:36.987028    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:36.987108 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:36.987102    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/309dbcdf-d468-4001-805b-e71bb5bdd775-metrics-tls podName:309dbcdf-d468-4001-805b-e71bb5bdd775 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:08.987088104 +0000 UTC m=+97.222021525 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/309dbcdf-d468-4001-805b-e71bb5bdd775-metrics-tls") pod "dns-default-86bz5" (UID: "309dbcdf-d468-4001-805b-e71bb5bdd775") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:37.793931 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:37.793896    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f1156b17-5b7a-48e4-aac5-d0a527142061-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sj4ck\" (UID: \"f1156b17-5b7a-48e4-aac5-d0a527142061\") " pod="openshift-insights/insights-runtime-extractor-sj4ck"
Sep 16 18:41:37.794098 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:37.793971    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f1156b17-5b7a-48e4-aac5-d0a527142061-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sj4ck\" (UID: \"f1156b17-5b7a-48e4-aac5-d0a527142061\") " pod="openshift-insights/insights-runtime-extractor-sj4ck"
Sep 16 18:41:37.794098 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:37.794014    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:37.794098 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:37.794083    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:37.794214 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:37.794111    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f1156b17-5b7a-48e4-aac5-d0a527142061-kube-rbac-proxy-cm podName:f1156b17-5b7a-48e4-aac5-d0a527142061 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:41.794096488 +0000 UTC m=+70.029029914 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f1156b17-5b7a-48e4-aac5-d0a527142061-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sj4ck" (UID: "f1156b17-5b7a-48e4-aac5-d0a527142061") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:37.794214 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:37.794135    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1156b17-5b7a-48e4-aac5-d0a527142061-insights-runtime-extractor-tls podName:f1156b17-5b7a-48e4-aac5-d0a527142061 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:41.794124007 +0000 UTC m=+70.029057443 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f1156b17-5b7a-48e4-aac5-d0a527142061-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sj4ck" (UID: "f1156b17-5b7a-48e4-aac5-d0a527142061") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:37.870090 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:37.870065    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-scmml_5dc0ed37-07b5-439b-94cf-1cf3104ad233/serve-healthcheck-canary/0.log"
Sep 16 18:41:37.894322 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:37.894298    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-8zxkm\" (UID: \"7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8zxkm"
Sep 16 18:41:37.894625 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:37.894347    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-cjpnz\" (UID: \"aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cjpnz"
Sep 16 18:41:37.894625 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:37.894419    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:41:37.894625 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:37.894420    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:41:37.894625 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:37.894453    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b-cluster-monitoring-operator-tls podName:aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b nodeName:}" failed. No retries permitted until 2026-09-16 18:41:45.89444338 +0000 UTC m=+74.129376802 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-cjpnz" (UID: "aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:41:37.894625 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:37.894465    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/733b59e1-b843-4255-bf37-71c81c244989-metrics-certs\") pod \"network-metrics-daemon-xq67c\" (UID: \"733b59e1-b843-4255-bf37-71c81c244989\") " pod="openshift-multus/network-metrics-daemon-xq67c"
Sep 16 18:41:37.894625 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:37.894474    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4-networking-console-plugin-cert podName:7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:45.894461759 +0000 UTC m=+74.129395181 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-8zxkm" (UID: "7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4") : secret "networking-console-plugin-cert" not found
Sep 16 18:41:37.894625 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:37.894501    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:41:37.894625 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:37.894522    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/733b59e1-b843-4255-bf37-71c81c244989-metrics-certs podName:733b59e1-b843-4255-bf37-71c81c244989 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:41.894514782 +0000 UTC m=+130.129448206 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/733b59e1-b843-4255-bf37-71c81c244989-metrics-certs") pod "network-metrics-daemon-xq67c" (UID: "733b59e1-b843-4255-bf37-71c81c244989") : secret "metrics-daemon-secret" not found
Sep 16 18:41:37.894625 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:37.894499    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-registry-tls\") pod \"image-registry-8fc56db65-5rbz9\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") " pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:41:37.894625 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:37.894545    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:37.894625 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:37.894554    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8fc56db65-5rbz9: secret "image-registry-tls" not found
Sep 16 18:41:37.894625 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:37.894583    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-registry-tls podName:f1abe079-27f1-4078-a08c-406b881a7bde nodeName:}" failed. No retries permitted until 2026-09-16 18:41:45.894574559 +0000 UTC m=+74.129507981 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-registry-tls") pod "image-registry-8fc56db65-5rbz9" (UID: "f1abe079-27f1-4078-a08c-406b881a7bde") : secret "image-registry-tls" not found
Sep 16 18:41:41.820162 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:41.820128    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f1156b17-5b7a-48e4-aac5-d0a527142061-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sj4ck\" (UID: \"f1156b17-5b7a-48e4-aac5-d0a527142061\") " pod="openshift-insights/insights-runtime-extractor-sj4ck"
Sep 16 18:41:41.820553 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:41.820186    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f1156b17-5b7a-48e4-aac5-d0a527142061-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sj4ck\" (UID: \"f1156b17-5b7a-48e4-aac5-d0a527142061\") " pod="openshift-insights/insights-runtime-extractor-sj4ck"
Sep 16 18:41:41.820553 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:41.820274    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:41.820553 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:41.820288    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:41.820553 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:41.820334    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f1156b17-5b7a-48e4-aac5-d0a527142061-kube-rbac-proxy-cm podName:f1156b17-5b7a-48e4-aac5-d0a527142061 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:49.820321208 +0000 UTC m=+78.055254630 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f1156b17-5b7a-48e4-aac5-d0a527142061-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sj4ck" (UID: "f1156b17-5b7a-48e4-aac5-d0a527142061") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:41.820553 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:41.820348    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1156b17-5b7a-48e4-aac5-d0a527142061-insights-runtime-extractor-tls podName:f1156b17-5b7a-48e4-aac5-d0a527142061 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:49.820341851 +0000 UTC m=+78.055275273 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f1156b17-5b7a-48e4-aac5-d0a527142061-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sj4ck" (UID: "f1156b17-5b7a-48e4-aac5-d0a527142061") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:43.463163 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:43.463134    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-djht2"
Sep 16 18:41:45.948497 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:45.948462    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-8zxkm\" (UID: \"7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8zxkm"
Sep 16 18:41:45.948852 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:45.948521    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-cjpnz\" (UID: \"aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cjpnz"
Sep 16 18:41:45.948852 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:45.948549    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-registry-tls\") pod \"image-registry-8fc56db65-5rbz9\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") " pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:41:45.948852 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:45.948600    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 18:41:45.948852 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:45.948663    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:41:45.948852 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:45.948670    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4-networking-console-plugin-cert podName:7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:01.948653832 +0000 UTC m=+90.183587273 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-8zxkm" (UID: "7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4") : secret "networking-console-plugin-cert" not found
Sep 16 18:41:45.948852 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:45.948733    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b-cluster-monitoring-operator-tls podName:aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b nodeName:}" failed. No retries permitted until 2026-09-16 18:42:01.948718668 +0000 UTC m=+90.183652090 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-cjpnz" (UID: "aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:41:45.950847 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:45.950819    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-registry-tls\") pod \"image-registry-8fc56db65-5rbz9\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") " pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:41:46.013622 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:46.013597    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xjxjk\""
Sep 16 18:41:46.021804 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:46.021784    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:41:46.305618 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:46.305547    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8fc56db65-5rbz9"]
Sep 16 18:41:46.309556 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:41:46.309528    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1abe079_27f1_4078_a08c_406b881a7bde.slice/crio-143efc13cc1cda370011b8e57adebcbe8669533f57a1fe3b0445a318f999b5c7 WatchSource:0}: Error finding container 143efc13cc1cda370011b8e57adebcbe8669533f57a1fe3b0445a318f999b5c7: Status 404 returned error can't find the container with id 143efc13cc1cda370011b8e57adebcbe8669533f57a1fe3b0445a318f999b5c7
Sep 16 18:41:46.550521 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:46.550478    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8fc56db65-5rbz9" event={"ID":"f1abe079-27f1-4078-a08c-406b881a7bde","Type":"ContainerStarted","Data":"75fc09651c26e0e54630f6e62a92319c0eef6b369c8f16059c1d71f9452be852"}
Sep 16 18:41:46.550709 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:46.550559    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8fc56db65-5rbz9" event={"ID":"f1abe079-27f1-4078-a08c-406b881a7bde","Type":"ContainerStarted","Data":"143efc13cc1cda370011b8e57adebcbe8669533f57a1fe3b0445a318f999b5c7"}
Sep 16 18:41:46.550709 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:46.550622    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:41:46.577941 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:46.577833    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8fc56db65-5rbz9" podStartSLOduration=16.577815405 podStartE2EDuration="16.577815405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:46.577557294 +0000 UTC m=+74.812490739" watchObservedRunningTime="2026-09-16 18:41:46.577815405 +0000 UTC m=+74.812748850"
Sep 16 18:41:49.881732 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:49.881702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f1156b17-5b7a-48e4-aac5-d0a527142061-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sj4ck\" (UID: \"f1156b17-5b7a-48e4-aac5-d0a527142061\") " pod="openshift-insights/insights-runtime-extractor-sj4ck"
Sep 16 18:41:49.882127 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:49.881764    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f1156b17-5b7a-48e4-aac5-d0a527142061-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sj4ck\" (UID: \"f1156b17-5b7a-48e4-aac5-d0a527142061\") " pod="openshift-insights/insights-runtime-extractor-sj4ck"
Sep 16 18:41:49.882127 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:49.881855    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:49.882127 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:41:49.881912    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1156b17-5b7a-48e4-aac5-d0a527142061-insights-runtime-extractor-tls podName:f1156b17-5b7a-48e4-aac5-d0a527142061 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:05.8818995 +0000 UTC m=+94.116832922 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f1156b17-5b7a-48e4-aac5-d0a527142061-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sj4ck" (UID: "f1156b17-5b7a-48e4-aac5-d0a527142061") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:49.882253 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:41:49.882183    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f1156b17-5b7a-48e4-aac5-d0a527142061-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sj4ck\" (UID: \"f1156b17-5b7a-48e4-aac5-d0a527142061\") " pod="openshift-insights/insights-runtime-extractor-sj4ck"
Sep 16 18:42:01.672696 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.672581    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c48874bf-p26bd"]
Sep 16 18:42:01.674972 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.674952    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c48874bf-p26bd"
Sep 16 18:42:01.677899 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.677880    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 18:42:01.679011 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.678986    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 16 18:42:01.679137 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.679009    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 16 18:42:01.679137 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.679077    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 16 18:42:01.679137 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.678999    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 18:42:01.679323 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.679157    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 16 18:42:01.679406 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.679392    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 18:42:01.708358 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.708333    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c48874bf-p26bd"]
Sep 16 18:42:01.723821 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.723798    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8fc56db65-5rbz9"]
Sep 16 18:42:01.729753 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.729730    2578 patch_prober.go:28] interesting pod/image-registry-8fc56db65-5rbz9 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 18:42:01.729831 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.729778    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-8fc56db65-5rbz9" podUID="f1abe079-27f1-4078-a08c-406b881a7bde" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:42:01.761024 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.760999    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cdca9b2f-4186-4af4-9644-ea8ddfd6440c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85c48874bf-p26bd\" (UID: \"cdca9b2f-4186-4af4-9644-ea8ddfd6440c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c48874bf-p26bd"
Sep 16 18:42:01.761125 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.761027    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cdca9b2f-4186-4af4-9644-ea8ddfd6440c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85c48874bf-p26bd\" (UID: \"cdca9b2f-4186-4af4-9644-ea8ddfd6440c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c48874bf-p26bd"
Sep 16 18:42:01.761125 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.761076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdca9b2f-4186-4af4-9644-ea8ddfd6440c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85c48874bf-p26bd\" (UID: \"cdca9b2f-4186-4af4-9644-ea8ddfd6440c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c48874bf-p26bd"
Sep 16 18:42:01.761205 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.761126    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48s42\" (UniqueName: \"kubernetes.io/projected/cdca9b2f-4186-4af4-9644-ea8ddfd6440c-kube-api-access-48s42\") pod \"cluster-proxy-proxy-agent-85c48874bf-p26bd\" (UID: \"cdca9b2f-4186-4af4-9644-ea8ddfd6440c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c48874bf-p26bd"
Sep 16 18:42:01.761205 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.761156    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cdca9b2f-4186-4af4-9644-ea8ddfd6440c-ca\") pod \"cluster-proxy-proxy-agent-85c48874bf-p26bd\" (UID: \"cdca9b2f-4186-4af4-9644-ea8ddfd6440c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c48874bf-p26bd"
Sep 16 18:42:01.761205 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.761175    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cdca9b2f-4186-4af4-9644-ea8ddfd6440c-hub\") pod \"cluster-proxy-proxy-agent-85c48874bf-p26bd\" (UID: \"cdca9b2f-4186-4af4-9644-ea8ddfd6440c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c48874bf-p26bd"
Sep 16 18:42:01.861763 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.861739    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cdca9b2f-4186-4af4-9644-ea8ddfd6440c-ca\") pod \"cluster-proxy-proxy-agent-85c48874bf-p26bd\" (UID: \"cdca9b2f-4186-4af4-9644-ea8ddfd6440c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c48874bf-p26bd"
Sep 16 18:42:01.861895 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.861773    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cdca9b2f-4186-4af4-9644-ea8ddfd6440c-hub\") pod \"cluster-proxy-proxy-agent-85c48874bf-p26bd\" (UID: \"cdca9b2f-4186-4af4-9644-ea8ddfd6440c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c48874bf-p26bd"
Sep 16 18:42:01.861895 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.861833    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cdca9b2f-4186-4af4-9644-ea8ddfd6440c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85c48874bf-p26bd\" (UID: \"cdca9b2f-4186-4af4-9644-ea8ddfd6440c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c48874bf-p26bd"
Sep 16 18:42:01.861895 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.861860    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cdca9b2f-4186-4af4-9644-ea8ddfd6440c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85c48874bf-p26bd\" (UID: \"cdca9b2f-4186-4af4-9644-ea8ddfd6440c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c48874bf-p26bd"
Sep 16 18:42:01.862072 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.862028    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdca9b2f-4186-4af4-9644-ea8ddfd6440c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85c48874bf-p26bd\" (UID: \"cdca9b2f-4186-4af4-9644-ea8ddfd6440c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c48874bf-p26bd"
Sep 16 18:42:01.862134 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.862084    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48s42\" (UniqueName: \"kubernetes.io/projected/cdca9b2f-4186-4af4-9644-ea8ddfd6440c-kube-api-access-48s42\") pod \"cluster-proxy-proxy-agent-85c48874bf-p26bd\" (UID: \"cdca9b2f-4186-4af4-9644-ea8ddfd6440c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c48874bf-p26bd"
Sep 16 18:42:01.862811 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.862788    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdca9b2f-4186-4af4-9644-ea8ddfd6440c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85c48874bf-p26bd\" (UID: \"cdca9b2f-4186-4af4-9644-ea8ddfd6440c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c48874bf-p26bd"
Sep 16 18:42:01.864355 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.864332    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cdca9b2f-4186-4af4-9644-ea8ddfd6440c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85c48874bf-p26bd\" (UID: \"cdca9b2f-4186-4af4-9644-ea8ddfd6440c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c48874bf-p26bd"
Sep 16 18:42:01.864466 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.864445    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cdca9b2f-4186-4af4-9644-ea8ddfd6440c-hub\") pod \"cluster-proxy-proxy-agent-85c48874bf-p26bd\" (UID: \"cdca9b2f-4186-4af4-9644-ea8ddfd6440c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c48874bf-p26bd"
Sep 16 18:42:01.864643 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.864624    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cdca9b2f-4186-4af4-9644-ea8ddfd6440c-ca\") pod \"cluster-proxy-proxy-agent-85c48874bf-p26bd\" (UID: \"cdca9b2f-4186-4af4-9644-ea8ddfd6440c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c48874bf-p26bd"
Sep 16 18:42:01.864725 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.864710    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cdca9b2f-4186-4af4-9644-ea8ddfd6440c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85c48874bf-p26bd\" (UID: \"cdca9b2f-4186-4af4-9644-ea8ddfd6440c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c48874bf-p26bd"
Sep 16 18:42:01.873213 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.873195    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48s42\" (UniqueName: \"kubernetes.io/projected/cdca9b2f-4186-4af4-9644-ea8ddfd6440c-kube-api-access-48s42\") pod \"cluster-proxy-proxy-agent-85c48874bf-p26bd\" (UID: \"cdca9b2f-4186-4af4-9644-ea8ddfd6440c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c48874bf-p26bd"
Sep 16 18:42:01.962366 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.962342    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-8zxkm\" (UID: \"7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8zxkm"
Sep 16 18:42:01.962461 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.962384    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-cjpnz\" (UID: \"aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cjpnz"
Sep 16 18:42:01.964387 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.964370    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-8zxkm\" (UID: \"7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8zxkm"
Sep 16 18:42:01.964454 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:01.964418    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-cjpnz\" (UID: \"aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cjpnz"
Sep 16 18:42:02.004359 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:02.004339    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c48874bf-p26bd"
Sep 16 18:42:02.185417 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:02.185382    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-5dnd8\""
Sep 16 18:42:02.194100 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:02.194077    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cjpnz"
Sep 16 18:42:02.230026 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:02.229959    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-qb52x\""
Sep 16 18:42:02.238214 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:02.238095    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8zxkm"
Sep 16 18:42:02.273162 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:02.273134    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c48874bf-p26bd"]
Sep 16 18:42:02.461529 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:02.461502    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-cjpnz"]
Sep 16 18:42:02.464571 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:42:02.464544    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeaf5a49_9e3f_46c9_992f_fa4e24a2c06b.slice/crio-083f94e06aa71f946606947368f854c526f2c499271f3d2f75fc5edc01e9ddec WatchSource:0}: Error finding container 083f94e06aa71f946606947368f854c526f2c499271f3d2f75fc5edc01e9ddec: Status 404 returned error can't find the container with id 083f94e06aa71f946606947368f854c526f2c499271f3d2f75fc5edc01e9ddec
Sep 16 18:42:02.510654 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:02.510603    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-8zxkm"]
Sep 16 18:42:02.514974 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:42:02.514951    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c12c8f8_59c4_4d2c_ab39_b2199dcfb1d4.slice/crio-9fb8a6c2d55acba1a95dbec0f32c9eabdbfbb17509e957f91d6d650eb48c056f WatchSource:0}: Error finding container 9fb8a6c2d55acba1a95dbec0f32c9eabdbfbb17509e957f91d6d650eb48c056f: Status 404 returned error can't find the container with id 9fb8a6c2d55acba1a95dbec0f32c9eabdbfbb17509e957f91d6d650eb48c056f
Sep 16 18:42:02.601897 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:02.601870    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cjpnz" event={"ID":"aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b","Type":"ContainerStarted","Data":"083f94e06aa71f946606947368f854c526f2c499271f3d2f75fc5edc01e9ddec"}
Sep 16 18:42:02.602886 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:02.602863    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8zxkm" event={"ID":"7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4","Type":"ContainerStarted","Data":"9fb8a6c2d55acba1a95dbec0f32c9eabdbfbb17509e957f91d6d650eb48c056f"}
Sep 16 18:42:02.603921 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:02.603901    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c48874bf-p26bd" event={"ID":"cdca9b2f-4186-4af4-9644-ea8ddfd6440c","Type":"ContainerStarted","Data":"6a5eb7538fd67c1982f4b11b1c664d40a2cc8454c4920f49bb4d8a0e3554e08d"}
Sep 16 18:42:04.615153 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:04.615069    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8zxkm" event={"ID":"7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4","Type":"ContainerStarted","Data":"5fe71447e09a6e22c87f0a505ea6a06a531553cabe4bfcfd6eab16d2ebd63f11"}
Sep 16 18:42:04.646194 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:04.646138    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8zxkm" podStartSLOduration=33.368897014 podStartE2EDuration="34.646117408s" totalImagesPullingTime="1.277220394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:04.645402137 +0000 UTC m=+92.880335594" watchObservedRunningTime="2026-09-16 18:42:04.646117408 +0000 UTC m=+92.881050852"
Sep 16 18:42:05.896099 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:05.896059    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f1156b17-5b7a-48e4-aac5-d0a527142061-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sj4ck\" (UID: \"f1156b17-5b7a-48e4-aac5-d0a527142061\") " pod="openshift-insights/insights-runtime-extractor-sj4ck"
Sep 16 18:42:05.898740 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:05.898705    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f1156b17-5b7a-48e4-aac5-d0a527142061-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sj4ck\" (UID: \"f1156b17-5b7a-48e4-aac5-d0a527142061\") " pod="openshift-insights/insights-runtime-extractor-sj4ck"
Sep 16 18:42:06.098429 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:06.098391    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sj4ck"
Sep 16 18:42:06.623616 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:06.623577    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c48874bf-p26bd" event={"ID":"cdca9b2f-4186-4af4-9644-ea8ddfd6440c","Type":"ContainerStarted","Data":"4ebc0eceb11cb4d1aaa31459e64df6c13dd3bc32af0f97b761ed7af830d722ef"}
Sep 16 18:42:06.625485 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:06.625460    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cjpnz" event={"ID":"aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b","Type":"ContainerStarted","Data":"55c75ae60cd43faa83859999b99ead45ffb4d09cb83f45019509a4f22fa2eac3"}
Sep 16 18:42:06.647354 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:06.647291    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-cjpnz" podStartSLOduration=32.57738855 podStartE2EDuration="36.647271312s" totalImagesPullingTime="4.069882762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:06.646970237 +0000 UTC m=+94.881903685" watchObservedRunningTime="2026-09-16 18:42:06.647271312 +0000 UTC m=+94.882204761"
Sep 16 18:42:06.812470 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:06.812399    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sj4ck"]
Sep 16 18:42:06.816283 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:42:06.816253    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1156b17_5b7a_48e4_aac5_d0a527142061.slice/crio-9f96c5a3cc642f6641c445fb4d2496059077c0855a3ff982172b1a077f4c6680 WatchSource:0}: Error finding container 9f96c5a3cc642f6641c445fb4d2496059077c0855a3ff982172b1a077f4c6680: Status 404 returned error can't find the container with id 9f96c5a3cc642f6641c445fb4d2496059077c0855a3ff982172b1a077f4c6680
Sep 16 18:42:07.630263 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:07.630230    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c48874bf-p26bd" event={"ID":"cdca9b2f-4186-4af4-9644-ea8ddfd6440c","Type":"ContainerStarted","Data":"7480d79e9c6137d6254485e720276a5c56d7bd15c0d944249290b58a96384a4f"}
Sep 16 18:42:07.630603 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:07.630278    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c48874bf-p26bd" event={"ID":"cdca9b2f-4186-4af4-9644-ea8ddfd6440c","Type":"ContainerStarted","Data":"b085cb98ebd2825c6a37a1a990481ff3b10af3bbcd01ea7c6715001b4083533a"}
Sep 16 18:42:07.632034 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:07.631990    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sj4ck" event={"ID":"f1156b17-5b7a-48e4-aac5-d0a527142061","Type":"ContainerStarted","Data":"a68ceff5d640d0e0258b8be1b35c97a5bf18dc5901979fcb4dd46e6f44e3f660"}
Sep 16 18:42:07.632115 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:07.632082    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sj4ck" event={"ID":"f1156b17-5b7a-48e4-aac5-d0a527142061","Type":"ContainerStarted","Data":"d3cb00d6d818f96e50acc9935e73c7dd17f14a4538b7150e93e4ecd203739647"}
Sep 16 18:42:07.632115 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:07.632102    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sj4ck" event={"ID":"f1156b17-5b7a-48e4-aac5-d0a527142061","Type":"ContainerStarted","Data":"9f96c5a3cc642f6641c445fb4d2496059077c0855a3ff982172b1a077f4c6680"}
Sep 16 18:42:09.020807 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:09.020753    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/309dbcdf-d468-4001-805b-e71bb5bdd775-metrics-tls\") pod \"dns-default-86bz5\" (UID: \"309dbcdf-d468-4001-805b-e71bb5bdd775\") " pod="openshift-dns/dns-default-86bz5"
Sep 16 18:42:09.022738 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:09.022719    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/309dbcdf-d468-4001-805b-e71bb5bdd775-metrics-tls\") pod \"dns-default-86bz5\" (UID: \"309dbcdf-d468-4001-805b-e71bb5bdd775\") " pod="openshift-dns/dns-default-86bz5"
Sep 16 18:42:09.042089 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:09.042068    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-67vjb\""
Sep 16 18:42:09.050112 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:09.050086    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-86bz5"
Sep 16 18:42:09.323601 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:09.323506    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85c48874bf-p26bd" podStartSLOduration=4.041301354 podStartE2EDuration="8.323484355s" totalImagesPullingTime="4.282183001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:07.651060489 +0000 UTC m=+95.885993927" watchObservedRunningTime="2026-09-16 18:42:09.323484355 +0000 UTC m=+97.558417801"
Sep 16 18:42:09.323985 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:09.323968    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-86bz5"]
Sep 16 18:42:09.326287 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:42:09.326264    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod309dbcdf_d468_4001_805b_e71bb5bdd775.slice/crio-bf6e600e2be16beb1ce2353daed75b98d4bb846cda6ad81df190443631e8562e WatchSource:0}: Error finding container bf6e600e2be16beb1ce2353daed75b98d4bb846cda6ad81df190443631e8562e: Status 404 returned error can't find the container with id bf6e600e2be16beb1ce2353daed75b98d4bb846cda6ad81df190443631e8562e
Sep 16 18:42:09.640057 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:09.639963    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-86bz5" event={"ID":"309dbcdf-d468-4001-805b-e71bb5bdd775","Type":"ContainerStarted","Data":"bf6e600e2be16beb1ce2353daed75b98d4bb846cda6ad81df190443631e8562e"}
Sep 16 18:42:09.641775 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:09.641747    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sj4ck" event={"ID":"f1156b17-5b7a-48e4-aac5-d0a527142061","Type":"ContainerStarted","Data":"77296bf507d75cc71ebec885f7cdcf4bdc347d164103822af35fa652e54825e0"}
Sep 16 18:42:09.664236 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:09.664194    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-sj4ck" podStartSLOduration=34.906311703 podStartE2EDuration="36.664178857s" totalImagesPullingTime="1.757867154s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:41:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:09.66218821 +0000 UTC m=+97.897121657" watchObservedRunningTime="2026-09-16 18:42:09.664178857 +0000 UTC m=+97.899112315"
Sep 16 18:42:11.650489 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:11.650413    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-86bz5" event={"ID":"309dbcdf-d468-4001-805b-e71bb5bdd775","Type":"ContainerStarted","Data":"63b7867f696bab6ce2a7e7d36889708ffea461b9fa7d45b1a10719cf167d25de"}
Sep 16 18:42:11.650852 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:11.650489    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-86bz5" event={"ID":"309dbcdf-d468-4001-805b-e71bb5bdd775","Type":"ContainerStarted","Data":"e281e7520575dec48be4ff12796777a26862faaa943cd08b60cd69130189604a"}
Sep 16 18:42:11.650852 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:11.650567    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-86bz5"
Sep 16 18:42:11.674605 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:11.674563    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-86bz5" podStartSLOduration=65.249430655 podStartE2EDuration="1m6.674550505s" totalImagesPullingTime="1.42511985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:11.673066445 +0000 UTC m=+99.907999888" watchObservedRunningTime="2026-09-16 18:42:11.674550505 +0000 UTC m=+99.909483971"
Sep 16 18:42:11.728620 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:11.728601    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:42:14.956400 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:14.956368    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sz8rh"]
Sep 16 18:42:14.958583 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:14.958567    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sz8rh"
Sep 16 18:42:14.961696 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:14.961672    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:42:14.961805 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:14.961783    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:42:14.961861 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:14.961840    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:42:14.961923 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:14.961840    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-t5v59\""
Sep 16 18:42:14.962814 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:14.962796    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:42:15.064544 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:15.064522    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/75757666-809b-4bd6-85f5-34118dd9955f-node-exporter-accelerators-collector-config\") pod \"node-exporter-sz8rh\" (UID: \"75757666-809b-4bd6-85f5-34118dd9955f\") " pod="openshift-monitoring/node-exporter-sz8rh"
Sep 16 18:42:15.064675 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:15.064560    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75757666-809b-4bd6-85f5-34118dd9955f-sys\") pod \"node-exporter-sz8rh\" (UID: \"75757666-809b-4bd6-85f5-34118dd9955f\") " pod="openshift-monitoring/node-exporter-sz8rh"
Sep 16 18:42:15.064675 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:15.064580    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/75757666-809b-4bd6-85f5-34118dd9955f-node-exporter-tls\") pod \"node-exporter-sz8rh\" (UID: \"75757666-809b-4bd6-85f5-34118dd9955f\") " pod="openshift-monitoring/node-exporter-sz8rh"
Sep 16 18:42:15.064675 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:15.064636    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/75757666-809b-4bd6-85f5-34118dd9955f-node-exporter-textfile\") pod \"node-exporter-sz8rh\" (UID: \"75757666-809b-4bd6-85f5-34118dd9955f\") " pod="openshift-monitoring/node-exporter-sz8rh"
Sep 16 18:42:15.064675 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:15.064668    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/75757666-809b-4bd6-85f5-34118dd9955f-node-exporter-wtmp\") pod \"node-exporter-sz8rh\" (UID: \"75757666-809b-4bd6-85f5-34118dd9955f\") " pod="openshift-monitoring/node-exporter-sz8rh"
Sep 16 18:42:15.064867 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:15.064685    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75757666-809b-4bd6-85f5-34118dd9955f-metrics-client-ca\") pod \"node-exporter-sz8rh\" (UID: \"75757666-809b-4bd6-85f5-34118dd9955f\") " pod="openshift-monitoring/node-exporter-sz8rh"
Sep 16 18:42:15.064867 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:15.064700    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75757666-809b-4bd6-85f5-34118dd9955f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sz8rh\" (UID: \"75757666-809b-4bd6-85f5-34118dd9955f\") " pod="openshift-monitoring/node-exporter-sz8rh"
Sep 16 18:42:15.064867 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:15.064723    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktlpx\" (UniqueName: \"kubernetes.io/projected/75757666-809b-4bd6-85f5-34118dd9955f-kube-api-access-ktlpx\") pod \"node-exporter-sz8rh\" (UID: \"75757666-809b-4bd6-85f5-34118dd9955f\") " pod="openshift-monitoring/node-exporter-sz8rh"
Sep 16 18:42:15.064867 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:15.064842    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/75757666-809b-4bd6-85f5-34118dd9955f-root\") pod \"node-exporter-sz8rh\" (UID: \"75757666-809b-4bd6-85f5-34118dd9955f\") " pod="openshift-monitoring/node-exporter-sz8rh"
Sep 16 18:42:15.166001 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:15.165973    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/75757666-809b-4bd6-85f5-34118dd9955f-root\") pod \"node-exporter-sz8rh\" (UID: \"75757666-809b-4bd6-85f5-34118dd9955f\") " pod="openshift-monitoring/node-exporter-sz8rh"
Sep 16 18:42:15.166142 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:15.166007    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/75757666-809b-4bd6-85f5-34118dd9955f-node-exporter-accelerators-collector-config\") pod \"node-exporter-sz8rh\" (UID: \"75757666-809b-4bd6-85f5-34118dd9955f\") " pod="openshift-monitoring/node-exporter-sz8rh"
Sep 16 18:42:15.166142 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:15.166028    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75757666-809b-4bd6-85f5-34118dd9955f-sys\") pod \"node-exporter-sz8rh\" (UID: \"75757666-809b-4bd6-85f5-34118dd9955f\") " pod="openshift-monitoring/node-exporter-sz8rh"
Sep 16 18:42:15.166142 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:15.166056    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/75757666-809b-4bd6-85f5-34118dd9955f-node-exporter-tls\") pod \"node-exporter-sz8rh\" (UID: \"75757666-809b-4bd6-85f5-34118dd9955f\") " pod="openshift-monitoring/node-exporter-sz8rh"
Sep 16 18:42:15.166142 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:15.166086    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/75757666-809b-4bd6-85f5-34118dd9955f-node-exporter-textfile\") pod \"node-exporter-sz8rh\" (UID: \"75757666-809b-4bd6-85f5-34118dd9955f\") " pod="openshift-monitoring/node-exporter-sz8rh"
Sep 16 18:42:15.166142 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:15.166108    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/75757666-809b-4bd6-85f5-34118dd9955f-root\") pod \"node-exporter-sz8rh\" (UID: \"75757666-809b-4bd6-85f5-34118dd9955f\") " pod="openshift-monitoring/node-exporter-sz8rh"
Sep 16 18:42:15.166142 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:15.166128    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75757666-809b-4bd6-85f5-34118dd9955f-sys\") pod \"node-exporter-sz8rh\" (UID: \"75757666-809b-4bd6-85f5-34118dd9955f\") " pod="openshift-monitoring/node-exporter-sz8rh"
Sep 16 18:42:15.166424 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:15.166172    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/75757666-809b-4bd6-85f5-34118dd9955f-node-exporter-wtmp\") pod \"node-exporter-sz8rh\" (UID: \"75757666-809b-4bd6-85f5-34118dd9955f\") " pod="openshift-monitoring/node-exporter-sz8rh"
Sep 16 18:42:15.166424 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:15.166198    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75757666-809b-4bd6-85f5-34118dd9955f-metrics-client-ca\") pod \"node-exporter-sz8rh\" (UID: \"75757666-809b-4bd6-85f5-34118dd9955f\") " pod="openshift-monitoring/node-exporter-sz8rh"
Sep 16 18:42:15.166424 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:15.166213    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75757666-809b-4bd6-85f5-34118dd9955f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sz8rh\" (UID: \"75757666-809b-4bd6-85f5-34118dd9955f\") " pod="openshift-monitoring/node-exporter-sz8rh"
Sep 16 18:42:15.166424 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:15.166254    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktlpx\" (UniqueName: \"kubernetes.io/projected/75757666-809b-4bd6-85f5-34118dd9955f-kube-api-access-ktlpx\") pod \"node-exporter-sz8rh\" (UID: \"75757666-809b-4bd6-85f5-34118dd9955f\") " pod="openshift-monitoring/node-exporter-sz8rh"
Sep 16 18:42:15.166424 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:15.166270    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/75757666-809b-4bd6-85f5-34118dd9955f-node-exporter-wtmp\") pod \"node-exporter-sz8rh\" (UID: \"75757666-809b-4bd6-85f5-34118dd9955f\") " pod="openshift-monitoring/node-exporter-sz8rh"
Sep 16 18:42:15.166424 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:15.166411    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/75757666-809b-4bd6-85f5-34118dd9955f-node-exporter-textfile\") pod \"node-exporter-sz8rh\" (UID: \"75757666-809b-4bd6-85f5-34118dd9955f\") " pod="openshift-monitoring/node-exporter-sz8rh"
Sep 16 18:42:15.166734 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:15.166710    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/75757666-809b-4bd6-85f5-34118dd9955f-node-exporter-accelerators-collector-config\") pod \"node-exporter-sz8rh\" (UID: \"75757666-809b-4bd6-85f5-34118dd9955f\") " pod="openshift-monitoring/node-exporter-sz8rh"
Sep 16 18:42:15.166777 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:15.166745    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75757666-809b-4bd6-85f5-34118dd9955f-metrics-client-ca\") pod \"node-exporter-sz8rh\" (UID: \"75757666-809b-4bd6-85f5-34118dd9955f\") " pod="openshift-monitoring/node-exporter-sz8rh"
Sep 16 18:42:15.168529 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:15.168504    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75757666-809b-4bd6-85f5-34118dd9955f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sz8rh\" (UID: \"75757666-809b-4bd6-85f5-34118dd9955f\") " pod="openshift-monitoring/node-exporter-sz8rh"
Sep 16 18:42:15.168626 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:15.168542    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/75757666-809b-4bd6-85f5-34118dd9955f-node-exporter-tls\") pod \"node-exporter-sz8rh\" (UID: \"75757666-809b-4bd6-85f5-34118dd9955f\") " pod="openshift-monitoring/node-exporter-sz8rh"
Sep 16 18:42:15.192821 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:15.192798    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktlpx\" (UniqueName: \"kubernetes.io/projected/75757666-809b-4bd6-85f5-34118dd9955f-kube-api-access-ktlpx\") pod \"node-exporter-sz8rh\" (UID: \"75757666-809b-4bd6-85f5-34118dd9955f\") " pod="openshift-monitoring/node-exporter-sz8rh"
Sep 16 18:42:15.267735 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:15.267685    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sz8rh"
Sep 16 18:42:15.276975 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:42:15.276955    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75757666_809b_4bd6_85f5_34118dd9955f.slice/crio-5a8d53d6421cec744dad819fe2e44633f2e2fbaed3f38a0d8fc1c6391d02f5b4 WatchSource:0}: Error finding container 5a8d53d6421cec744dad819fe2e44633f2e2fbaed3f38a0d8fc1c6391d02f5b4: Status 404 returned error can't find the container with id 5a8d53d6421cec744dad819fe2e44633f2e2fbaed3f38a0d8fc1c6391d02f5b4
Sep 16 18:42:15.663549 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:15.663467    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sz8rh" event={"ID":"75757666-809b-4bd6-85f5-34118dd9955f","Type":"ContainerStarted","Data":"5a8d53d6421cec744dad819fe2e44633f2e2fbaed3f38a0d8fc1c6391d02f5b4"}
Sep 16 18:42:16.668596 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:16.668520    2578 generic.go:356] "Generic (PLEG): container finished" podID="75757666-809b-4bd6-85f5-34118dd9955f" containerID="85ac71e756e03f86c9693e44561e586af30b2e7c2021415df564311287e31bf4" exitCode=0
Sep 16 18:42:16.668596 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:16.668570    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sz8rh" event={"ID":"75757666-809b-4bd6-85f5-34118dd9955f","Type":"ContainerDied","Data":"85ac71e756e03f86c9693e44561e586af30b2e7c2021415df564311287e31bf4"}
Sep 16 18:42:17.674148 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:17.674113    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sz8rh" event={"ID":"75757666-809b-4bd6-85f5-34118dd9955f","Type":"ContainerStarted","Data":"d39bf08a1ec4ef1da7cab19af1e9999b50077ad0330bb6ce4555b5c939e4d2b7"}
Sep 16 18:42:17.674517 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:17.674163    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sz8rh" event={"ID":"75757666-809b-4bd6-85f5-34118dd9955f","Type":"ContainerStarted","Data":"bfd1fba2844d6374baec9bbf13fe217e455dd23e78a814a0190602db7bdfc218"}
Sep 16 18:42:17.709551 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:17.709504    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sz8rh" podStartSLOduration=2.654243062 podStartE2EDuration="3.709484612s" totalImagesPullingTime="993.933452ms" totalInitContainerRuntime="61.308098ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:42:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:17.709284159 +0000 UTC m=+105.944217604" watchObservedRunningTime="2026-09-16 18:42:17.709484612 +0000 UTC m=+105.944418056"
Sep 16 18:42:21.655912 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:21.655882    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-86bz5"
Sep 16 18:42:24.276905 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:24.276876    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-5rf9x"]
Sep 16 18:42:24.279773 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:24.279756    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-5rf9x"
Sep 16 18:42:24.282754 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:24.282733    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-phd8g\""
Sep 16 18:42:24.282860 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:24.282756    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 18:42:24.283110 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:24.283096    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 18:42:24.293539 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:24.293518    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-5rf9x"]
Sep 16 18:42:24.430567 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:24.430542    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmdl2\" (UniqueName: \"kubernetes.io/projected/f5ea7094-a8a5-487d-be9d-195bd5c0fa11-kube-api-access-fmdl2\") pod \"downloads-76f99584f4-5rf9x\" (UID: \"f5ea7094-a8a5-487d-be9d-195bd5c0fa11\") " pod="openshift-console/downloads-76f99584f4-5rf9x"
Sep 16 18:42:24.531431 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:24.531369    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmdl2\" (UniqueName: \"kubernetes.io/projected/f5ea7094-a8a5-487d-be9d-195bd5c0fa11-kube-api-access-fmdl2\") pod \"downloads-76f99584f4-5rf9x\" (UID: \"f5ea7094-a8a5-487d-be9d-195bd5c0fa11\") " pod="openshift-console/downloads-76f99584f4-5rf9x"
Sep 16 18:42:24.545934 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:24.545914    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmdl2\" (UniqueName: \"kubernetes.io/projected/f5ea7094-a8a5-487d-be9d-195bd5c0fa11-kube-api-access-fmdl2\") pod \"downloads-76f99584f4-5rf9x\" (UID: \"f5ea7094-a8a5-487d-be9d-195bd5c0fa11\") " pod="openshift-console/downloads-76f99584f4-5rf9x"
Sep 16 18:42:24.588810 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:24.588788    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-5rf9x"
Sep 16 18:42:24.884194 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:24.884149    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-5rf9x"]
Sep 16 18:42:24.886290 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:42:24.886261    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ea7094_a8a5_487d_be9d_195bd5c0fa11.slice/crio-4f07bbecf18cda7730b2ff268cd0907deab0c97aee21d96e71dc168a2070c3f9 WatchSource:0}: Error finding container 4f07bbecf18cda7730b2ff268cd0907deab0c97aee21d96e71dc168a2070c3f9: Status 404 returned error can't find the container with id 4f07bbecf18cda7730b2ff268cd0907deab0c97aee21d96e71dc168a2070c3f9
Sep 16 18:42:25.701862 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:25.701825    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-5rf9x" event={"ID":"f5ea7094-a8a5-487d-be9d-195bd5c0fa11","Type":"ContainerStarted","Data":"4f07bbecf18cda7730b2ff268cd0907deab0c97aee21d96e71dc168a2070c3f9"}
Sep 16 18:42:26.742772 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:26.742727    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-8fc56db65-5rbz9" podUID="f1abe079-27f1-4078-a08c-406b881a7bde" containerName="registry" containerID="cri-o://75fc09651c26e0e54630f6e62a92319c0eef6b369c8f16059c1d71f9452be852" gracePeriod=30
Sep 16 18:42:26.992936 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:26.992865    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:42:27.149611 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.149574    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f1abe079-27f1-4078-a08c-406b881a7bde-image-registry-private-configuration\") pod \"f1abe079-27f1-4078-a08c-406b881a7bde\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") "
Sep 16 18:42:27.149774 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.149630    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f1abe079-27f1-4078-a08c-406b881a7bde-trusted-ca\") pod \"f1abe079-27f1-4078-a08c-406b881a7bde\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") "
Sep 16 18:42:27.149774 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.149662    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f1abe079-27f1-4078-a08c-406b881a7bde-registry-certificates\") pod \"f1abe079-27f1-4078-a08c-406b881a7bde\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") "
Sep 16 18:42:27.149774 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.149697    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-bound-sa-token\") pod \"f1abe079-27f1-4078-a08c-406b881a7bde\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") "
Sep 16 18:42:27.149774 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.149728    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f1abe079-27f1-4078-a08c-406b881a7bde-installation-pull-secrets\") pod \"f1abe079-27f1-4078-a08c-406b881a7bde\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") "
Sep 16 18:42:27.149774 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.149758    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-registry-tls\") pod \"f1abe079-27f1-4078-a08c-406b881a7bde\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") "
Sep 16 18:42:27.150111 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.149785    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f1abe079-27f1-4078-a08c-406b881a7bde-ca-trust-extracted\") pod \"f1abe079-27f1-4078-a08c-406b881a7bde\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") "
Sep 16 18:42:27.150111 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.149824    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7chl\" (UniqueName: \"kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-kube-api-access-b7chl\") pod \"f1abe079-27f1-4078-a08c-406b881a7bde\" (UID: \"f1abe079-27f1-4078-a08c-406b881a7bde\") "
Sep 16 18:42:27.150304 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.150276    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1abe079-27f1-4078-a08c-406b881a7bde-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f1abe079-27f1-4078-a08c-406b881a7bde" (UID: "f1abe079-27f1-4078-a08c-406b881a7bde"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:27.150388 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.150359    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1abe079-27f1-4078-a08c-406b881a7bde-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f1abe079-27f1-4078-a08c-406b881a7bde" (UID: "f1abe079-27f1-4078-a08c-406b881a7bde"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:27.152658 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.152615    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f1abe079-27f1-4078-a08c-406b881a7bde" (UID: "f1abe079-27f1-4078-a08c-406b881a7bde"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:27.152766 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.152656    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-kube-api-access-b7chl" (OuterVolumeSpecName: "kube-api-access-b7chl") pod "f1abe079-27f1-4078-a08c-406b881a7bde" (UID: "f1abe079-27f1-4078-a08c-406b881a7bde"). InnerVolumeSpecName "kube-api-access-b7chl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:27.152766 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.152720    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1abe079-27f1-4078-a08c-406b881a7bde-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f1abe079-27f1-4078-a08c-406b881a7bde" (UID: "f1abe079-27f1-4078-a08c-406b881a7bde"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:27.152998 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.152975    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f1abe079-27f1-4078-a08c-406b881a7bde" (UID: "f1abe079-27f1-4078-a08c-406b881a7bde"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:27.153143 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.153125    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1abe079-27f1-4078-a08c-406b881a7bde-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f1abe079-27f1-4078-a08c-406b881a7bde" (UID: "f1abe079-27f1-4078-a08c-406b881a7bde"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:27.159577 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.159547    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1abe079-27f1-4078-a08c-406b881a7bde-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f1abe079-27f1-4078-a08c-406b881a7bde" (UID: "f1abe079-27f1-4078-a08c-406b881a7bde"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:27.251127 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.251063    2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f1abe079-27f1-4078-a08c-406b881a7bde-image-registry-private-configuration\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:42:27.251127 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.251091    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f1abe079-27f1-4078-a08c-406b881a7bde-trusted-ca\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:42:27.251127 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.251106    2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f1abe079-27f1-4078-a08c-406b881a7bde-registry-certificates\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:42:27.251127 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.251119    2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-bound-sa-token\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:42:27.251127 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.251131    2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f1abe079-27f1-4078-a08c-406b881a7bde-installation-pull-secrets\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:42:27.251355 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.251139    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-registry-tls\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:42:27.251355 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.251147    2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f1abe079-27f1-4078-a08c-406b881a7bde-ca-trust-extracted\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:42:27.251355 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.251154    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7chl\" (UniqueName: \"kubernetes.io/projected/f1abe079-27f1-4078-a08c-406b881a7bde-kube-api-access-b7chl\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:42:27.710935 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.710902    2578 generic.go:356] "Generic (PLEG): container finished" podID="f1abe079-27f1-4078-a08c-406b881a7bde" containerID="75fc09651c26e0e54630f6e62a92319c0eef6b369c8f16059c1d71f9452be852" exitCode=0
Sep 16 18:42:27.711145 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.710956    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8fc56db65-5rbz9"
Sep 16 18:42:27.711145 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.710994    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8fc56db65-5rbz9" event={"ID":"f1abe079-27f1-4078-a08c-406b881a7bde","Type":"ContainerDied","Data":"75fc09651c26e0e54630f6e62a92319c0eef6b369c8f16059c1d71f9452be852"}
Sep 16 18:42:27.711145 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.711074    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8fc56db65-5rbz9" event={"ID":"f1abe079-27f1-4078-a08c-406b881a7bde","Type":"ContainerDied","Data":"143efc13cc1cda370011b8e57adebcbe8669533f57a1fe3b0445a318f999b5c7"}
Sep 16 18:42:27.711145 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.711105    2578 scope.go:117] "RemoveContainer" containerID="75fc09651c26e0e54630f6e62a92319c0eef6b369c8f16059c1d71f9452be852"
Sep 16 18:42:27.721212 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.721178    2578 scope.go:117] "RemoveContainer" containerID="75fc09651c26e0e54630f6e62a92319c0eef6b369c8f16059c1d71f9452be852"
Sep 16 18:42:27.721581 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:42:27.721549    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75fc09651c26e0e54630f6e62a92319c0eef6b369c8f16059c1d71f9452be852\": container with ID starting with 75fc09651c26e0e54630f6e62a92319c0eef6b369c8f16059c1d71f9452be852 not found: ID does not exist" containerID="75fc09651c26e0e54630f6e62a92319c0eef6b369c8f16059c1d71f9452be852"
Sep 16 18:42:27.721687 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.721591    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75fc09651c26e0e54630f6e62a92319c0eef6b369c8f16059c1d71f9452be852"} err="failed to get container status \"75fc09651c26e0e54630f6e62a92319c0eef6b369c8f16059c1d71f9452be852\": rpc error: code = NotFound desc = could not find container \"75fc09651c26e0e54630f6e62a92319c0eef6b369c8f16059c1d71f9452be852\": container with ID starting with 75fc09651c26e0e54630f6e62a92319c0eef6b369c8f16059c1d71f9452be852 not found: ID does not exist"
Sep 16 18:42:27.742910 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.742886    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8fc56db65-5rbz9"]
Sep 16 18:42:27.751098 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:27.751074    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-8fc56db65-5rbz9"]
Sep 16 18:42:28.217470 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:28.217435    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1abe079-27f1-4078-a08c-406b881a7bde" path="/var/lib/kubelet/pods/f1abe079-27f1-4078-a08c-406b881a7bde/volumes"
Sep 16 18:42:34.454755 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:34.454716    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7fd49c8b6d-t4trc"]
Sep 16 18:42:34.455169 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:34.454989    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1abe079-27f1-4078-a08c-406b881a7bde" containerName="registry"
Sep 16 18:42:34.455169 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:34.455006    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1abe079-27f1-4078-a08c-406b881a7bde" containerName="registry"
Sep 16 18:42:34.455169 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:34.455081    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1abe079-27f1-4078-a08c-406b881a7bde" containerName="registry"
Sep 16 18:42:34.458153 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:34.458130    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fd49c8b6d-t4trc"
Sep 16 18:42:34.460899 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:34.460777    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 18:42:34.460899 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:34.460791    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 18:42:34.460899 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:34.460802    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-xpcxq\""
Sep 16 18:42:34.460899 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:34.460821    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 18:42:34.461211 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:34.461190    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 18:42:34.461710 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:34.461689    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 18:42:34.471907 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:34.471749    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fd49c8b6d-t4trc"]
Sep 16 18:42:34.609391 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:34.609355    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19a2f06e-440f-44dd-b8f8-039862e81971-oauth-serving-cert\") pod \"console-7fd49c8b6d-t4trc\" (UID: \"19a2f06e-440f-44dd-b8f8-039862e81971\") " pod="openshift-console/console-7fd49c8b6d-t4trc"
Sep 16 18:42:34.609543 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:34.609411    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19a2f06e-440f-44dd-b8f8-039862e81971-service-ca\") pod \"console-7fd49c8b6d-t4trc\" (UID: \"19a2f06e-440f-44dd-b8f8-039862e81971\") " pod="openshift-console/console-7fd49c8b6d-t4trc"
Sep 16 18:42:34.609543 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:34.609493    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19a2f06e-440f-44dd-b8f8-039862e81971-console-oauth-config\") pod \"console-7fd49c8b6d-t4trc\" (UID: \"19a2f06e-440f-44dd-b8f8-039862e81971\") " pod="openshift-console/console-7fd49c8b6d-t4trc"
Sep 16 18:42:34.609543 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:34.609536    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19a2f06e-440f-44dd-b8f8-039862e81971-console-serving-cert\") pod \"console-7fd49c8b6d-t4trc\" (UID: \"19a2f06e-440f-44dd-b8f8-039862e81971\") " pod="openshift-console/console-7fd49c8b6d-t4trc"
Sep 16 18:42:34.609683 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:34.609580    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rm85\" (UniqueName: \"kubernetes.io/projected/19a2f06e-440f-44dd-b8f8-039862e81971-kube-api-access-2rm85\") pod \"console-7fd49c8b6d-t4trc\" (UID: \"19a2f06e-440f-44dd-b8f8-039862e81971\") " pod="openshift-console/console-7fd49c8b6d-t4trc"
Sep 16 18:42:34.609683 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:34.609622    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19a2f06e-440f-44dd-b8f8-039862e81971-console-config\") pod \"console-7fd49c8b6d-t4trc\" (UID: \"19a2f06e-440f-44dd-b8f8-039862e81971\") " pod="openshift-console/console-7fd49c8b6d-t4trc"
Sep 16 18:42:34.710841 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:34.710773    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19a2f06e-440f-44dd-b8f8-039862e81971-console-oauth-config\") pod \"console-7fd49c8b6d-t4trc\" (UID: \"19a2f06e-440f-44dd-b8f8-039862e81971\") " pod="openshift-console/console-7fd49c8b6d-t4trc"
Sep 16 18:42:34.710841 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:34.710810    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19a2f06e-440f-44dd-b8f8-039862e81971-console-serving-cert\") pod \"console-7fd49c8b6d-t4trc\" (UID: \"19a2f06e-440f-44dd-b8f8-039862e81971\") " pod="openshift-console/console-7fd49c8b6d-t4trc"
Sep 16 18:42:34.711056 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:34.710863    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rm85\" (UniqueName: \"kubernetes.io/projected/19a2f06e-440f-44dd-b8f8-039862e81971-kube-api-access-2rm85\") pod \"console-7fd49c8b6d-t4trc\" (UID: \"19a2f06e-440f-44dd-b8f8-039862e81971\") " pod="openshift-console/console-7fd49c8b6d-t4trc"
Sep 16 18:42:34.711056 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:34.710897    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19a2f06e-440f-44dd-b8f8-039862e81971-console-config\") pod \"console-7fd49c8b6d-t4trc\" (UID: \"19a2f06e-440f-44dd-b8f8-039862e81971\") " pod="openshift-console/console-7fd49c8b6d-t4trc"
Sep 16 18:42:34.711056 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:34.710920    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19a2f06e-440f-44dd-b8f8-039862e81971-oauth-serving-cert\") pod \"console-7fd49c8b6d-t4trc\" (UID: \"19a2f06e-440f-44dd-b8f8-039862e81971\") " pod="openshift-console/console-7fd49c8b6d-t4trc"
Sep 16 18:42:34.711056 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:34.710943    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19a2f06e-440f-44dd-b8f8-039862e81971-service-ca\") pod \"console-7fd49c8b6d-t4trc\" (UID: \"19a2f06e-440f-44dd-b8f8-039862e81971\") " pod="openshift-console/console-7fd49c8b6d-t4trc"
Sep 16 18:42:34.711948 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:34.711917    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19a2f06e-440f-44dd-b8f8-039862e81971-service-ca\") pod \"console-7fd49c8b6d-t4trc\" (UID: \"19a2f06e-440f-44dd-b8f8-039862e81971\") " pod="openshift-console/console-7fd49c8b6d-t4trc"
Sep 16 18:42:34.712081 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:34.711957    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19a2f06e-440f-44dd-b8f8-039862e81971-console-config\") pod \"console-7fd49c8b6d-t4trc\" (UID: \"19a2f06e-440f-44dd-b8f8-039862e81971\") " pod="openshift-console/console-7fd49c8b6d-t4trc"
Sep 16 18:42:34.712332 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:34.712314    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19a2f06e-440f-44dd-b8f8-039862e81971-oauth-serving-cert\") pod \"console-7fd49c8b6d-t4trc\" (UID: \"19a2f06e-440f-44dd-b8f8-039862e81971\") " pod="openshift-console/console-7fd49c8b6d-t4trc"
Sep 16 18:42:34.713547 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:34.713525    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19a2f06e-440f-44dd-b8f8-039862e81971-console-serving-cert\") pod \"console-7fd49c8b6d-t4trc\" (UID: \"19a2f06e-440f-44dd-b8f8-039862e81971\") " pod="openshift-console/console-7fd49c8b6d-t4trc"
Sep 16 18:42:34.713668 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:34.713646    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19a2f06e-440f-44dd-b8f8-039862e81971-console-oauth-config\") pod \"console-7fd49c8b6d-t4trc\" (UID: \"19a2f06e-440f-44dd-b8f8-039862e81971\") " pod="openshift-console/console-7fd49c8b6d-t4trc"
Sep 16 18:42:34.730272 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:34.730247    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rm85\" (UniqueName: \"kubernetes.io/projected/19a2f06e-440f-44dd-b8f8-039862e81971-kube-api-access-2rm85\") pod \"console-7fd49c8b6d-t4trc\" (UID: \"19a2f06e-440f-44dd-b8f8-039862e81971\") " pod="openshift-console/console-7fd49c8b6d-t4trc"
Sep 16 18:42:34.768616 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:34.768588    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fd49c8b6d-t4trc"
Sep 16 18:42:41.036218 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:41.036148    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fd49c8b6d-t4trc"]
Sep 16 18:42:41.117389 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:42:41.117361    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a2f06e_440f_44dd_b8f8_039862e81971.slice/crio-d0aeabc9b7351396b6eb128c3f9701082260febde0dda450f3b1e1f93fadee3e WatchSource:0}: Error finding container d0aeabc9b7351396b6eb128c3f9701082260febde0dda450f3b1e1f93fadee3e: Status 404 returned error can't find the container with id d0aeabc9b7351396b6eb128c3f9701082260febde0dda450f3b1e1f93fadee3e
Sep 16 18:42:41.771174 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:41.771075    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fd49c8b6d-t4trc" event={"ID":"19a2f06e-440f-44dd-b8f8-039862e81971","Type":"ContainerStarted","Data":"d0aeabc9b7351396b6eb128c3f9701082260febde0dda450f3b1e1f93fadee3e"}
Sep 16 18:42:41.773626 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:41.773527    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-5rf9x" event={"ID":"f5ea7094-a8a5-487d-be9d-195bd5c0fa11","Type":"ContainerStarted","Data":"1edcecbeff228c3f1f14dd7b3eaf13be835fce04858d63342b00cb029258a6bc"}
Sep 16 18:42:41.774061 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:41.774007    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-5rf9x"
Sep 16 18:42:41.788300 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:41.788248    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-5rf9x"
Sep 16 18:42:41.798573 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:41.798514    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-5rf9x" podStartSLOduration=1.4964782730000001 podStartE2EDuration="17.798493228s" totalImagesPullingTime="16.302014955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:41.79563706 +0000 UTC m=+130.030570507" watchObservedRunningTime="2026-09-16 18:42:41.798493228 +0000 UTC m=+130.033426673"
Sep 16 18:42:41.970371 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:41.970330    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/733b59e1-b843-4255-bf37-71c81c244989-metrics-certs\") pod \"network-metrics-daemon-xq67c\" (UID: \"733b59e1-b843-4255-bf37-71c81c244989\") " pod="openshift-multus/network-metrics-daemon-xq67c"
Sep 16 18:42:41.973713 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:41.973663    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/733b59e1-b843-4255-bf37-71c81c244989-metrics-certs\") pod \"network-metrics-daemon-xq67c\" (UID: \"733b59e1-b843-4255-bf37-71c81c244989\") " pod="openshift-multus/network-metrics-daemon-xq67c"
Sep 16 18:42:42.131958 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:42.131858    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s9z5g\""
Sep 16 18:42:42.139453 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:42.139271    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xq67c"
Sep 16 18:42:42.430562 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:42.430529    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xq67c"]
Sep 16 18:42:42.435698 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:42:42.435273    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod733b59e1_b843_4255_bf37_71c81c244989.slice/crio-eb35c44d958871b63eb51019b8f63d20589fc76b4341409797f883bd983cd584 WatchSource:0}: Error finding container eb35c44d958871b63eb51019b8f63d20589fc76b4341409797f883bd983cd584: Status 404 returned error can't find the container with id eb35c44d958871b63eb51019b8f63d20589fc76b4341409797f883bd983cd584
Sep 16 18:42:42.779973 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:42.779879    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xq67c" event={"ID":"733b59e1-b843-4255-bf37-71c81c244989","Type":"ContainerStarted","Data":"eb35c44d958871b63eb51019b8f63d20589fc76b4341409797f883bd983cd584"}
Sep 16 18:42:43.128850 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:43.128602    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74df8dc5ff-6lv5f"]
Sep 16 18:42:43.131242 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:43.131029    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74df8dc5ff-6lv5f"
Sep 16 18:42:43.142081 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:43.142032    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 18:42:43.146687 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:43.146643    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74df8dc5ff-6lv5f"]
Sep 16 18:42:43.280853 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:43.280816    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25b9ad89-711c-403f-9d5c-8f24f26ec617-trusted-ca-bundle\") pod \"console-74df8dc5ff-6lv5f\" (UID: \"25b9ad89-711c-403f-9d5c-8f24f26ec617\") " pod="openshift-console/console-74df8dc5ff-6lv5f"
Sep 16 18:42:43.281065 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:43.280867    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/25b9ad89-711c-403f-9d5c-8f24f26ec617-service-ca\") pod \"console-74df8dc5ff-6lv5f\" (UID: \"25b9ad89-711c-403f-9d5c-8f24f26ec617\") " pod="openshift-console/console-74df8dc5ff-6lv5f"
Sep 16 18:42:43.281065 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:43.280893    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/25b9ad89-711c-403f-9d5c-8f24f26ec617-console-config\") pod \"console-74df8dc5ff-6lv5f\" (UID: \"25b9ad89-711c-403f-9d5c-8f24f26ec617\") " pod="openshift-console/console-74df8dc5ff-6lv5f"
Sep 16 18:42:43.281065 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:43.280917    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/25b9ad89-711c-403f-9d5c-8f24f26ec617-oauth-serving-cert\") pod \"console-74df8dc5ff-6lv5f\" (UID: \"25b9ad89-711c-403f-9d5c-8f24f26ec617\") " pod="openshift-console/console-74df8dc5ff-6lv5f"
Sep 16 18:42:43.281065 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:43.280951    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/25b9ad89-711c-403f-9d5c-8f24f26ec617-console-serving-cert\") pod \"console-74df8dc5ff-6lv5f\" (UID: \"25b9ad89-711c-403f-9d5c-8f24f26ec617\") " pod="openshift-console/console-74df8dc5ff-6lv5f"
Sep 16 18:42:43.281065 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:43.280983    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/25b9ad89-711c-403f-9d5c-8f24f26ec617-console-oauth-config\") pod \"console-74df8dc5ff-6lv5f\" (UID: \"25b9ad89-711c-403f-9d5c-8f24f26ec617\") " pod="openshift-console/console-74df8dc5ff-6lv5f"
Sep 16 18:42:43.281065 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:43.281020    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh5zm\" (UniqueName: \"kubernetes.io/projected/25b9ad89-711c-403f-9d5c-8f24f26ec617-kube-api-access-jh5zm\") pod \"console-74df8dc5ff-6lv5f\" (UID: \"25b9ad89-711c-403f-9d5c-8f24f26ec617\") " pod="openshift-console/console-74df8dc5ff-6lv5f"
Sep 16 18:42:43.382208 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:43.382094    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25b9ad89-711c-403f-9d5c-8f24f26ec617-trusted-ca-bundle\") pod \"console-74df8dc5ff-6lv5f\" (UID: \"25b9ad89-711c-403f-9d5c-8f24f26ec617\") " pod="openshift-console/console-74df8dc5ff-6lv5f"
Sep 16 18:42:43.382208 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:43.382149    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/25b9ad89-711c-403f-9d5c-8f24f26ec617-service-ca\") pod \"console-74df8dc5ff-6lv5f\" (UID: \"25b9ad89-711c-403f-9d5c-8f24f26ec617\") " pod="openshift-console/console-74df8dc5ff-6lv5f"
Sep 16 18:42:43.382208 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:43.382176    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/25b9ad89-711c-403f-9d5c-8f24f26ec617-console-config\") pod \"console-74df8dc5ff-6lv5f\" (UID: \"25b9ad89-711c-403f-9d5c-8f24f26ec617\") " pod="openshift-console/console-74df8dc5ff-6lv5f"
Sep 16 18:42:43.382208 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:43.382202    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/25b9ad89-711c-403f-9d5c-8f24f26ec617-oauth-serving-cert\") pod \"console-74df8dc5ff-6lv5f\" (UID: \"25b9ad89-711c-403f-9d5c-8f24f26ec617\") " pod="openshift-console/console-74df8dc5ff-6lv5f"
Sep 16 18:42:43.382511 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:43.382235    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/25b9ad89-711c-403f-9d5c-8f24f26ec617-console-serving-cert\") pod \"console-74df8dc5ff-6lv5f\" (UID: \"25b9ad89-711c-403f-9d5c-8f24f26ec617\") " pod="openshift-console/console-74df8dc5ff-6lv5f"
Sep 16 18:42:43.382511 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:43.382273    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/25b9ad89-711c-403f-9d5c-8f24f26ec617-console-oauth-config\") pod \"console-74df8dc5ff-6lv5f\" (UID: \"25b9ad89-711c-403f-9d5c-8f24f26ec617\") " pod="openshift-console/console-74df8dc5ff-6lv5f"
Sep 16 18:42:43.383156 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:43.383121    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/25b9ad89-711c-403f-9d5c-8f24f26ec617-service-ca\") pod \"console-74df8dc5ff-6lv5f\" (UID: \"25b9ad89-711c-403f-9d5c-8f24f26ec617\") " pod="openshift-console/console-74df8dc5ff-6lv5f"
Sep 16 18:42:43.383280 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:43.383221    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh5zm\" (UniqueName: \"kubernetes.io/projected/25b9ad89-711c-403f-9d5c-8f24f26ec617-kube-api-access-jh5zm\") pod \"console-74df8dc5ff-6lv5f\" (UID: \"25b9ad89-711c-403f-9d5c-8f24f26ec617\") " pod="openshift-console/console-74df8dc5ff-6lv5f"
Sep 16 18:42:43.383280 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:43.383248    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/25b9ad89-711c-403f-9d5c-8f24f26ec617-oauth-serving-cert\") pod \"console-74df8dc5ff-6lv5f\" (UID: \"25b9ad89-711c-403f-9d5c-8f24f26ec617\") " pod="openshift-console/console-74df8dc5ff-6lv5f"
Sep 16 18:42:43.383676 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:43.383587    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/25b9ad89-711c-403f-9d5c-8f24f26ec617-console-config\") pod \"console-74df8dc5ff-6lv5f\" (UID: \"25b9ad89-711c-403f-9d5c-8f24f26ec617\") " pod="openshift-console/console-74df8dc5ff-6lv5f"
Sep 16 18:42:43.385805 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:43.385783    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25b9ad89-711c-403f-9d5c-8f24f26ec617-trusted-ca-bundle\") pod \"console-74df8dc5ff-6lv5f\" (UID: \"25b9ad89-711c-403f-9d5c-8f24f26ec617\") " pod="openshift-console/console-74df8dc5ff-6lv5f"
Sep 16 18:42:43.386899 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:43.386819    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/25b9ad89-711c-403f-9d5c-8f24f26ec617-console-serving-cert\") pod \"console-74df8dc5ff-6lv5f\" (UID: \"25b9ad89-711c-403f-9d5c-8f24f26ec617\") " pod="openshift-console/console-74df8dc5ff-6lv5f"
Sep 16 18:42:43.387588 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:43.387558    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/25b9ad89-711c-403f-9d5c-8f24f26ec617-console-oauth-config\") pod \"console-74df8dc5ff-6lv5f\" (UID: \"25b9ad89-711c-403f-9d5c-8f24f26ec617\") " pod="openshift-console/console-74df8dc5ff-6lv5f"
Sep 16 18:42:43.395758 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:43.395728    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh5zm\" (UniqueName: \"kubernetes.io/projected/25b9ad89-711c-403f-9d5c-8f24f26ec617-kube-api-access-jh5zm\") pod \"console-74df8dc5ff-6lv5f\" (UID: \"25b9ad89-711c-403f-9d5c-8f24f26ec617\") " pod="openshift-console/console-74df8dc5ff-6lv5f"
Sep 16 18:42:43.444574 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:43.444532    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74df8dc5ff-6lv5f"
Sep 16 18:42:44.530167 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:44.530093    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74df8dc5ff-6lv5f"]
Sep 16 18:42:44.657030 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:42:44.656924    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b9ad89_711c_403f_9d5c_8f24f26ec617.slice/crio-b234b72864f3313f4807b5f78002341e8ae9ecb0848b31c0d8d5e87b9af203b9 WatchSource:0}: Error finding container b234b72864f3313f4807b5f78002341e8ae9ecb0848b31c0d8d5e87b9af203b9: Status 404 returned error can't find the container with id b234b72864f3313f4807b5f78002341e8ae9ecb0848b31c0d8d5e87b9af203b9
Sep 16 18:42:44.790934 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:44.790860    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74df8dc5ff-6lv5f" event={"ID":"25b9ad89-711c-403f-9d5c-8f24f26ec617","Type":"ContainerStarted","Data":"b234b72864f3313f4807b5f78002341e8ae9ecb0848b31c0d8d5e87b9af203b9"}
Sep 16 18:42:45.797355 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:45.797320    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74df8dc5ff-6lv5f" event={"ID":"25b9ad89-711c-403f-9d5c-8f24f26ec617","Type":"ContainerStarted","Data":"aa9733be3e73b34afe3f306a1a0b1e48233a4d9ca2e090dd272552d0678b4063"}
Sep 16 18:42:45.799215 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:45.799178    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fd49c8b6d-t4trc" event={"ID":"19a2f06e-440f-44dd-b8f8-039862e81971","Type":"ContainerStarted","Data":"eb506c739d7372c3bb0aa46b5defafb1741ca1df7c68488e0da2a47bd3fae3bb"}
Sep 16 18:42:45.818055 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:45.817991    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74df8dc5ff-6lv5f" podStartSLOduration=2.817973057 podStartE2EDuration="2.817973057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:45.816808961 +0000 UTC m=+134.051742408" watchObservedRunningTime="2026-09-16 18:42:45.817973057 +0000 UTC m=+134.052906521"
Sep 16 18:42:45.847221 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:45.847145    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7fd49c8b6d-t4trc" podStartSLOduration=8.300580192 podStartE2EDuration="11.847124841s" totalImagesPullingTime="3.546544649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:45.846373365 +0000 UTC m=+134.081306810" watchObservedRunningTime="2026-09-16 18:42:45.847124841 +0000 UTC m=+134.082058300"
Sep 16 18:42:46.805247 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:46.805202    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xq67c" event={"ID":"733b59e1-b843-4255-bf37-71c81c244989","Type":"ContainerStarted","Data":"2cd7a10c93917d6cc0b35c96d055ff6afbbf4950547c0c78cf1841cae8c7b53a"}
Sep 16 18:42:46.805698 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:46.805279    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xq67c" event={"ID":"733b59e1-b843-4255-bf37-71c81c244989","Type":"ContainerStarted","Data":"b6c2ad26f98b43dabbd0a78a6c46ea0c64ef0736db7fd9e0e849f36ecd6a142a"}
Sep 16 18:42:46.824941 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:46.824882    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xq67c" podStartSLOduration=131.582120399 podStartE2EDuration="2m14.824863845s" totalImagesPullingTime="3.242743446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:46.823457986 +0000 UTC m=+135.058391430" watchObservedRunningTime="2026-09-16 18:42:46.824863845 +0000 UTC m=+135.059797289"
Sep 16 18:42:53.445639 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:53.445606    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74df8dc5ff-6lv5f"
Sep 16 18:42:53.446254 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:53.445661    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74df8dc5ff-6lv5f"
Sep 16 18:42:53.450344 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:53.450325    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74df8dc5ff-6lv5f"
Sep 16 18:42:53.833524 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:53.833466    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74df8dc5ff-6lv5f"
Sep 16 18:42:53.888971 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:53.888935    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fd49c8b6d-t4trc"]
Sep 16 18:42:54.768854 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:54.768819    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7fd49c8b6d-t4trc"
Sep 16 18:42:58.855324 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:58.855296    2578 generic.go:356] "Generic (PLEG): container finished" podID="2260af00-cdea-4591-9ef1-cf8631deb0fc" containerID="d09b66b1a20056fdfbd401f9566f9a8addc372fb14b42f25665074a167e579f3" exitCode=0
Sep 16 18:42:58.855713 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:58.855377    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xfz8z" event={"ID":"2260af00-cdea-4591-9ef1-cf8631deb0fc","Type":"ContainerDied","Data":"d09b66b1a20056fdfbd401f9566f9a8addc372fb14b42f25665074a167e579f3"}
Sep 16 18:42:58.855713 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:58.855700    2578 scope.go:117] "RemoveContainer" containerID="d09b66b1a20056fdfbd401f9566f9a8addc372fb14b42f25665074a167e579f3"
Sep 16 18:42:58.856993 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:58.856971    2578 generic.go:356] "Generic (PLEG): container finished" podID="6f49e184-c6c2-4ffe-bccb-681321fa052a" containerID="d7fa6774515ed01591a0fd53edb1835ccae5128277ad96335d63b7cc4ff1871a" exitCode=0
Sep 16 18:42:58.857107 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:58.857009    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2n2dw" event={"ID":"6f49e184-c6c2-4ffe-bccb-681321fa052a","Type":"ContainerDied","Data":"d7fa6774515ed01591a0fd53edb1835ccae5128277ad96335d63b7cc4ff1871a"}
Sep 16 18:42:58.857270 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:58.857257    2578 scope.go:117] "RemoveContainer" containerID="d7fa6774515ed01591a0fd53edb1835ccae5128277ad96335d63b7cc4ff1871a"
Sep 16 18:42:59.862198 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:59.862161    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2n2dw" event={"ID":"6f49e184-c6c2-4ffe-bccb-681321fa052a","Type":"ContainerStarted","Data":"0ac1d04280ba5addf4acd18383288b309a75693edad641d83edfee5ca94a08c2"}
Sep 16 18:42:59.864051 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:42:59.864013    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-xfz8z" event={"ID":"2260af00-cdea-4591-9ef1-cf8631deb0fc","Type":"ContainerStarted","Data":"aeb7154f90d89e50efa2cfadd2d7a6b21ce23427a4579d8c9275270135ab0802"}
Sep 16 18:43:18.911222 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:18.911159    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7fd49c8b6d-t4trc" podUID="19a2f06e-440f-44dd-b8f8-039862e81971" containerName="console" containerID="cri-o://eb506c739d7372c3bb0aa46b5defafb1741ca1df7c68488e0da2a47bd3fae3bb" gracePeriod=15
Sep 16 18:43:19.171601 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:19.171548    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fd49c8b6d-t4trc"
Sep 16 18:43:19.218340 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:19.218319    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19a2f06e-440f-44dd-b8f8-039862e81971-console-oauth-config\") pod \"19a2f06e-440f-44dd-b8f8-039862e81971\" (UID: \"19a2f06e-440f-44dd-b8f8-039862e81971\") "
Sep 16 18:43:19.218463 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:19.218348    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19a2f06e-440f-44dd-b8f8-039862e81971-oauth-serving-cert\") pod \"19a2f06e-440f-44dd-b8f8-039862e81971\" (UID: \"19a2f06e-440f-44dd-b8f8-039862e81971\") "
Sep 16 18:43:19.218463 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:19.218384    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19a2f06e-440f-44dd-b8f8-039862e81971-console-config\") pod \"19a2f06e-440f-44dd-b8f8-039862e81971\" (UID: \"19a2f06e-440f-44dd-b8f8-039862e81971\") "
Sep 16 18:43:19.218463 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:19.218425    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19a2f06e-440f-44dd-b8f8-039862e81971-service-ca\") pod \"19a2f06e-440f-44dd-b8f8-039862e81971\" (UID: \"19a2f06e-440f-44dd-b8f8-039862e81971\") "
Sep 16 18:43:19.218463 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:19.218455    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rm85\" (UniqueName: \"kubernetes.io/projected/19a2f06e-440f-44dd-b8f8-039862e81971-kube-api-access-2rm85\") pod \"19a2f06e-440f-44dd-b8f8-039862e81971\" (UID: \"19a2f06e-440f-44dd-b8f8-039862e81971\") "
Sep 16 18:43:19.218643 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:19.218493    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19a2f06e-440f-44dd-b8f8-039862e81971-console-serving-cert\") pod \"19a2f06e-440f-44dd-b8f8-039862e81971\" (UID: \"19a2f06e-440f-44dd-b8f8-039862e81971\") "
Sep 16 18:43:19.219250 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:19.218958    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19a2f06e-440f-44dd-b8f8-039862e81971-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "19a2f06e-440f-44dd-b8f8-039862e81971" (UID: "19a2f06e-440f-44dd-b8f8-039862e81971"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:19.219250 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:19.219174    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19a2f06e-440f-44dd-b8f8-039862e81971-console-config" (OuterVolumeSpecName: "console-config") pod "19a2f06e-440f-44dd-b8f8-039862e81971" (UID: "19a2f06e-440f-44dd-b8f8-039862e81971"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:19.219250 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:19.219216    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19a2f06e-440f-44dd-b8f8-039862e81971-service-ca" (OuterVolumeSpecName: "service-ca") pod "19a2f06e-440f-44dd-b8f8-039862e81971" (UID: "19a2f06e-440f-44dd-b8f8-039862e81971"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:19.223713 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:19.222846    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19a2f06e-440f-44dd-b8f8-039862e81971-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "19a2f06e-440f-44dd-b8f8-039862e81971" (UID: "19a2f06e-440f-44dd-b8f8-039862e81971"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:19.223809 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:19.223772    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19a2f06e-440f-44dd-b8f8-039862e81971-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "19a2f06e-440f-44dd-b8f8-039862e81971" (UID: "19a2f06e-440f-44dd-b8f8-039862e81971"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:19.225005 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:19.224981    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a2f06e-440f-44dd-b8f8-039862e81971-kube-api-access-2rm85" (OuterVolumeSpecName: "kube-api-access-2rm85") pod "19a2f06e-440f-44dd-b8f8-039862e81971" (UID: "19a2f06e-440f-44dd-b8f8-039862e81971"). InnerVolumeSpecName "kube-api-access-2rm85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:43:19.319686 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:19.319663    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rm85\" (UniqueName: \"kubernetes.io/projected/19a2f06e-440f-44dd-b8f8-039862e81971-kube-api-access-2rm85\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:43:19.319774 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:19.319690    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19a2f06e-440f-44dd-b8f8-039862e81971-console-serving-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:43:19.319774 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:19.319700    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19a2f06e-440f-44dd-b8f8-039862e81971-console-oauth-config\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:43:19.319774 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:19.319708    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19a2f06e-440f-44dd-b8f8-039862e81971-oauth-serving-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:43:19.319774 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:19.319717    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19a2f06e-440f-44dd-b8f8-039862e81971-console-config\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:43:19.319774 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:19.319741    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19a2f06e-440f-44dd-b8f8-039862e81971-service-ca\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:43:19.931570 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:19.931539    2578 generic.go:356] "Generic (PLEG): container finished" podID="19a2f06e-440f-44dd-b8f8-039862e81971" containerID="eb506c739d7372c3bb0aa46b5defafb1741ca1df7c68488e0da2a47bd3fae3bb" exitCode=0
Sep 16 18:43:19.931920 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:19.931591    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fd49c8b6d-t4trc"
Sep 16 18:43:19.931920 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:19.931635    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fd49c8b6d-t4trc" event={"ID":"19a2f06e-440f-44dd-b8f8-039862e81971","Type":"ContainerDied","Data":"eb506c739d7372c3bb0aa46b5defafb1741ca1df7c68488e0da2a47bd3fae3bb"}
Sep 16 18:43:19.931920 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:19.931710    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fd49c8b6d-t4trc" event={"ID":"19a2f06e-440f-44dd-b8f8-039862e81971","Type":"ContainerDied","Data":"d0aeabc9b7351396b6eb128c3f9701082260febde0dda450f3b1e1f93fadee3e"}
Sep 16 18:43:19.931920 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:19.931732    2578 scope.go:117] "RemoveContainer" containerID="eb506c739d7372c3bb0aa46b5defafb1741ca1df7c68488e0da2a47bd3fae3bb"
Sep 16 18:43:19.939906 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:19.939890    2578 scope.go:117] "RemoveContainer" containerID="eb506c739d7372c3bb0aa46b5defafb1741ca1df7c68488e0da2a47bd3fae3bb"
Sep 16 18:43:19.940212 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:43:19.940192    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb506c739d7372c3bb0aa46b5defafb1741ca1df7c68488e0da2a47bd3fae3bb\": container with ID starting with eb506c739d7372c3bb0aa46b5defafb1741ca1df7c68488e0da2a47bd3fae3bb not found: ID does not exist" containerID="eb506c739d7372c3bb0aa46b5defafb1741ca1df7c68488e0da2a47bd3fae3bb"
Sep 16 18:43:19.940291 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:19.940222    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb506c739d7372c3bb0aa46b5defafb1741ca1df7c68488e0da2a47bd3fae3bb"} err="failed to get container status \"eb506c739d7372c3bb0aa46b5defafb1741ca1df7c68488e0da2a47bd3fae3bb\": rpc error: code = NotFound desc = could not find container \"eb506c739d7372c3bb0aa46b5defafb1741ca1df7c68488e0da2a47bd3fae3bb\": container with ID starting with eb506c739d7372c3bb0aa46b5defafb1741ca1df7c68488e0da2a47bd3fae3bb not found: ID does not exist"
Sep 16 18:43:19.953608 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:19.953588    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fd49c8b6d-t4trc"]
Sep 16 18:43:19.957861 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:19.957840    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7fd49c8b6d-t4trc"]
Sep 16 18:43:20.216301 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:20.216230    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19a2f06e-440f-44dd-b8f8-039862e81971" path="/var/lib/kubelet/pods/19a2f06e-440f-44dd-b8f8-039862e81971/volumes"
Sep 16 18:43:34.182802 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:34.182766    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67d786cbdf-vcw8m"]
Sep 16 18:43:34.183238 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:34.183020    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19a2f06e-440f-44dd-b8f8-039862e81971" containerName="console"
Sep 16 18:43:34.183238 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:34.183032    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="19a2f06e-440f-44dd-b8f8-039862e81971" containerName="console"
Sep 16 18:43:34.183238 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:34.183085    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="19a2f06e-440f-44dd-b8f8-039862e81971" containerName="console"
Sep 16 18:43:34.185220 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:34.185205    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67d786cbdf-vcw8m"
Sep 16 18:43:34.196024 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:34.196000    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67d786cbdf-vcw8m"]
Sep 16 18:43:34.220583 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:34.220560    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48e92cab-8ddd-4b46-9e79-c75c475399d9-trusted-ca-bundle\") pod \"console-67d786cbdf-vcw8m\" (UID: \"48e92cab-8ddd-4b46-9e79-c75c475399d9\") " pod="openshift-console/console-67d786cbdf-vcw8m"
Sep 16 18:43:34.220687 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:34.220593    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bs2d\" (UniqueName: \"kubernetes.io/projected/48e92cab-8ddd-4b46-9e79-c75c475399d9-kube-api-access-7bs2d\") pod \"console-67d786cbdf-vcw8m\" (UID: \"48e92cab-8ddd-4b46-9e79-c75c475399d9\") " pod="openshift-console/console-67d786cbdf-vcw8m"
Sep 16 18:43:34.220687 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:34.220614    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/48e92cab-8ddd-4b46-9e79-c75c475399d9-console-oauth-config\") pod \"console-67d786cbdf-vcw8m\" (UID: \"48e92cab-8ddd-4b46-9e79-c75c475399d9\") " pod="openshift-console/console-67d786cbdf-vcw8m"
Sep 16 18:43:34.220687 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:34.220627    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/48e92cab-8ddd-4b46-9e79-c75c475399d9-service-ca\") pod \"console-67d786cbdf-vcw8m\" (UID: \"48e92cab-8ddd-4b46-9e79-c75c475399d9\") " pod="openshift-console/console-67d786cbdf-vcw8m"
Sep 16 18:43:34.220687 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:34.220651    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/48e92cab-8ddd-4b46-9e79-c75c475399d9-console-serving-cert\") pod \"console-67d786cbdf-vcw8m\" (UID: \"48e92cab-8ddd-4b46-9e79-c75c475399d9\") " pod="openshift-console/console-67d786cbdf-vcw8m"
Sep 16 18:43:34.220817 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:34.220714    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/48e92cab-8ddd-4b46-9e79-c75c475399d9-console-config\") pod \"console-67d786cbdf-vcw8m\" (UID: \"48e92cab-8ddd-4b46-9e79-c75c475399d9\") " pod="openshift-console/console-67d786cbdf-vcw8m"
Sep 16 18:43:34.220817 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:34.220751    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/48e92cab-8ddd-4b46-9e79-c75c475399d9-oauth-serving-cert\") pod \"console-67d786cbdf-vcw8m\" (UID: \"48e92cab-8ddd-4b46-9e79-c75c475399d9\") " pod="openshift-console/console-67d786cbdf-vcw8m"
Sep 16 18:43:34.321231 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:34.321206    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48e92cab-8ddd-4b46-9e79-c75c475399d9-trusted-ca-bundle\") pod \"console-67d786cbdf-vcw8m\" (UID: \"48e92cab-8ddd-4b46-9e79-c75c475399d9\") " pod="openshift-console/console-67d786cbdf-vcw8m"
Sep 16 18:43:34.321333 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:34.321237    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bs2d\" (UniqueName: \"kubernetes.io/projected/48e92cab-8ddd-4b46-9e79-c75c475399d9-kube-api-access-7bs2d\") pod \"console-67d786cbdf-vcw8m\" (UID: \"48e92cab-8ddd-4b46-9e79-c75c475399d9\") " pod="openshift-console/console-67d786cbdf-vcw8m"
Sep 16 18:43:34.321333 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:34.321261    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/48e92cab-8ddd-4b46-9e79-c75c475399d9-console-oauth-config\") pod \"console-67d786cbdf-vcw8m\" (UID: \"48e92cab-8ddd-4b46-9e79-c75c475399d9\") " pod="openshift-console/console-67d786cbdf-vcw8m"
Sep 16 18:43:34.321410 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:34.321385    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/48e92cab-8ddd-4b46-9e79-c75c475399d9-service-ca\") pod \"console-67d786cbdf-vcw8m\" (UID: \"48e92cab-8ddd-4b46-9e79-c75c475399d9\") " pod="openshift-console/console-67d786cbdf-vcw8m"
Sep 16 18:43:34.321445 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:34.321428    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/48e92cab-8ddd-4b46-9e79-c75c475399d9-console-serving-cert\") pod \"console-67d786cbdf-vcw8m\" (UID: \"48e92cab-8ddd-4b46-9e79-c75c475399d9\") " pod="openshift-console/console-67d786cbdf-vcw8m"
Sep 16 18:43:34.321499 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:34.321459    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/48e92cab-8ddd-4b46-9e79-c75c475399d9-console-config\") pod \"console-67d786cbdf-vcw8m\" (UID: \"48e92cab-8ddd-4b46-9e79-c75c475399d9\") " pod="openshift-console/console-67d786cbdf-vcw8m"
Sep 16 18:43:34.321608 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:34.321587    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/48e92cab-8ddd-4b46-9e79-c75c475399d9-oauth-serving-cert\") pod \"console-67d786cbdf-vcw8m\" (UID: \"48e92cab-8ddd-4b46-9e79-c75c475399d9\") " pod="openshift-console/console-67d786cbdf-vcw8m"
Sep 16 18:43:34.322097 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:34.322077    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/48e92cab-8ddd-4b46-9e79-c75c475399d9-console-config\") pod \"console-67d786cbdf-vcw8m\" (UID: \"48e92cab-8ddd-4b46-9e79-c75c475399d9\") " pod="openshift-console/console-67d786cbdf-vcw8m"
Sep 16 18:43:34.322204 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:34.322174    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/48e92cab-8ddd-4b46-9e79-c75c475399d9-service-ca\") pod \"console-67d786cbdf-vcw8m\" (UID: \"48e92cab-8ddd-4b46-9e79-c75c475399d9\") " pod="openshift-console/console-67d786cbdf-vcw8m"
Sep 16 18:43:34.322270 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:34.322237    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/48e92cab-8ddd-4b46-9e79-c75c475399d9-oauth-serving-cert\") pod \"console-67d786cbdf-vcw8m\" (UID: \"48e92cab-8ddd-4b46-9e79-c75c475399d9\") " pod="openshift-console/console-67d786cbdf-vcw8m"
Sep 16 18:43:34.322270 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:34.322249    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48e92cab-8ddd-4b46-9e79-c75c475399d9-trusted-ca-bundle\") pod \"console-67d786cbdf-vcw8m\" (UID: \"48e92cab-8ddd-4b46-9e79-c75c475399d9\") " pod="openshift-console/console-67d786cbdf-vcw8m"
Sep 16 18:43:34.323643 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:34.323617    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/48e92cab-8ddd-4b46-9e79-c75c475399d9-console-oauth-config\") pod \"console-67d786cbdf-vcw8m\" (UID: \"48e92cab-8ddd-4b46-9e79-c75c475399d9\") " pod="openshift-console/console-67d786cbdf-vcw8m"
Sep 16 18:43:34.323832 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:34.323812    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/48e92cab-8ddd-4b46-9e79-c75c475399d9-console-serving-cert\") pod \"console-67d786cbdf-vcw8m\" (UID: \"48e92cab-8ddd-4b46-9e79-c75c475399d9\") " pod="openshift-console/console-67d786cbdf-vcw8m"
Sep 16 18:43:34.330762 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:34.330745    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bs2d\" (UniqueName: \"kubernetes.io/projected/48e92cab-8ddd-4b46-9e79-c75c475399d9-kube-api-access-7bs2d\") pod \"console-67d786cbdf-vcw8m\" (UID: \"48e92cab-8ddd-4b46-9e79-c75c475399d9\") " pod="openshift-console/console-67d786cbdf-vcw8m"
Sep 16 18:43:34.494523 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:34.494468    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67d786cbdf-vcw8m"
Sep 16 18:43:34.764001 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:34.763932    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67d786cbdf-vcw8m"]
Sep 16 18:43:34.766850 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:43:34.766826    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e92cab_8ddd_4b46_9e79_c75c475399d9.slice/crio-49bdcce1bc201e87a86d0413cf976299de9d9353ffdb7af34d72f38117c27b64 WatchSource:0}: Error finding container 49bdcce1bc201e87a86d0413cf976299de9d9353ffdb7af34d72f38117c27b64: Status 404 returned error can't find the container with id 49bdcce1bc201e87a86d0413cf976299de9d9353ffdb7af34d72f38117c27b64
Sep 16 18:43:34.988564 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:34.988529    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67d786cbdf-vcw8m" event={"ID":"48e92cab-8ddd-4b46-9e79-c75c475399d9","Type":"ContainerStarted","Data":"e2d3bd559e20339dafbfa1d5cd0bc7667d1a37f032ec46e6a26843b6b03c57bb"}
Sep 16 18:43:34.988724 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:34.988590    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67d786cbdf-vcw8m" event={"ID":"48e92cab-8ddd-4b46-9e79-c75c475399d9","Type":"ContainerStarted","Data":"49bdcce1bc201e87a86d0413cf976299de9d9353ffdb7af34d72f38117c27b64"}
Sep 16 18:43:35.017388 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:35.015324    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67d786cbdf-vcw8m" podStartSLOduration=1.015305128 podStartE2EDuration="1.015305128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:35.013087118 +0000 UTC m=+183.248020566" watchObservedRunningTime="2026-09-16 18:43:35.015305128 +0000 UTC m=+183.250238573"
Sep 16 18:43:44.495563 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:44.495531    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67d786cbdf-vcw8m"
Sep 16 18:43:44.495949 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:44.495651    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67d786cbdf-vcw8m"
Sep 16 18:43:44.499904 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:44.499886    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67d786cbdf-vcw8m"
Sep 16 18:43:45.025891 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:45.025855    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67d786cbdf-vcw8m"
Sep 16 18:43:45.077776 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:43:45.077751    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74df8dc5ff-6lv5f"]
Sep 16 18:44:10.096469 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:10.096415    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-74df8dc5ff-6lv5f" podUID="25b9ad89-711c-403f-9d5c-8f24f26ec617" containerName="console" containerID="cri-o://aa9733be3e73b34afe3f306a1a0b1e48233a4d9ca2e090dd272552d0678b4063" gracePeriod=15
Sep 16 18:44:10.330516 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:10.330493    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74df8dc5ff-6lv5f"
Sep 16 18:44:10.461854 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:10.461824    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/25b9ad89-711c-403f-9d5c-8f24f26ec617-console-serving-cert\") pod \"25b9ad89-711c-403f-9d5c-8f24f26ec617\" (UID: \"25b9ad89-711c-403f-9d5c-8f24f26ec617\") "
Sep 16 18:44:10.462018 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:10.461891    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/25b9ad89-711c-403f-9d5c-8f24f26ec617-service-ca\") pod \"25b9ad89-711c-403f-9d5c-8f24f26ec617\" (UID: \"25b9ad89-711c-403f-9d5c-8f24f26ec617\") "
Sep 16 18:44:10.462111 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:10.462059    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jh5zm\" (UniqueName: \"kubernetes.io/projected/25b9ad89-711c-403f-9d5c-8f24f26ec617-kube-api-access-jh5zm\") pod \"25b9ad89-711c-403f-9d5c-8f24f26ec617\" (UID: \"25b9ad89-711c-403f-9d5c-8f24f26ec617\") "
Sep 16 18:44:10.462111 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:10.462088    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25b9ad89-711c-403f-9d5c-8f24f26ec617-trusted-ca-bundle\") pod \"25b9ad89-711c-403f-9d5c-8f24f26ec617\" (UID: \"25b9ad89-711c-403f-9d5c-8f24f26ec617\") "
Sep 16 18:44:10.462224 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:10.462118    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/25b9ad89-711c-403f-9d5c-8f24f26ec617-console-oauth-config\") pod \"25b9ad89-711c-403f-9d5c-8f24f26ec617\" (UID: \"25b9ad89-711c-403f-9d5c-8f24f26ec617\") "
Sep 16 18:44:10.462224 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:10.462141    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/25b9ad89-711c-403f-9d5c-8f24f26ec617-oauth-serving-cert\") pod \"25b9ad89-711c-403f-9d5c-8f24f26ec617\" (UID: \"25b9ad89-711c-403f-9d5c-8f24f26ec617\") "
Sep 16 18:44:10.462224 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:10.462170    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/25b9ad89-711c-403f-9d5c-8f24f26ec617-console-config\") pod \"25b9ad89-711c-403f-9d5c-8f24f26ec617\" (UID: \"25b9ad89-711c-403f-9d5c-8f24f26ec617\") "
Sep 16 18:44:10.462374 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:10.462239    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25b9ad89-711c-403f-9d5c-8f24f26ec617-service-ca" (OuterVolumeSpecName: "service-ca") pod "25b9ad89-711c-403f-9d5c-8f24f26ec617" (UID: "25b9ad89-711c-403f-9d5c-8f24f26ec617"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:10.462433 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:10.462371    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/25b9ad89-711c-403f-9d5c-8f24f26ec617-service-ca\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:44:10.462518 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:10.462501    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25b9ad89-711c-403f-9d5c-8f24f26ec617-console-config" (OuterVolumeSpecName: "console-config") pod "25b9ad89-711c-403f-9d5c-8f24f26ec617" (UID: "25b9ad89-711c-403f-9d5c-8f24f26ec617"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:10.462573 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:10.462519    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25b9ad89-711c-403f-9d5c-8f24f26ec617-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "25b9ad89-711c-403f-9d5c-8f24f26ec617" (UID: "25b9ad89-711c-403f-9d5c-8f24f26ec617"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:10.462573 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:10.462494    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25b9ad89-711c-403f-9d5c-8f24f26ec617-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "25b9ad89-711c-403f-9d5c-8f24f26ec617" (UID: "25b9ad89-711c-403f-9d5c-8f24f26ec617"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:10.463965 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:10.463935    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b9ad89-711c-403f-9d5c-8f24f26ec617-kube-api-access-jh5zm" (OuterVolumeSpecName: "kube-api-access-jh5zm") pod "25b9ad89-711c-403f-9d5c-8f24f26ec617" (UID: "25b9ad89-711c-403f-9d5c-8f24f26ec617"). InnerVolumeSpecName "kube-api-access-jh5zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:44:10.464354 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:10.464330    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25b9ad89-711c-403f-9d5c-8f24f26ec617-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "25b9ad89-711c-403f-9d5c-8f24f26ec617" (UID: "25b9ad89-711c-403f-9d5c-8f24f26ec617"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:10.464354 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:10.464349    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25b9ad89-711c-403f-9d5c-8f24f26ec617-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "25b9ad89-711c-403f-9d5c-8f24f26ec617" (UID: "25b9ad89-711c-403f-9d5c-8f24f26ec617"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:10.562717 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:10.562692    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/25b9ad89-711c-403f-9d5c-8f24f26ec617-console-oauth-config\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:44:10.562717 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:10.562716    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/25b9ad89-711c-403f-9d5c-8f24f26ec617-oauth-serving-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:44:10.562849 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:10.562730    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/25b9ad89-711c-403f-9d5c-8f24f26ec617-console-config\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:44:10.562849 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:10.562743    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/25b9ad89-711c-403f-9d5c-8f24f26ec617-console-serving-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:44:10.562849 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:10.562756    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh5zm\" (UniqueName: \"kubernetes.io/projected/25b9ad89-711c-403f-9d5c-8f24f26ec617-kube-api-access-jh5zm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:44:10.562849 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:10.562769    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25b9ad89-711c-403f-9d5c-8f24f26ec617-trusted-ca-bundle\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:44:11.113456 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:11.113423    2578 generic.go:356] "Generic (PLEG): container finished" podID="25b9ad89-711c-403f-9d5c-8f24f26ec617" containerID="aa9733be3e73b34afe3f306a1a0b1e48233a4d9ca2e090dd272552d0678b4063" exitCode=0
Sep 16 18:44:11.113847 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:11.113466    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74df8dc5ff-6lv5f"
Sep 16 18:44:11.113847 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:11.113497    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74df8dc5ff-6lv5f" event={"ID":"25b9ad89-711c-403f-9d5c-8f24f26ec617","Type":"ContainerDied","Data":"aa9733be3e73b34afe3f306a1a0b1e48233a4d9ca2e090dd272552d0678b4063"}
Sep 16 18:44:11.113847 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:11.113543    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74df8dc5ff-6lv5f" event={"ID":"25b9ad89-711c-403f-9d5c-8f24f26ec617","Type":"ContainerDied","Data":"b234b72864f3313f4807b5f78002341e8ae9ecb0848b31c0d8d5e87b9af203b9"}
Sep 16 18:44:11.113847 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:11.113563    2578 scope.go:117] "RemoveContainer" containerID="aa9733be3e73b34afe3f306a1a0b1e48233a4d9ca2e090dd272552d0678b4063"
Sep 16 18:44:11.121904 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:11.121886    2578 scope.go:117] "RemoveContainer" containerID="aa9733be3e73b34afe3f306a1a0b1e48233a4d9ca2e090dd272552d0678b4063"
Sep 16 18:44:11.122179 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:44:11.122161    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa9733be3e73b34afe3f306a1a0b1e48233a4d9ca2e090dd272552d0678b4063\": container with ID starting with aa9733be3e73b34afe3f306a1a0b1e48233a4d9ca2e090dd272552d0678b4063 not found: ID does not exist" containerID="aa9733be3e73b34afe3f306a1a0b1e48233a4d9ca2e090dd272552d0678b4063"
Sep 16 18:44:11.122233 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:11.122189    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa9733be3e73b34afe3f306a1a0b1e48233a4d9ca2e090dd272552d0678b4063"} err="failed to get container status \"aa9733be3e73b34afe3f306a1a0b1e48233a4d9ca2e090dd272552d0678b4063\": rpc error: code = NotFound desc = could not find container \"aa9733be3e73b34afe3f306a1a0b1e48233a4d9ca2e090dd272552d0678b4063\": container with ID starting with aa9733be3e73b34afe3f306a1a0b1e48233a4d9ca2e090dd272552d0678b4063 not found: ID does not exist"
Sep 16 18:44:11.138047 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:11.138023    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74df8dc5ff-6lv5f"]
Sep 16 18:44:11.142347 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:11.142328    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74df8dc5ff-6lv5f"]
Sep 16 18:44:12.215924 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:12.215892    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25b9ad89-711c-403f-9d5c-8f24f26ec617" path="/var/lib/kubelet/pods/25b9ad89-711c-403f-9d5c-8f24f26ec617/volumes"
Sep 16 18:44:52.980371 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:52.980332    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ajcjcc"]
Sep 16 18:44:52.980796 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:52.980581    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25b9ad89-711c-403f-9d5c-8f24f26ec617" containerName="console"
Sep 16 18:44:52.980796 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:52.980592    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="25b9ad89-711c-403f-9d5c-8f24f26ec617" containerName="console"
Sep 16 18:44:52.980796 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:52.980637    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="25b9ad89-711c-403f-9d5c-8f24f26ec617" containerName="console"
Sep 16 18:44:52.982857 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:52.982840    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ajcjcc"
Sep 16 18:44:52.986793 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:52.986766    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 18:44:52.987069 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:52.987028    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 18:44:52.987910 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:52.987891    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6zvtq\""
Sep 16 18:44:52.994080 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:52.993994    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ajcjcc"]
Sep 16 18:44:53.135566 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:53.135535    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44c20595-1eda-46e8-b978-3bb217284cd3-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ajcjcc\" (UID: \"44c20595-1eda-46e8-b978-3bb217284cd3\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ajcjcc"
Sep 16 18:44:53.135734 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:53.135572    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp5cn\" (UniqueName: \"kubernetes.io/projected/44c20595-1eda-46e8-b978-3bb217284cd3-kube-api-access-bp5cn\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ajcjcc\" (UID: \"44c20595-1eda-46e8-b978-3bb217284cd3\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ajcjcc"
Sep 16 18:44:53.135734 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:53.135626    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44c20595-1eda-46e8-b978-3bb217284cd3-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ajcjcc\" (UID: \"44c20595-1eda-46e8-b978-3bb217284cd3\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ajcjcc"
Sep 16 18:44:53.236073 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:53.235990    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44c20595-1eda-46e8-b978-3bb217284cd3-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ajcjcc\" (UID: \"44c20595-1eda-46e8-b978-3bb217284cd3\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ajcjcc"
Sep 16 18:44:53.236073 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:53.236020    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp5cn\" (UniqueName: \"kubernetes.io/projected/44c20595-1eda-46e8-b978-3bb217284cd3-kube-api-access-bp5cn\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ajcjcc\" (UID: \"44c20595-1eda-46e8-b978-3bb217284cd3\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ajcjcc"
Sep 16 18:44:53.236073 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:53.236062    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44c20595-1eda-46e8-b978-3bb217284cd3-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ajcjcc\" (UID: \"44c20595-1eda-46e8-b978-3bb217284cd3\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ajcjcc"
Sep 16 18:44:53.236327 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:53.236311    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44c20595-1eda-46e8-b978-3bb217284cd3-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ajcjcc\" (UID: \"44c20595-1eda-46e8-b978-3bb217284cd3\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ajcjcc"
Sep 16 18:44:53.236393 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:53.236335    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44c20595-1eda-46e8-b978-3bb217284cd3-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ajcjcc\" (UID: \"44c20595-1eda-46e8-b978-3bb217284cd3\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ajcjcc"
Sep 16 18:44:53.246469 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:53.246445    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp5cn\" (UniqueName: \"kubernetes.io/projected/44c20595-1eda-46e8-b978-3bb217284cd3-kube-api-access-bp5cn\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ajcjcc\" (UID: \"44c20595-1eda-46e8-b978-3bb217284cd3\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ajcjcc"
Sep 16 18:44:53.291820 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:53.291799    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ajcjcc"
Sep 16 18:44:53.564186 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:53.564120    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ajcjcc"]
Sep 16 18:44:53.567981 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:44:53.567950    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c20595_1eda_46e8_b978_3bb217284cd3.slice/crio-2c0aec0b68b98e74f0fd07c40091f3f9ead346d5ebf4a2efeb87bd9a01966291 WatchSource:0}: Error finding container 2c0aec0b68b98e74f0fd07c40091f3f9ead346d5ebf4a2efeb87bd9a01966291: Status 404 returned error can't find the container with id 2c0aec0b68b98e74f0fd07c40091f3f9ead346d5ebf4a2efeb87bd9a01966291
Sep 16 18:44:54.257850 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:44:54.257817    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ajcjcc" event={"ID":"44c20595-1eda-46e8-b978-3bb217284cd3","Type":"ContainerStarted","Data":"2c0aec0b68b98e74f0fd07c40091f3f9ead346d5ebf4a2efeb87bd9a01966291"}
Sep 16 18:45:00.279164 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:00.279129    2578 generic.go:356] "Generic (PLEG): container finished" podID="44c20595-1eda-46e8-b978-3bb217284cd3" containerID="2eec6c56737ba4482264f8897cdbba1e47b919dcfeb6df89b57fbb8f37784e14" exitCode=0
Sep 16 18:45:00.279514 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:00.279204    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ajcjcc" event={"ID":"44c20595-1eda-46e8-b978-3bb217284cd3","Type":"ContainerDied","Data":"2eec6c56737ba4482264f8897cdbba1e47b919dcfeb6df89b57fbb8f37784e14"}
Sep 16 18:45:02.294763 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:02.294732    2578 generic.go:356] "Generic (PLEG): container finished" podID="44c20595-1eda-46e8-b978-3bb217284cd3" containerID="fa5b7f4edda1acdfeee94196d794045cd98602d6cff30e457d9742b9980cc49e" exitCode=0
Sep 16 18:45:02.295084 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:02.294786    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ajcjcc" event={"ID":"44c20595-1eda-46e8-b978-3bb217284cd3","Type":"ContainerDied","Data":"fa5b7f4edda1acdfeee94196d794045cd98602d6cff30e457d9742b9980cc49e"}
Sep 16 18:45:10.328324 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:10.328293    2578 generic.go:356] "Generic (PLEG): container finished" podID="44c20595-1eda-46e8-b978-3bb217284cd3" containerID="1a993199bebd94006f01dc1b7d554be425e4d3ac46199651f0d63fa8a1318bdc" exitCode=0
Sep 16 18:45:10.328736 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:10.328345    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ajcjcc" event={"ID":"44c20595-1eda-46e8-b978-3bb217284cd3","Type":"ContainerDied","Data":"1a993199bebd94006f01dc1b7d554be425e4d3ac46199651f0d63fa8a1318bdc"}
Sep 16 18:45:11.451090 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:11.451068    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ajcjcc"
Sep 16 18:45:11.566688 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:11.566665    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44c20595-1eda-46e8-b978-3bb217284cd3-util\") pod \"44c20595-1eda-46e8-b978-3bb217284cd3\" (UID: \"44c20595-1eda-46e8-b978-3bb217284cd3\") "
Sep 16 18:45:11.566780 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:11.566698    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bp5cn\" (UniqueName: \"kubernetes.io/projected/44c20595-1eda-46e8-b978-3bb217284cd3-kube-api-access-bp5cn\") pod \"44c20595-1eda-46e8-b978-3bb217284cd3\" (UID: \"44c20595-1eda-46e8-b978-3bb217284cd3\") "
Sep 16 18:45:11.566780 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:11.566767    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44c20595-1eda-46e8-b978-3bb217284cd3-bundle\") pod \"44c20595-1eda-46e8-b978-3bb217284cd3\" (UID: \"44c20595-1eda-46e8-b978-3bb217284cd3\") "
Sep 16 18:45:11.567404 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:11.567381    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c20595-1eda-46e8-b978-3bb217284cd3-bundle" (OuterVolumeSpecName: "bundle") pod "44c20595-1eda-46e8-b978-3bb217284cd3" (UID: "44c20595-1eda-46e8-b978-3bb217284cd3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:45:11.568721 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:11.568699    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c20595-1eda-46e8-b978-3bb217284cd3-kube-api-access-bp5cn" (OuterVolumeSpecName: "kube-api-access-bp5cn") pod "44c20595-1eda-46e8-b978-3bb217284cd3" (UID: "44c20595-1eda-46e8-b978-3bb217284cd3"). InnerVolumeSpecName "kube-api-access-bp5cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:45:11.571146 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:11.571126    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44c20595-1eda-46e8-b978-3bb217284cd3-util" (OuterVolumeSpecName: "util") pod "44c20595-1eda-46e8-b978-3bb217284cd3" (UID: "44c20595-1eda-46e8-b978-3bb217284cd3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:45:11.667222 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:11.667198    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44c20595-1eda-46e8-b978-3bb217284cd3-bundle\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:45:11.667222 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:11.667221    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44c20595-1eda-46e8-b978-3bb217284cd3-util\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:45:11.667351 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:11.667234    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp5cn\" (UniqueName: \"kubernetes.io/projected/44c20595-1eda-46e8-b978-3bb217284cd3-kube-api-access-bp5cn\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:45:12.337368 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:12.337336    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ajcjcc" event={"ID":"44c20595-1eda-46e8-b978-3bb217284cd3","Type":"ContainerDied","Data":"2c0aec0b68b98e74f0fd07c40091f3f9ead346d5ebf4a2efeb87bd9a01966291"}
Sep 16 18:45:12.337368 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:12.337360    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ajcjcc"
Sep 16 18:45:12.337515 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:12.337376    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c0aec0b68b98e74f0fd07c40091f3f9ead346d5ebf4a2efeb87bd9a01966291"
Sep 16 18:45:15.780526 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:15.780497    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-6l6cs"]
Sep 16 18:45:15.780894 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:15.780713    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44c20595-1eda-46e8-b978-3bb217284cd3" containerName="util"
Sep 16 18:45:15.780894 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:15.780723    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="44c20595-1eda-46e8-b978-3bb217284cd3" containerName="util"
Sep 16 18:45:15.780894 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:15.780731    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44c20595-1eda-46e8-b978-3bb217284cd3" containerName="pull"
Sep 16 18:45:15.780894 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:15.780737    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="44c20595-1eda-46e8-b978-3bb217284cd3" containerName="pull"
Sep 16 18:45:15.780894 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:15.780745    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44c20595-1eda-46e8-b978-3bb217284cd3" containerName="extract"
Sep 16 18:45:15.780894 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:15.780750    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="44c20595-1eda-46e8-b978-3bb217284cd3" containerName="extract"
Sep 16 18:45:15.780894 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:15.780797    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="44c20595-1eda-46e8-b978-3bb217284cd3" containerName="extract"
Sep 16 18:45:15.826231 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:15.826206    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-6l6cs"]
Sep 16 18:45:15.826343 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:15.826294    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-6l6cs"
Sep 16 18:45:15.830137 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:15.830115    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 16 18:45:15.830259 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:15.830177    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-46rht\""
Sep 16 18:45:15.830339 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:15.830259    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 16 18:45:15.830499 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:15.830484    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 16 18:45:15.894767 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:15.894742    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhzdm\" (UniqueName: \"kubernetes.io/projected/e5fcfc13-1c16-48ae-930e-72552ff1330b-kube-api-access-rhzdm\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-6l6cs\" (UID: \"e5fcfc13-1c16-48ae-930e-72552ff1330b\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-6l6cs"
Sep 16 18:45:15.894879 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:15.894778    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/e5fcfc13-1c16-48ae-930e-72552ff1330b-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-6l6cs\" (UID: \"e5fcfc13-1c16-48ae-930e-72552ff1330b\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-6l6cs"
Sep 16 18:45:15.995375 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:15.995350    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhzdm\" (UniqueName: \"kubernetes.io/projected/e5fcfc13-1c16-48ae-930e-72552ff1330b-kube-api-access-rhzdm\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-6l6cs\" (UID: \"e5fcfc13-1c16-48ae-930e-72552ff1330b\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-6l6cs"
Sep 16 18:45:15.995465 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:15.995383    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/e5fcfc13-1c16-48ae-930e-72552ff1330b-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-6l6cs\" (UID: \"e5fcfc13-1c16-48ae-930e-72552ff1330b\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-6l6cs"
Sep 16 18:45:15.997487 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:15.997469    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/e5fcfc13-1c16-48ae-930e-72552ff1330b-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-6l6cs\" (UID: \"e5fcfc13-1c16-48ae-930e-72552ff1330b\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-6l6cs"
Sep 16 18:45:16.005626 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:16.005609    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhzdm\" (UniqueName: \"kubernetes.io/projected/e5fcfc13-1c16-48ae-930e-72552ff1330b-kube-api-access-rhzdm\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-6l6cs\" (UID: \"e5fcfc13-1c16-48ae-930e-72552ff1330b\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-6l6cs"
Sep 16 18:45:16.135887 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:16.135837    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-6l6cs"
Sep 16 18:45:16.432763 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:16.432729    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-6l6cs"]
Sep 16 18:45:16.433992 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:45:16.433969    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5fcfc13_1c16_48ae_930e_72552ff1330b.slice/crio-06b3c5cf93a8b9dd71d1254a62324bf4a1602be51d84cda02df8f0727565e408 WatchSource:0}: Error finding container 06b3c5cf93a8b9dd71d1254a62324bf4a1602be51d84cda02df8f0727565e408: Status 404 returned error can't find the container with id 06b3c5cf93a8b9dd71d1254a62324bf4a1602be51d84cda02df8f0727565e408
Sep 16 18:45:17.354758 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:17.354721    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-6l6cs" event={"ID":"e5fcfc13-1c16-48ae-930e-72552ff1330b","Type":"ContainerStarted","Data":"06b3c5cf93a8b9dd71d1254a62324bf4a1602be51d84cda02df8f0727565e408"}
Sep 16 18:45:20.367202 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:20.367164    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-6l6cs" event={"ID":"e5fcfc13-1c16-48ae-930e-72552ff1330b","Type":"ContainerStarted","Data":"0370f37ef4170d0191c884af1fa91bdb7fd99c61c42344266de030cc23091700"}
Sep 16 18:45:20.367593 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:20.367290    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-6l6cs"
Sep 16 18:45:20.388732 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:20.388688    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-6l6cs" podStartSLOduration=1.9111542479999999 podStartE2EDuration="5.388674105s" totalImagesPullingTime="3.477519857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:20.388235973 +0000 UTC m=+288.623169418" watchObservedRunningTime="2026-09-16 18:45:20.388674105 +0000 UTC m=+288.623607549"
Sep 16 18:45:20.687476 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:20.687446    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-2t458"]
Sep 16 18:45:20.708574 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:20.708550    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-2t458"]
Sep 16 18:45:20.708717 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:20.708656    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-2t458"
Sep 16 18:45:20.711410 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:20.711291    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 16 18:45:20.711410 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:20.711293    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-c9wjp\""
Sep 16 18:45:20.711621 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:20.711462    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 16 18:45:20.726416 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:20.726398    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdfrp\" (UniqueName: \"kubernetes.io/projected/04896ead-41f2-40a9-b5aa-1b4529703ec2-kube-api-access-gdfrp\") pod \"keda-operator-579c8958f7-2t458\" (UID: \"04896ead-41f2-40a9-b5aa-1b4529703ec2\") " pod="openshift-keda/keda-operator-579c8958f7-2t458"
Sep 16 18:45:20.726506 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:20.726443    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/04896ead-41f2-40a9-b5aa-1b4529703ec2-cabundle0\") pod \"keda-operator-579c8958f7-2t458\" (UID: \"04896ead-41f2-40a9-b5aa-1b4529703ec2\") " pod="openshift-keda/keda-operator-579c8958f7-2t458"
Sep 16 18:45:20.726506 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:20.726473    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/04896ead-41f2-40a9-b5aa-1b4529703ec2-certificates\") pod \"keda-operator-579c8958f7-2t458\" (UID: \"04896ead-41f2-40a9-b5aa-1b4529703ec2\") " pod="openshift-keda/keda-operator-579c8958f7-2t458"
Sep 16 18:45:20.827760 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:20.827731    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/04896ead-41f2-40a9-b5aa-1b4529703ec2-cabundle0\") pod \"keda-operator-579c8958f7-2t458\" (UID: \"04896ead-41f2-40a9-b5aa-1b4529703ec2\") " pod="openshift-keda/keda-operator-579c8958f7-2t458"
Sep 16 18:45:20.827874 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:20.827781    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/04896ead-41f2-40a9-b5aa-1b4529703ec2-certificates\") pod \"keda-operator-579c8958f7-2t458\" (UID: \"04896ead-41f2-40a9-b5aa-1b4529703ec2\") " pod="openshift-keda/keda-operator-579c8958f7-2t458"
Sep 16 18:45:20.827874 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:20.827812    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdfrp\" (UniqueName: \"kubernetes.io/projected/04896ead-41f2-40a9-b5aa-1b4529703ec2-kube-api-access-gdfrp\") pod \"keda-operator-579c8958f7-2t458\" (UID: \"04896ead-41f2-40a9-b5aa-1b4529703ec2\") " pod="openshift-keda/keda-operator-579c8958f7-2t458"
Sep 16 18:45:20.827984 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:45:20.827961    2578 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Sep 16 18:45:20.828062 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:45:20.827985    2578 secret.go:281] references non-existent secret key: ca.crt
Sep 16 18:45:20.828062 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:45:20.827997    2578 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 16 18:45:20.828062 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:45:20.828014    2578 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-2t458: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 16 18:45:20.828205 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:45:20.828099    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04896ead-41f2-40a9-b5aa-1b4529703ec2-certificates podName:04896ead-41f2-40a9-b5aa-1b4529703ec2 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:21.328078303 +0000 UTC m=+289.563011745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/04896ead-41f2-40a9-b5aa-1b4529703ec2-certificates") pod "keda-operator-579c8958f7-2t458" (UID: "04896ead-41f2-40a9-b5aa-1b4529703ec2") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 16 18:45:20.828318 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:20.828298    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/04896ead-41f2-40a9-b5aa-1b4529703ec2-cabundle0\") pod \"keda-operator-579c8958f7-2t458\" (UID: \"04896ead-41f2-40a9-b5aa-1b4529703ec2\") " pod="openshift-keda/keda-operator-579c8958f7-2t458"
Sep 16 18:45:20.838104 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:20.838083    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdfrp\" (UniqueName: \"kubernetes.io/projected/04896ead-41f2-40a9-b5aa-1b4529703ec2-kube-api-access-gdfrp\") pod \"keda-operator-579c8958f7-2t458\" (UID: \"04896ead-41f2-40a9-b5aa-1b4529703ec2\") " pod="openshift-keda/keda-operator-579c8958f7-2t458"
Sep 16 18:45:21.332960 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:21.332926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/04896ead-41f2-40a9-b5aa-1b4529703ec2-certificates\") pod \"keda-operator-579c8958f7-2t458\" (UID: \"04896ead-41f2-40a9-b5aa-1b4529703ec2\") " pod="openshift-keda/keda-operator-579c8958f7-2t458"
Sep 16 18:45:21.335470 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:21.335436    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/04896ead-41f2-40a9-b5aa-1b4529703ec2-certificates\") pod \"keda-operator-579c8958f7-2t458\" (UID: \"04896ead-41f2-40a9-b5aa-1b4529703ec2\") " pod="openshift-keda/keda-operator-579c8958f7-2t458"
Sep 16 18:45:21.619164 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:21.619087    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-2t458"
Sep 16 18:45:21.903196 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:21.903119    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-2t458"]
Sep 16 18:45:21.908537 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:45:21.908509    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04896ead_41f2_40a9_b5aa_1b4529703ec2.slice/crio-5ee5929264931c8c2018d7d937810de3edf9fb95168be2ba0d194084da139df5 WatchSource:0}: Error finding container 5ee5929264931c8c2018d7d937810de3edf9fb95168be2ba0d194084da139df5: Status 404 returned error can't find the container with id 5ee5929264931c8c2018d7d937810de3edf9fb95168be2ba0d194084da139df5
Sep 16 18:45:22.376264 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:22.376229    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-2t458" event={"ID":"04896ead-41f2-40a9-b5aa-1b4529703ec2","Type":"ContainerStarted","Data":"5ee5929264931c8c2018d7d937810de3edf9fb95168be2ba0d194084da139df5"}
Sep 16 18:45:25.388432 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:25.388397    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-2t458" event={"ID":"04896ead-41f2-40a9-b5aa-1b4529703ec2","Type":"ContainerStarted","Data":"2aa2ba3ee4e29194be98f4e09cd147cdbbc516ee5434d8cc6a73b89d3f114f12"}
Sep 16 18:45:25.388783 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:25.388543    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-2t458"
Sep 16 18:45:25.406313 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:25.406268    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-2t458" podStartSLOduration=2.572592202 podStartE2EDuration="5.4062537s" totalImagesPullingTime="2.833661498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:25.405298457 +0000 UTC m=+293.640231901" watchObservedRunningTime="2026-09-16 18:45:25.4062537 +0000 UTC m=+293.641187173"
Sep 16 18:45:32.142498 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:32.141488    2578 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:45:41.374030 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:41.373997    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-6l6cs"
Sep 16 18:45:46.394509 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:45:46.394474    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-2t458"
Sep 16 18:46:13.456873 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:13.456842    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cwlmx"]
Sep 16 18:46:13.461074 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:13.461024    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cwlmx"
Sep 16 18:46:13.465495 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:13.465473    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 18:46:13.465621 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:13.465474    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 18:46:13.466619 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:13.466598    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6zvtq\""
Sep 16 18:46:13.474639 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:13.474619    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cwlmx"]
Sep 16 18:46:13.588077 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:13.588051    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea46658e-64ec-41a5-9d4b-c7389f5f84a3-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cwlmx\" (UID: \"ea46658e-64ec-41a5-9d4b-c7389f5f84a3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cwlmx"
Sep 16 18:46:13.588161 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:13.588096    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea46658e-64ec-41a5-9d4b-c7389f5f84a3-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cwlmx\" (UID: \"ea46658e-64ec-41a5-9d4b-c7389f5f84a3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cwlmx"
Sep 16 18:46:13.588161 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:13.588128    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75kh5\" (UniqueName: \"kubernetes.io/projected/ea46658e-64ec-41a5-9d4b-c7389f5f84a3-kube-api-access-75kh5\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cwlmx\" (UID: \"ea46658e-64ec-41a5-9d4b-c7389f5f84a3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cwlmx"
Sep 16 18:46:13.688889 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:13.688856    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea46658e-64ec-41a5-9d4b-c7389f5f84a3-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cwlmx\" (UID: \"ea46658e-64ec-41a5-9d4b-c7389f5f84a3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cwlmx"
Sep 16 18:46:13.688974 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:13.688912    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea46658e-64ec-41a5-9d4b-c7389f5f84a3-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cwlmx\" (UID: \"ea46658e-64ec-41a5-9d4b-c7389f5f84a3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cwlmx"
Sep 16 18:46:13.688974 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:13.688940    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75kh5\" (UniqueName: \"kubernetes.io/projected/ea46658e-64ec-41a5-9d4b-c7389f5f84a3-kube-api-access-75kh5\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cwlmx\" (UID: \"ea46658e-64ec-41a5-9d4b-c7389f5f84a3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cwlmx"
Sep 16 18:46:13.689293 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:13.689272    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea46658e-64ec-41a5-9d4b-c7389f5f84a3-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cwlmx\" (UID: \"ea46658e-64ec-41a5-9d4b-c7389f5f84a3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cwlmx"
Sep 16 18:46:13.689334 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:13.689285    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea46658e-64ec-41a5-9d4b-c7389f5f84a3-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cwlmx\" (UID: \"ea46658e-64ec-41a5-9d4b-c7389f5f84a3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cwlmx"
Sep 16 18:46:13.699263 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:13.699238    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75kh5\" (UniqueName: \"kubernetes.io/projected/ea46658e-64ec-41a5-9d4b-c7389f5f84a3-kube-api-access-75kh5\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cwlmx\" (UID: \"ea46658e-64ec-41a5-9d4b-c7389f5f84a3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cwlmx"
Sep 16 18:46:13.769688 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:13.769638    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cwlmx"
Sep 16 18:46:14.041473 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:14.041400    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cwlmx"]
Sep 16 18:46:14.044429 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:46:14.044404    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea46658e_64ec_41a5_9d4b_c7389f5f84a3.slice/crio-aa00df177640331d95e6b2fbdeae94b0e05b2f6117562484175b202ee8cf9522 WatchSource:0}: Error finding container aa00df177640331d95e6b2fbdeae94b0e05b2f6117562484175b202ee8cf9522: Status 404 returned error can't find the container with id aa00df177640331d95e6b2fbdeae94b0e05b2f6117562484175b202ee8cf9522
Sep 16 18:46:14.558727 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:14.558697    2578 generic.go:356] "Generic (PLEG): container finished" podID="ea46658e-64ec-41a5-9d4b-c7389f5f84a3" containerID="79a61a90e04ae8c5e0563d373db79cc9c0bc35dadd1746820d2b43874b290f1f" exitCode=0
Sep 16 18:46:14.559191 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:14.558738    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cwlmx" event={"ID":"ea46658e-64ec-41a5-9d4b-c7389f5f84a3","Type":"ContainerDied","Data":"79a61a90e04ae8c5e0563d373db79cc9c0bc35dadd1746820d2b43874b290f1f"}
Sep 16 18:46:14.559191 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:14.558770    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cwlmx" event={"ID":"ea46658e-64ec-41a5-9d4b-c7389f5f84a3","Type":"ContainerStarted","Data":"aa00df177640331d95e6b2fbdeae94b0e05b2f6117562484175b202ee8cf9522"}
Sep 16 18:46:14.559864 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:14.559846    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:46:17.573032 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:17.572994    2578 generic.go:356] "Generic (PLEG): container finished" podID="ea46658e-64ec-41a5-9d4b-c7389f5f84a3" containerID="c339ede6c46712df929dabae050c05a4165e57ff569a1a0f8c84c58fa8cf194c" exitCode=0
Sep 16 18:46:17.573431 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:17.573067    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cwlmx" event={"ID":"ea46658e-64ec-41a5-9d4b-c7389f5f84a3","Type":"ContainerDied","Data":"c339ede6c46712df929dabae050c05a4165e57ff569a1a0f8c84c58fa8cf194c"}
Sep 16 18:46:18.578905 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:18.578869    2578 generic.go:356] "Generic (PLEG): container finished" podID="ea46658e-64ec-41a5-9d4b-c7389f5f84a3" containerID="93f797a80468db8f1a930f9e1f010f683514d3d65850d4525907c2e4a12913a2" exitCode=0
Sep 16 18:46:18.579315 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:18.578949    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cwlmx" event={"ID":"ea46658e-64ec-41a5-9d4b-c7389f5f84a3","Type":"ContainerDied","Data":"93f797a80468db8f1a930f9e1f010f683514d3d65850d4525907c2e4a12913a2"}
Sep 16 18:46:19.699123 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:19.699102    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cwlmx"
Sep 16 18:46:19.829129 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:19.829089    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75kh5\" (UniqueName: \"kubernetes.io/projected/ea46658e-64ec-41a5-9d4b-c7389f5f84a3-kube-api-access-75kh5\") pod \"ea46658e-64ec-41a5-9d4b-c7389f5f84a3\" (UID: \"ea46658e-64ec-41a5-9d4b-c7389f5f84a3\") "
Sep 16 18:46:19.829261 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:19.829148    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea46658e-64ec-41a5-9d4b-c7389f5f84a3-util\") pod \"ea46658e-64ec-41a5-9d4b-c7389f5f84a3\" (UID: \"ea46658e-64ec-41a5-9d4b-c7389f5f84a3\") "
Sep 16 18:46:19.829261 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:19.829179    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea46658e-64ec-41a5-9d4b-c7389f5f84a3-bundle\") pod \"ea46658e-64ec-41a5-9d4b-c7389f5f84a3\" (UID: \"ea46658e-64ec-41a5-9d4b-c7389f5f84a3\") "
Sep 16 18:46:19.829841 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:19.829811    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea46658e-64ec-41a5-9d4b-c7389f5f84a3-bundle" (OuterVolumeSpecName: "bundle") pod "ea46658e-64ec-41a5-9d4b-c7389f5f84a3" (UID: "ea46658e-64ec-41a5-9d4b-c7389f5f84a3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:19.831034 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:19.831014    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea46658e-64ec-41a5-9d4b-c7389f5f84a3-kube-api-access-75kh5" (OuterVolumeSpecName: "kube-api-access-75kh5") pod "ea46658e-64ec-41a5-9d4b-c7389f5f84a3" (UID: "ea46658e-64ec-41a5-9d4b-c7389f5f84a3"). InnerVolumeSpecName "kube-api-access-75kh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:46:19.834123 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:19.834078    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea46658e-64ec-41a5-9d4b-c7389f5f84a3-util" (OuterVolumeSpecName: "util") pod "ea46658e-64ec-41a5-9d4b-c7389f5f84a3" (UID: "ea46658e-64ec-41a5-9d4b-c7389f5f84a3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:19.930530 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:19.930503    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75kh5\" (UniqueName: \"kubernetes.io/projected/ea46658e-64ec-41a5-9d4b-c7389f5f84a3-kube-api-access-75kh5\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:46:19.930530 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:19.930528    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea46658e-64ec-41a5-9d4b-c7389f5f84a3-util\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:46:19.930530 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:19.930538    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea46658e-64ec-41a5-9d4b-c7389f5f84a3-bundle\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:46:20.588701 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:20.588626    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cwlmx"
Sep 16 18:46:20.588821 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:20.588628    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cwlmx" event={"ID":"ea46658e-64ec-41a5-9d4b-c7389f5f84a3","Type":"ContainerDied","Data":"aa00df177640331d95e6b2fbdeae94b0e05b2f6117562484175b202ee8cf9522"}
Sep 16 18:46:20.588821 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:20.588763    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa00df177640331d95e6b2fbdeae94b0e05b2f6117562484175b202ee8cf9522"
Sep 16 18:46:26.374412 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:26.374341    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qwbtz"]
Sep 16 18:46:26.374744 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:26.374572    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea46658e-64ec-41a5-9d4b-c7389f5f84a3" containerName="util"
Sep 16 18:46:26.374744 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:26.374583    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea46658e-64ec-41a5-9d4b-c7389f5f84a3" containerName="util"
Sep 16 18:46:26.374744 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:26.374592    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea46658e-64ec-41a5-9d4b-c7389f5f84a3" containerName="pull"
Sep 16 18:46:26.374744 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:26.374597    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea46658e-64ec-41a5-9d4b-c7389f5f84a3" containerName="pull"
Sep 16 18:46:26.374744 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:26.374606    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea46658e-64ec-41a5-9d4b-c7389f5f84a3" containerName="extract"
Sep 16 18:46:26.374744 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:26.374612    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea46658e-64ec-41a5-9d4b-c7389f5f84a3" containerName="extract"
Sep 16 18:46:26.374744 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:26.374655    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea46658e-64ec-41a5-9d4b-c7389f5f84a3" containerName="extract"
Sep 16 18:46:26.377285 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:26.377268    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qwbtz"
Sep 16 18:46:26.380627 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:26.380603    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 16 18:46:26.380627 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:26.380615    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:46:26.380803 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:26.380637    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-djkmm\""
Sep 16 18:46:26.404668 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:26.404633    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qwbtz"]
Sep 16 18:46:26.479519 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:26.479497    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b567d48b-af8d-4559-a783-4202a2e7f53b-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-qwbtz\" (UID: \"b567d48b-af8d-4559-a783-4202a2e7f53b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qwbtz"
Sep 16 18:46:26.479610 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:26.479525    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6tzb\" (UniqueName: \"kubernetes.io/projected/b567d48b-af8d-4559-a783-4202a2e7f53b-kube-api-access-h6tzb\") pod \"cert-manager-operator-controller-manager-684f79c9b9-qwbtz\" (UID: \"b567d48b-af8d-4559-a783-4202a2e7f53b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qwbtz"
Sep 16 18:46:26.579890 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:26.579866    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b567d48b-af8d-4559-a783-4202a2e7f53b-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-qwbtz\" (UID: \"b567d48b-af8d-4559-a783-4202a2e7f53b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qwbtz"
Sep 16 18:46:26.579977 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:26.579894    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6tzb\" (UniqueName: \"kubernetes.io/projected/b567d48b-af8d-4559-a783-4202a2e7f53b-kube-api-access-h6tzb\") pod \"cert-manager-operator-controller-manager-684f79c9b9-qwbtz\" (UID: \"b567d48b-af8d-4559-a783-4202a2e7f53b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qwbtz"
Sep 16 18:46:26.580287 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:26.580270    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b567d48b-af8d-4559-a783-4202a2e7f53b-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-qwbtz\" (UID: \"b567d48b-af8d-4559-a783-4202a2e7f53b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qwbtz"
Sep 16 18:46:26.589647 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:26.589620    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6tzb\" (UniqueName: \"kubernetes.io/projected/b567d48b-af8d-4559-a783-4202a2e7f53b-kube-api-access-h6tzb\") pod \"cert-manager-operator-controller-manager-684f79c9b9-qwbtz\" (UID: \"b567d48b-af8d-4559-a783-4202a2e7f53b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qwbtz"
Sep 16 18:46:26.686013 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:26.685990    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qwbtz"
Sep 16 18:46:26.967927 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:26.967904    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qwbtz"]
Sep 16 18:46:26.969878 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:46:26.969856    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb567d48b_af8d_4559_a783_4202a2e7f53b.slice/crio-f6296c47bab0542d24cc8eef4a869bae8ec77be46443e181cf5546a6c768562f WatchSource:0}: Error finding container f6296c47bab0542d24cc8eef4a869bae8ec77be46443e181cf5546a6c768562f: Status 404 returned error can't find the container with id f6296c47bab0542d24cc8eef4a869bae8ec77be46443e181cf5546a6c768562f
Sep 16 18:46:27.615916 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:27.615879    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qwbtz" event={"ID":"b567d48b-af8d-4559-a783-4202a2e7f53b","Type":"ContainerStarted","Data":"f6296c47bab0542d24cc8eef4a869bae8ec77be46443e181cf5546a6c768562f"}
Sep 16 18:46:28.622819 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:28.622787    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qwbtz" event={"ID":"b567d48b-af8d-4559-a783-4202a2e7f53b","Type":"ContainerStarted","Data":"d6699337bbf993671651acd4a530775f10faff239587cdca96a5387e46a41c13"}
Sep 16 18:46:28.672290 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:28.672247    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qwbtz" podStartSLOduration=1.211665829 podStartE2EDuration="2.672231395s" totalImagesPullingTime="1.460565566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:28.671548454 +0000 UTC m=+356.906481898" watchObservedRunningTime="2026-09-16 18:46:28.672231395 +0000 UTC m=+356.907164838"
Sep 16 18:46:30.226510 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:30.226479    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwzjb2"]
Sep 16 18:46:30.229729 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:30.229713    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwzjb2"
Sep 16 18:46:30.232547 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:30.232516    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6zvtq\""
Sep 16 18:46:30.232653 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:30.232581    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 18:46:30.232653 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:30.232597    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 18:46:30.244556 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:30.244537    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwzjb2"]
Sep 16 18:46:30.305682 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:30.305658    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7709fd6-af76-4ae1-af59-fa515dd979dc-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwzjb2\" (UID: \"b7709fd6-af76-4ae1-af59-fa515dd979dc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwzjb2"
Sep 16 18:46:30.305793 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:30.305702    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7709fd6-af76-4ae1-af59-fa515dd979dc-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwzjb2\" (UID: \"b7709fd6-af76-4ae1-af59-fa515dd979dc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwzjb2"
Sep 16 18:46:30.305793 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:30.305736    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mbp4\" (UniqueName: \"kubernetes.io/projected/b7709fd6-af76-4ae1-af59-fa515dd979dc-kube-api-access-2mbp4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwzjb2\" (UID: \"b7709fd6-af76-4ae1-af59-fa515dd979dc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwzjb2"
Sep 16 18:46:30.406540 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:30.406514    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7709fd6-af76-4ae1-af59-fa515dd979dc-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwzjb2\" (UID: \"b7709fd6-af76-4ae1-af59-fa515dd979dc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwzjb2"
Sep 16 18:46:30.406637 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:30.406555    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7709fd6-af76-4ae1-af59-fa515dd979dc-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwzjb2\" (UID: \"b7709fd6-af76-4ae1-af59-fa515dd979dc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwzjb2"
Sep 16 18:46:30.406637 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:30.406601    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mbp4\" (UniqueName: \"kubernetes.io/projected/b7709fd6-af76-4ae1-af59-fa515dd979dc-kube-api-access-2mbp4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwzjb2\" (UID: \"b7709fd6-af76-4ae1-af59-fa515dd979dc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwzjb2"
Sep 16 18:46:30.406877 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:30.406863    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7709fd6-af76-4ae1-af59-fa515dd979dc-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwzjb2\" (UID: \"b7709fd6-af76-4ae1-af59-fa515dd979dc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwzjb2"
Sep 16 18:46:30.407002 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:30.406982    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7709fd6-af76-4ae1-af59-fa515dd979dc-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwzjb2\" (UID: \"b7709fd6-af76-4ae1-af59-fa515dd979dc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwzjb2"
Sep 16 18:46:30.416908 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:30.416888    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mbp4\" (UniqueName: \"kubernetes.io/projected/b7709fd6-af76-4ae1-af59-fa515dd979dc-kube-api-access-2mbp4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwzjb2\" (UID: \"b7709fd6-af76-4ae1-af59-fa515dd979dc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwzjb2"
Sep 16 18:46:30.538949 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:30.538900    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwzjb2"
Sep 16 18:46:30.827161 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:30.827111    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwzjb2"]
Sep 16 18:46:30.830301 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:46:30.830264    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7709fd6_af76_4ae1_af59_fa515dd979dc.slice/crio-33eeb327eb46cba01569194cbe0525603f56247f78f866b1e65c99eaa3c29ff9 WatchSource:0}: Error finding container 33eeb327eb46cba01569194cbe0525603f56247f78f866b1e65c99eaa3c29ff9: Status 404 returned error can't find the container with id 33eeb327eb46cba01569194cbe0525603f56247f78f866b1e65c99eaa3c29ff9
Sep 16 18:46:31.637378 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:31.637348    2578 generic.go:356] "Generic (PLEG): container finished" podID="b7709fd6-af76-4ae1-af59-fa515dd979dc" containerID="66b89031a905aca5281c0c4e3c3ed9b2a3e2efe6dd6e9adbb0c627f7a2d248b4" exitCode=0
Sep 16 18:46:31.637706 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:31.637389    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwzjb2" event={"ID":"b7709fd6-af76-4ae1-af59-fa515dd979dc","Type":"ContainerDied","Data":"66b89031a905aca5281c0c4e3c3ed9b2a3e2efe6dd6e9adbb0c627f7a2d248b4"}
Sep 16 18:46:31.637706 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:31.637431    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwzjb2" event={"ID":"b7709fd6-af76-4ae1-af59-fa515dd979dc","Type":"ContainerStarted","Data":"33eeb327eb46cba01569194cbe0525603f56247f78f866b1e65c99eaa3c29ff9"}
Sep 16 18:46:34.652239 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:34.652196    2578 generic.go:356] "Generic (PLEG): container finished" podID="b7709fd6-af76-4ae1-af59-fa515dd979dc" containerID="3afb7ae42ab67a3dc0f7aed19db650c6e7e402430ce120440ae5c0dc11b7723e" exitCode=0
Sep 16 18:46:34.652623 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:34.652278    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwzjb2" event={"ID":"b7709fd6-af76-4ae1-af59-fa515dd979dc","Type":"ContainerDied","Data":"3afb7ae42ab67a3dc0f7aed19db650c6e7e402430ce120440ae5c0dc11b7723e"}
Sep 16 18:46:35.658996 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:35.658965    2578 generic.go:356] "Generic (PLEG): container finished" podID="b7709fd6-af76-4ae1-af59-fa515dd979dc" containerID="aa91850c5e8541add3c43a00433f34d05cc9674597da60aa15fb5856c12abfc0" exitCode=0
Sep 16 18:46:35.659371 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:35.659001    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwzjb2" event={"ID":"b7709fd6-af76-4ae1-af59-fa515dd979dc","Type":"ContainerDied","Data":"aa91850c5e8541add3c43a00433f34d05cc9674597da60aa15fb5856c12abfc0"}
Sep 16 18:46:36.780759 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:36.780737    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwzjb2"
Sep 16 18:46:36.852624 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:36.852597    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mbp4\" (UniqueName: \"kubernetes.io/projected/b7709fd6-af76-4ae1-af59-fa515dd979dc-kube-api-access-2mbp4\") pod \"b7709fd6-af76-4ae1-af59-fa515dd979dc\" (UID: \"b7709fd6-af76-4ae1-af59-fa515dd979dc\") "
Sep 16 18:46:36.852726 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:36.852648    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7709fd6-af76-4ae1-af59-fa515dd979dc-util\") pod \"b7709fd6-af76-4ae1-af59-fa515dd979dc\" (UID: \"b7709fd6-af76-4ae1-af59-fa515dd979dc\") "
Sep 16 18:46:36.852726 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:36.852677    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7709fd6-af76-4ae1-af59-fa515dd979dc-bundle\") pod \"b7709fd6-af76-4ae1-af59-fa515dd979dc\" (UID: \"b7709fd6-af76-4ae1-af59-fa515dd979dc\") "
Sep 16 18:46:36.853149 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:36.853109    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7709fd6-af76-4ae1-af59-fa515dd979dc-bundle" (OuterVolumeSpecName: "bundle") pod "b7709fd6-af76-4ae1-af59-fa515dd979dc" (UID: "b7709fd6-af76-4ae1-af59-fa515dd979dc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:36.854579 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:36.854555    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7709fd6-af76-4ae1-af59-fa515dd979dc-kube-api-access-2mbp4" (OuterVolumeSpecName: "kube-api-access-2mbp4") pod "b7709fd6-af76-4ae1-af59-fa515dd979dc" (UID: "b7709fd6-af76-4ae1-af59-fa515dd979dc"). InnerVolumeSpecName "kube-api-access-2mbp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:46:36.857155 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:36.857137    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7709fd6-af76-4ae1-af59-fa515dd979dc-util" (OuterVolumeSpecName: "util") pod "b7709fd6-af76-4ae1-af59-fa515dd979dc" (UID: "b7709fd6-af76-4ae1-af59-fa515dd979dc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:36.953156 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:36.953117    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7709fd6-af76-4ae1-af59-fa515dd979dc-util\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:46:36.953268 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:36.953177    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7709fd6-af76-4ae1-af59-fa515dd979dc-bundle\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:46:36.953268 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:36.953194    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mbp4\" (UniqueName: \"kubernetes.io/projected/b7709fd6-af76-4ae1-af59-fa515dd979dc-kube-api-access-2mbp4\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:46:37.668848 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:37.668821    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwzjb2"
Sep 16 18:46:37.668997 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:37.668821    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwzjb2" event={"ID":"b7709fd6-af76-4ae1-af59-fa515dd979dc","Type":"ContainerDied","Data":"33eeb327eb46cba01569194cbe0525603f56247f78f866b1e65c99eaa3c29ff9"}
Sep 16 18:46:37.668997 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:37.668941    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33eeb327eb46cba01569194cbe0525603f56247f78f866b1e65c99eaa3c29ff9"
Sep 16 18:46:38.274691 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:38.274659    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-69n4k"]
Sep 16 18:46:38.275089 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:38.274902    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7709fd6-af76-4ae1-af59-fa515dd979dc" containerName="extract"
Sep 16 18:46:38.275089 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:38.274912    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7709fd6-af76-4ae1-af59-fa515dd979dc" containerName="extract"
Sep 16 18:46:38.275089 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:38.274924    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7709fd6-af76-4ae1-af59-fa515dd979dc" containerName="util"
Sep 16 18:46:38.275089 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:38.274929    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7709fd6-af76-4ae1-af59-fa515dd979dc" containerName="util"
Sep 16 18:46:38.275089 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:38.274938    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7709fd6-af76-4ae1-af59-fa515dd979dc" containerName="pull"
Sep 16 18:46:38.275089 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:38.274943    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7709fd6-af76-4ae1-af59-fa515dd979dc" containerName="pull"
Sep 16 18:46:38.275089 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:38.274986    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7709fd6-af76-4ae1-af59-fa515dd979dc" containerName="extract"
Sep 16 18:46:38.278642 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:38.278625    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-69n4k"
Sep 16 18:46:38.281174 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:38.281149    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 18:46:38.281289 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:38.281267    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-g9kzn\""
Sep 16 18:46:38.281364 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:38.281350    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 18:46:38.286934 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:38.286911    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-69n4k"]
Sep 16 18:46:38.361890 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:38.361861    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf669dc7-c393-4a49-87b9-4ea0478a4ced-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-69n4k\" (UID: \"cf669dc7-c393-4a49-87b9-4ea0478a4ced\") " pod="cert-manager/cert-manager-webhook-6486ffd997-69n4k"
Sep 16 18:46:38.362007 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:38.361909    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfklp\" (UniqueName: \"kubernetes.io/projected/cf669dc7-c393-4a49-87b9-4ea0478a4ced-kube-api-access-cfklp\") pod \"cert-manager-webhook-6486ffd997-69n4k\" (UID: \"cf669dc7-c393-4a49-87b9-4ea0478a4ced\") " pod="cert-manager/cert-manager-webhook-6486ffd997-69n4k"
Sep 16 18:46:38.462453 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:38.462421    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf669dc7-c393-4a49-87b9-4ea0478a4ced-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-69n4k\" (UID: \"cf669dc7-c393-4a49-87b9-4ea0478a4ced\") " pod="cert-manager/cert-manager-webhook-6486ffd997-69n4k"
Sep 16 18:46:38.462548 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:38.462463    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfklp\" (UniqueName: \"kubernetes.io/projected/cf669dc7-c393-4a49-87b9-4ea0478a4ced-kube-api-access-cfklp\") pod \"cert-manager-webhook-6486ffd997-69n4k\" (UID: \"cf669dc7-c393-4a49-87b9-4ea0478a4ced\") " pod="cert-manager/cert-manager-webhook-6486ffd997-69n4k"
Sep 16 18:46:38.477701 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:38.477671    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfklp\" (UniqueName: \"kubernetes.io/projected/cf669dc7-c393-4a49-87b9-4ea0478a4ced-kube-api-access-cfklp\") pod \"cert-manager-webhook-6486ffd997-69n4k\" (UID: \"cf669dc7-c393-4a49-87b9-4ea0478a4ced\") " pod="cert-manager/cert-manager-webhook-6486ffd997-69n4k"
Sep 16 18:46:38.477779 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:38.477700    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf669dc7-c393-4a49-87b9-4ea0478a4ced-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-69n4k\" (UID: \"cf669dc7-c393-4a49-87b9-4ea0478a4ced\") " pod="cert-manager/cert-manager-webhook-6486ffd997-69n4k"
Sep 16 18:46:38.588784 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:38.588730    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-69n4k"
Sep 16 18:46:38.861266 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:38.861122    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-69n4k"]
Sep 16 18:46:38.863670 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:46:38.863637    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf669dc7_c393_4a49_87b9_4ea0478a4ced.slice/crio-1e29fe9204e350f8bf0c7308c44ccc167bffde50d0eeb7747f7040f80bcea615 WatchSource:0}: Error finding container 1e29fe9204e350f8bf0c7308c44ccc167bffde50d0eeb7747f7040f80bcea615: Status 404 returned error can't find the container with id 1e29fe9204e350f8bf0c7308c44ccc167bffde50d0eeb7747f7040f80bcea615
Sep 16 18:46:39.679604 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:39.679567    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-69n4k" event={"ID":"cf669dc7-c393-4a49-87b9-4ea0478a4ced","Type":"ContainerStarted","Data":"1e29fe9204e350f8bf0c7308c44ccc167bffde50d0eeb7747f7040f80bcea615"}
Sep 16 18:46:42.693665 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:42.693628    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-69n4k" event={"ID":"cf669dc7-c393-4a49-87b9-4ea0478a4ced","Type":"ContainerStarted","Data":"c9261b6199f33a0bc10d2eff722efb62e3081e65f5acc0a3a6cffd68a57a93c1"}
Sep 16 18:46:42.694110 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:42.693699    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-69n4k"
Sep 16 18:46:42.710745 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:42.710696    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-69n4k" podStartSLOduration=1.9263216380000001 podStartE2EDuration="4.710683544s" totalImagesPullingTime="2.784361906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:42.709433992 +0000 UTC m=+370.944367435" watchObservedRunningTime="2026-09-16 18:46:42.710683544 +0000 UTC m=+370.945616988"
Sep 16 18:46:48.700586 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:48.700555    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-69n4k"
Sep 16 18:46:57.804767 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:57.804733    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"]
Sep 16 18:46:57.807911 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:57.807895    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"
Sep 16 18:46:57.810531 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:57.810497    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 16 18:46:57.810662 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:57.810538    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 16 18:46:57.810662 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:57.810554    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 18:46:57.810777 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:57.810750    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 18:46:57.810777 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:57.810768    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 18:46:57.810872 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:57.810854    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 16 18:46:57.810974 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:57.810962    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7g9jt\""
Sep 16 18:46:57.822999 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:57.822978    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"]
Sep 16 18:46:57.891717 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:57.891694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-2ch46\" (UID: \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"
Sep 16 18:46:57.891813 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:57.891723    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-2ch46\" (UID: \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"
Sep 16 18:46:57.891813 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:57.891747    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-2ch46\" (UID: \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"
Sep 16 18:46:57.891813 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:57.891784    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-2ch46\" (UID: \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"
Sep 16 18:46:57.891813 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:57.891804    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-2ch46\" (UID: \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"
Sep 16 18:46:57.891967 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:57.891828    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n47dg\" (UniqueName: \"kubernetes.io/projected/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-kube-api-access-n47dg\") pod \"istiod-openshift-gateway-7b7995d5cd-2ch46\" (UID: \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"
Sep 16 18:46:57.891967 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:57.891887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-2ch46\" (UID: \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"
Sep 16 18:46:57.992317 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:57.992295    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-2ch46\" (UID: \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"
Sep 16 18:46:57.992389 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:57.992320    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-2ch46\" (UID: \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"
Sep 16 18:46:57.992389 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:57.992340    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-2ch46\" (UID: \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"
Sep 16 18:46:57.992389 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:57.992357    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-2ch46\" (UID: \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"
Sep 16 18:46:57.992389 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:57.992375    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-2ch46\" (UID: \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"
Sep 16 18:46:57.992650 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:57.992510    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n47dg\" (UniqueName: \"kubernetes.io/projected/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-kube-api-access-n47dg\") pod \"istiod-openshift-gateway-7b7995d5cd-2ch46\" (UID: \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"
Sep 16 18:46:57.992650 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:57.992592    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-2ch46\" (UID: \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"
Sep 16 18:46:57.993010 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:57.992980    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-2ch46\" (UID: \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"
Sep 16 18:46:57.994742 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:57.994711    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-2ch46\" (UID: \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"
Sep 16 18:46:57.994840 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:57.994715    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-2ch46\" (UID: \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"
Sep 16 18:46:57.994978 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:57.994960    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-2ch46\" (UID: \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"
Sep 16 18:46:57.995057 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:57.995009    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-2ch46\" (UID: \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"
Sep 16 18:46:58.002227 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:58.002195    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n47dg\" (UniqueName: \"kubernetes.io/projected/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-kube-api-access-n47dg\") pod \"istiod-openshift-gateway-7b7995d5cd-2ch46\" (UID: \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"
Sep 16 18:46:58.002315 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:58.002235    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-2ch46\" (UID: \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"
Sep 16 18:46:58.116638 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:58.116588    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"
Sep 16 18:46:58.392076 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:58.391988    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"]
Sep 16 18:46:58.394265 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:46:58.394239    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b44e19_36c7_4ce8_b2fa_3f997d3dbd2f.slice/crio-28200fad8e764abd61d1299bc148a57a5aacf4cf55c8e4c3317e8ce81e557a4d WatchSource:0}: Error finding container 28200fad8e764abd61d1299bc148a57a5aacf4cf55c8e4c3317e8ce81e557a4d: Status 404 returned error can't find the container with id 28200fad8e764abd61d1299bc148a57a5aacf4cf55c8e4c3317e8ce81e557a4d
Sep 16 18:46:58.761720 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:46:58.761684    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46" event={"ID":"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f","Type":"ContainerStarted","Data":"28200fad8e764abd61d1299bc148a57a5aacf4cf55c8e4c3317e8ce81e557a4d"}
Sep 16 18:47:00.973757 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:00.973707    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:47:00.974052 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:00.973782    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:47:01.777380 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:01.777344    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46" event={"ID":"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f","Type":"ContainerStarted","Data":"6c115d4295a63271f01027e78b1b0636af17e8cf1de37ed4bb86b87d79e80b85"}
Sep 16 18:47:01.777542 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:01.777467    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"
Sep 16 18:47:01.804001 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:01.803956    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46" podStartSLOduration=2.226721049 podStartE2EDuration="4.803942443s" totalImagesPullingTime="2.577221394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:01.802430908 +0000 UTC m=+390.037364353" watchObservedRunningTime="2026-09-16 18:47:01.803942443 +0000 UTC m=+390.038875886"
Sep 16 18:47:02.784166 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:02.784139    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"
Sep 16 18:47:04.573597 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:04.573565    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"]
Sep 16 18:47:04.577012 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:04.576991    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:04.579614 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:04.579593    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-sdjp5\""
Sep 16 18:47:04.597216 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:04.597194    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"]
Sep 16 18:47:04.745208 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:04.745182    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e2c88c-45e8-4432-b974-05ea559e2be6-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7\" (UID: \"79e2c88c-45e8-4432-b974-05ea559e2be6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:04.745308 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:04.745214    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e2c88c-45e8-4432-b974-05ea559e2be6-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7\" (UID: \"79e2c88c-45e8-4432-b974-05ea559e2be6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:04.745308 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:04.745239    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e2c88c-45e8-4432-b974-05ea559e2be6-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7\" (UID: \"79e2c88c-45e8-4432-b974-05ea559e2be6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:04.745380 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:04.745334    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f87ms\" (UniqueName: \"kubernetes.io/projected/79e2c88c-45e8-4432-b974-05ea559e2be6-kube-api-access-f87ms\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7\" (UID: \"79e2c88c-45e8-4432-b974-05ea559e2be6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:04.745380 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:04.745364    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e2c88c-45e8-4432-b974-05ea559e2be6-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7\" (UID: \"79e2c88c-45e8-4432-b974-05ea559e2be6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:04.745442 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:04.745380    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e2c88c-45e8-4432-b974-05ea559e2be6-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7\" (UID: \"79e2c88c-45e8-4432-b974-05ea559e2be6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:04.745442 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:04.745404    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e2c88c-45e8-4432-b974-05ea559e2be6-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7\" (UID: \"79e2c88c-45e8-4432-b974-05ea559e2be6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:04.745442 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:04.745429    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e2c88c-45e8-4432-b974-05ea559e2be6-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7\" (UID: \"79e2c88c-45e8-4432-b974-05ea559e2be6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:04.745537 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:04.745446    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e2c88c-45e8-4432-b974-05ea559e2be6-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7\" (UID: \"79e2c88c-45e8-4432-b974-05ea559e2be6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:04.845985 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:04.845930    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f87ms\" (UniqueName: \"kubernetes.io/projected/79e2c88c-45e8-4432-b974-05ea559e2be6-kube-api-access-f87ms\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7\" (UID: \"79e2c88c-45e8-4432-b974-05ea559e2be6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:04.845985 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:04.845958    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e2c88c-45e8-4432-b974-05ea559e2be6-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7\" (UID: \"79e2c88c-45e8-4432-b974-05ea559e2be6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:04.845985 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:04.845977    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e2c88c-45e8-4432-b974-05ea559e2be6-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7\" (UID: \"79e2c88c-45e8-4432-b974-05ea559e2be6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:04.846207 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:04.845994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e2c88c-45e8-4432-b974-05ea559e2be6-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7\" (UID: \"79e2c88c-45e8-4432-b974-05ea559e2be6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:04.846207 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:04.846019    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e2c88c-45e8-4432-b974-05ea559e2be6-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7\" (UID: \"79e2c88c-45e8-4432-b974-05ea559e2be6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:04.846207 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:04.846173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e2c88c-45e8-4432-b974-05ea559e2be6-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7\" (UID: \"79e2c88c-45e8-4432-b974-05ea559e2be6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:04.846362 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:04.846245    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e2c88c-45e8-4432-b974-05ea559e2be6-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7\" (UID: \"79e2c88c-45e8-4432-b974-05ea559e2be6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:04.846362 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:04.846283    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e2c88c-45e8-4432-b974-05ea559e2be6-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7\" (UID: \"79e2c88c-45e8-4432-b974-05ea559e2be6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:04.846362 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:04.846326    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e2c88c-45e8-4432-b974-05ea559e2be6-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7\" (UID: \"79e2c88c-45e8-4432-b974-05ea559e2be6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:04.846507 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:04.846432    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e2c88c-45e8-4432-b974-05ea559e2be6-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7\" (UID: \"79e2c88c-45e8-4432-b974-05ea559e2be6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:04.846507 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:04.846470    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e2c88c-45e8-4432-b974-05ea559e2be6-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7\" (UID: \"79e2c88c-45e8-4432-b974-05ea559e2be6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:04.846507 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:04.846438    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e2c88c-45e8-4432-b974-05ea559e2be6-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7\" (UID: \"79e2c88c-45e8-4432-b974-05ea559e2be6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:04.846659 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:04.846582    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e2c88c-45e8-4432-b974-05ea559e2be6-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7\" (UID: \"79e2c88c-45e8-4432-b974-05ea559e2be6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:04.846773 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:04.846752    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e2c88c-45e8-4432-b974-05ea559e2be6-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7\" (UID: \"79e2c88c-45e8-4432-b974-05ea559e2be6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:04.848507 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:04.848487    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e2c88c-45e8-4432-b974-05ea559e2be6-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7\" (UID: \"79e2c88c-45e8-4432-b974-05ea559e2be6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:04.848714 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:04.848696    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e2c88c-45e8-4432-b974-05ea559e2be6-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7\" (UID: \"79e2c88c-45e8-4432-b974-05ea559e2be6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:04.856082 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:04.856055    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f87ms\" (UniqueName: \"kubernetes.io/projected/79e2c88c-45e8-4432-b974-05ea559e2be6-kube-api-access-f87ms\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7\" (UID: \"79e2c88c-45e8-4432-b974-05ea559e2be6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:04.856165 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:04.856146    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e2c88c-45e8-4432-b974-05ea559e2be6-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7\" (UID: \"79e2c88c-45e8-4432-b974-05ea559e2be6\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:04.887600 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:04.887581    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:05.164096 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:05.164014    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"]
Sep 16 18:47:05.167234 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:47:05.167210    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e2c88c_45e8_4432_b974_05ea559e2be6.slice/crio-f97f097c54ae3844d58a3a436603731b1a9fce05fdb63f796c2f6583f477e520 WatchSource:0}: Error finding container f97f097c54ae3844d58a3a436603731b1a9fce05fdb63f796c2f6583f477e520: Status 404 returned error can't find the container with id f97f097c54ae3844d58a3a436603731b1a9fce05fdb63f796c2f6583f477e520
Sep 16 18:47:05.796722 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:05.796692    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7" event={"ID":"79e2c88c-45e8-4432-b974-05ea559e2be6","Type":"ContainerStarted","Data":"f97f097c54ae3844d58a3a436603731b1a9fce05fdb63f796c2f6583f477e520"}
Sep 16 18:47:07.847636 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:07.847602    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:47:07.847898 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:07.847668    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:47:07.847898 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:07.847698    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:47:08.811954 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:08.811915    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7" event={"ID":"79e2c88c-45e8-4432-b974-05ea559e2be6","Type":"ContainerStarted","Data":"5d88ec55edceeaad007a9a1cbf16d86e5820eefc1ed485285e959583fc9c647d"}
Sep 16 18:47:08.833618 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:08.833565    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7" podStartSLOduration=2.155410961 podStartE2EDuration="4.833547821s" totalImagesPullingTime="2.67813686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:08.83346209 +0000 UTC m=+397.068395525" watchObservedRunningTime="2026-09-16 18:47:08.833547821 +0000 UTC m=+397.068481329"
Sep 16 18:47:08.888403 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:08.888377    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:08.892466 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:08.892443    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:09.816301 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:09.816269    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:09.817170 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:09.817147    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:47:17.666934 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:17.666901    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck84gf"]
Sep 16 18:47:17.670074 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:17.670057    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck84gf"
Sep 16 18:47:17.672679 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:17.672657    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 18:47:17.672855 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:17.672831    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6zvtq\""
Sep 16 18:47:17.673865 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:17.673850    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 18:47:17.680394 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:17.680370    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck84gf"]
Sep 16 18:47:17.767277 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:17.767253    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apmtmb"]
Sep 16 18:47:17.769788 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:17.769772    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apmtmb"
Sep 16 18:47:17.779822 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:17.779802    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apmtmb"]
Sep 16 18:47:17.834489 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:17.834469    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrwfb\" (UniqueName: \"kubernetes.io/projected/9282b531-722b-429a-9dca-b8d98c874dd6-kube-api-access-vrwfb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck84gf\" (UID: \"9282b531-722b-429a-9dca-b8d98c874dd6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck84gf"
Sep 16 18:47:17.834594 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:17.834500    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9282b531-722b-429a-9dca-b8d98c874dd6-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck84gf\" (UID: \"9282b531-722b-429a-9dca-b8d98c874dd6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck84gf"
Sep 16 18:47:17.834594 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:17.834536    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9282b531-722b-429a-9dca-b8d98c874dd6-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck84gf\" (UID: \"9282b531-722b-429a-9dca-b8d98c874dd6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck84gf"
Sep 16 18:47:17.930113 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:17.930092    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jv8c"]
Sep 16 18:47:17.932525 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:17.932511    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jv8c"
Sep 16 18:47:17.934903 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:17.934881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrwfb\" (UniqueName: \"kubernetes.io/projected/9282b531-722b-429a-9dca-b8d98c874dd6-kube-api-access-vrwfb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck84gf\" (UID: \"9282b531-722b-429a-9dca-b8d98c874dd6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck84gf"
Sep 16 18:47:17.934997 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:17.934914    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9282b531-722b-429a-9dca-b8d98c874dd6-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck84gf\" (UID: \"9282b531-722b-429a-9dca-b8d98c874dd6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck84gf"
Sep 16 18:47:17.934997 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:17.934939    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4c9444a-1f29-4f6d-bda7-d7b6824ba147-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apmtmb\" (UID: \"a4c9444a-1f29-4f6d-bda7-d7b6824ba147\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apmtmb"
Sep 16 18:47:17.934997 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:17.934971    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4c9444a-1f29-4f6d-bda7-d7b6824ba147-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apmtmb\" (UID: \"a4c9444a-1f29-4f6d-bda7-d7b6824ba147\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apmtmb"
Sep 16 18:47:17.935167 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:17.935008    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9282b531-722b-429a-9dca-b8d98c874dd6-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck84gf\" (UID: \"9282b531-722b-429a-9dca-b8d98c874dd6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck84gf"
Sep 16 18:47:17.935167 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:17.935080    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x46r2\" (UniqueName: \"kubernetes.io/projected/a4c9444a-1f29-4f6d-bda7-d7b6824ba147-kube-api-access-x46r2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apmtmb\" (UID: \"a4c9444a-1f29-4f6d-bda7-d7b6824ba147\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apmtmb"
Sep 16 18:47:17.935303 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:17.935286    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9282b531-722b-429a-9dca-b8d98c874dd6-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck84gf\" (UID: \"9282b531-722b-429a-9dca-b8d98c874dd6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck84gf"
Sep 16 18:47:17.935348 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:17.935325    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9282b531-722b-429a-9dca-b8d98c874dd6-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck84gf\" (UID: \"9282b531-722b-429a-9dca-b8d98c874dd6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck84gf"
Sep 16 18:47:17.946683 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:17.946659    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jv8c"]
Sep 16 18:47:17.947625 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:17.947601    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrwfb\" (UniqueName: \"kubernetes.io/projected/9282b531-722b-429a-9dca-b8d98c874dd6-kube-api-access-vrwfb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck84gf\" (UID: \"9282b531-722b-429a-9dca-b8d98c874dd6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck84gf"
Sep 16 18:47:17.979253 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:17.979228    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck84gf"
Sep 16 18:47:18.004120 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.004091    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mkp7p"]
Sep 16 18:47:18.007314 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.007298    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mkp7p"
Sep 16 18:47:18.022909 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.022884    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mkp7p"]
Sep 16 18:47:18.036076 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.035450    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x46r2\" (UniqueName: \"kubernetes.io/projected/a4c9444a-1f29-4f6d-bda7-d7b6824ba147-kube-api-access-x46r2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apmtmb\" (UID: \"a4c9444a-1f29-4f6d-bda7-d7b6824ba147\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apmtmb"
Sep 16 18:47:18.036076 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.035483    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bftzx\" (UniqueName: \"kubernetes.io/projected/2c6baccf-3ec0-4e97-97b1-a60b5c937778-kube-api-access-bftzx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jv8c\" (UID: \"2c6baccf-3ec0-4e97-97b1-a60b5c937778\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jv8c"
Sep 16 18:47:18.036076 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.035542    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4c9444a-1f29-4f6d-bda7-d7b6824ba147-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apmtmb\" (UID: \"a4c9444a-1f29-4f6d-bda7-d7b6824ba147\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apmtmb"
Sep 16 18:47:18.036076 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.035582    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4c9444a-1f29-4f6d-bda7-d7b6824ba147-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apmtmb\" (UID: \"a4c9444a-1f29-4f6d-bda7-d7b6824ba147\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apmtmb"
Sep 16 18:47:18.036076 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.035619    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c6baccf-3ec0-4e97-97b1-a60b5c937778-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jv8c\" (UID: \"2c6baccf-3ec0-4e97-97b1-a60b5c937778\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jv8c"
Sep 16 18:47:18.036076 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.035652    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c6baccf-3ec0-4e97-97b1-a60b5c937778-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jv8c\" (UID: \"2c6baccf-3ec0-4e97-97b1-a60b5c937778\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jv8c"
Sep 16 18:47:18.036076 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.035929    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4c9444a-1f29-4f6d-bda7-d7b6824ba147-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apmtmb\" (UID: \"a4c9444a-1f29-4f6d-bda7-d7b6824ba147\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apmtmb"
Sep 16 18:47:18.036076 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.035975    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4c9444a-1f29-4f6d-bda7-d7b6824ba147-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apmtmb\" (UID: \"a4c9444a-1f29-4f6d-bda7-d7b6824ba147\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apmtmb"
Sep 16 18:47:18.050747 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.050722    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x46r2\" (UniqueName: \"kubernetes.io/projected/a4c9444a-1f29-4f6d-bda7-d7b6824ba147-kube-api-access-x46r2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apmtmb\" (UID: \"a4c9444a-1f29-4f6d-bda7-d7b6824ba147\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apmtmb"
Sep 16 18:47:18.077650 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.077625    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apmtmb"
Sep 16 18:47:18.136517 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.136489    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c6baccf-3ec0-4e97-97b1-a60b5c937778-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jv8c\" (UID: \"2c6baccf-3ec0-4e97-97b1-a60b5c937778\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jv8c"
Sep 16 18:47:18.136619 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.136541    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bftzx\" (UniqueName: \"kubernetes.io/projected/2c6baccf-3ec0-4e97-97b1-a60b5c937778-kube-api-access-bftzx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jv8c\" (UID: \"2c6baccf-3ec0-4e97-97b1-a60b5c937778\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jv8c"
Sep 16 18:47:18.136619 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.136581    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mkp7p\" (UID: \"82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mkp7p"
Sep 16 18:47:18.136619 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.136614    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mkp7p\" (UID: \"82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mkp7p"
Sep 16 18:47:18.136729 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.136633    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c6baccf-3ec0-4e97-97b1-a60b5c937778-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jv8c\" (UID: \"2c6baccf-3ec0-4e97-97b1-a60b5c937778\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jv8c"
Sep 16 18:47:18.136729 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.136649    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmd96\" (UniqueName: \"kubernetes.io/projected/82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a-kube-api-access-fmd96\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mkp7p\" (UID: \"82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mkp7p"
Sep 16 18:47:18.136886 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.136867    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c6baccf-3ec0-4e97-97b1-a60b5c937778-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jv8c\" (UID: \"2c6baccf-3ec0-4e97-97b1-a60b5c937778\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jv8c"
Sep 16 18:47:18.136927 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.136906    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c6baccf-3ec0-4e97-97b1-a60b5c937778-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jv8c\" (UID: \"2c6baccf-3ec0-4e97-97b1-a60b5c937778\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jv8c"
Sep 16 18:47:18.155034 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.155012    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bftzx\" (UniqueName: \"kubernetes.io/projected/2c6baccf-3ec0-4e97-97b1-a60b5c937778-kube-api-access-bftzx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jv8c\" (UID: \"2c6baccf-3ec0-4e97-97b1-a60b5c937778\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jv8c"
Sep 16 18:47:18.237207 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.237160    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mkp7p\" (UID: \"82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mkp7p"
Sep 16 18:47:18.237207 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.237202    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mkp7p\" (UID: \"82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mkp7p"
Sep 16 18:47:18.237325 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.237225    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmd96\" (UniqueName: \"kubernetes.io/projected/82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a-kube-api-access-fmd96\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mkp7p\" (UID: \"82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mkp7p"
Sep 16 18:47:18.237530 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.237512    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mkp7p\" (UID: \"82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mkp7p"
Sep 16 18:47:18.237589 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.237571    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mkp7p\" (UID: \"82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mkp7p"
Sep 16 18:47:18.241109 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.241090    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jv8c"
Sep 16 18:47:18.250150 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.250131    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmd96\" (UniqueName: \"kubernetes.io/projected/82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a-kube-api-access-fmd96\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mkp7p\" (UID: \"82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mkp7p"
Sep 16 18:47:18.266356 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.266335    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck84gf"]
Sep 16 18:47:18.269240 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:47:18.269200    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9282b531_722b_429a_9dca_b8d98c874dd6.slice/crio-86c5d954aeaab070fe5398d67b2a4e6b9b0e75256c2e62a0c6dfa3a397c6e6a0 WatchSource:0}: Error finding container 86c5d954aeaab070fe5398d67b2a4e6b9b0e75256c2e62a0c6dfa3a397c6e6a0: Status 404 returned error can't find the container with id 86c5d954aeaab070fe5398d67b2a4e6b9b0e75256c2e62a0c6dfa3a397c6e6a0
Sep 16 18:47:18.316026 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.315997    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mkp7p"
Sep 16 18:47:18.369027 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.368512    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apmtmb"]
Sep 16 18:47:18.415764 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:47:18.415737    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4c9444a_1f29_4f6d_bda7_d7b6824ba147.slice/crio-409bd704444ae08eea1f3e1de233a880de7c6922f92a6f2c83bbba48e5bfceb0 WatchSource:0}: Error finding container 409bd704444ae08eea1f3e1de233a880de7c6922f92a6f2c83bbba48e5bfceb0: Status 404 returned error can't find the container with id 409bd704444ae08eea1f3e1de233a880de7c6922f92a6f2c83bbba48e5bfceb0
Sep 16 18:47:18.522643 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.522622    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jv8c"]
Sep 16 18:47:18.555800 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:47:18.555775    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c6baccf_3ec0_4e97_97b1_a60b5c937778.slice/crio-c31af485f058da734314c4de45e1f6c6b68c8f2e197d03dc9fc39b851dba8a6d WatchSource:0}: Error finding container c31af485f058da734314c4de45e1f6c6b68c8f2e197d03dc9fc39b851dba8a6d: Status 404 returned error can't find the container with id c31af485f058da734314c4de45e1f6c6b68c8f2e197d03dc9fc39b851dba8a6d
Sep 16 18:47:18.618398 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.618375    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mkp7p"]
Sep 16 18:47:18.620909 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:47:18.620886    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f9c4e7_0e07_4ba7_bdfb_6abdcc61505a.slice/crio-424d6d4c136a73da74f0b14972669e91a110060a546d8f7bcf13528e3bc628b4 WatchSource:0}: Error finding container 424d6d4c136a73da74f0b14972669e91a110060a546d8f7bcf13528e3bc628b4: Status 404 returned error can't find the container with id 424d6d4c136a73da74f0b14972669e91a110060a546d8f7bcf13528e3bc628b4
Sep 16 18:47:18.856086 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.856062    2578 generic.go:356] "Generic (PLEG): container finished" podID="a4c9444a-1f29-4f6d-bda7-d7b6824ba147" containerID="17b7875f42f98ddfac2bce5beac7f6a28887eeb3b29e594a6a6d9580f6fc327c" exitCode=0
Sep 16 18:47:18.856397 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.856128    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apmtmb" event={"ID":"a4c9444a-1f29-4f6d-bda7-d7b6824ba147","Type":"ContainerDied","Data":"17b7875f42f98ddfac2bce5beac7f6a28887eeb3b29e594a6a6d9580f6fc327c"}
Sep 16 18:47:18.856397 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.856195    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apmtmb" event={"ID":"a4c9444a-1f29-4f6d-bda7-d7b6824ba147","Type":"ContainerStarted","Data":"409bd704444ae08eea1f3e1de233a880de7c6922f92a6f2c83bbba48e5bfceb0"}
Sep 16 18:47:18.857753 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.857727    2578 generic.go:356] "Generic (PLEG): container finished" podID="9282b531-722b-429a-9dca-b8d98c874dd6" containerID="4dc5b41f03f37e8492968ae23634ae29d0863b90f59a24ff4516735c1ca652b0" exitCode=0
Sep 16 18:47:18.857850 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.857766    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck84gf" event={"ID":"9282b531-722b-429a-9dca-b8d98c874dd6","Type":"ContainerDied","Data":"4dc5b41f03f37e8492968ae23634ae29d0863b90f59a24ff4516735c1ca652b0"}
Sep 16 18:47:18.857850 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.857812    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck84gf" event={"ID":"9282b531-722b-429a-9dca-b8d98c874dd6","Type":"ContainerStarted","Data":"86c5d954aeaab070fe5398d67b2a4e6b9b0e75256c2e62a0c6dfa3a397c6e6a0"}
Sep 16 18:47:18.859439 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.859419    2578 generic.go:356] "Generic (PLEG): container finished" podID="82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a" containerID="a9f15a1201f1bf496f48d7efd284f8e6358d9b523e7a47e8e4ba758e599fdb8f" exitCode=0
Sep 16 18:47:18.859525 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.859500    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mkp7p" event={"ID":"82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a","Type":"ContainerDied","Data":"a9f15a1201f1bf496f48d7efd284f8e6358d9b523e7a47e8e4ba758e599fdb8f"}
Sep 16 18:47:18.859592 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.859537    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mkp7p" event={"ID":"82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a","Type":"ContainerStarted","Data":"424d6d4c136a73da74f0b14972669e91a110060a546d8f7bcf13528e3bc628b4"}
Sep 16 18:47:18.861462 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.861443    2578 generic.go:356] "Generic (PLEG): container finished" podID="2c6baccf-3ec0-4e97-97b1-a60b5c937778" containerID="75d0ad29c38c4bcc579a309fff8dbe9c344081fb0fe74fd0477da0df946177a5" exitCode=0
Sep 16 18:47:18.861564 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.861483    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jv8c" event={"ID":"2c6baccf-3ec0-4e97-97b1-a60b5c937778","Type":"ContainerDied","Data":"75d0ad29c38c4bcc579a309fff8dbe9c344081fb0fe74fd0477da0df946177a5"}
Sep 16 18:47:18.861564 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:18.861509    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jv8c" event={"ID":"2c6baccf-3ec0-4e97-97b1-a60b5c937778","Type":"ContainerStarted","Data":"c31af485f058da734314c4de45e1f6c6b68c8f2e197d03dc9fc39b851dba8a6d"}
Sep 16 18:47:19.867432 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:19.867407    2578 generic.go:356] "Generic (PLEG): container finished" podID="a4c9444a-1f29-4f6d-bda7-d7b6824ba147" containerID="1da59ade8784faded1d396580c2c9dbf7a7fd7260f743cdcde1cc0967632451a" exitCode=0
Sep 16 18:47:19.867756 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:19.867470    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apmtmb" event={"ID":"a4c9444a-1f29-4f6d-bda7-d7b6824ba147","Type":"ContainerDied","Data":"1da59ade8784faded1d396580c2c9dbf7a7fd7260f743cdcde1cc0967632451a"}
Sep 16 18:47:19.869470 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:19.869448    2578 generic.go:356] "Generic (PLEG): container finished" podID="9282b531-722b-429a-9dca-b8d98c874dd6" containerID="1a0f11c0a8848c8f7977a3cd12cf018fd6860a8791eee0a0da846b21fad72f67" exitCode=0
Sep 16 18:47:19.869560 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:19.869524    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck84gf" event={"ID":"9282b531-722b-429a-9dca-b8d98c874dd6","Type":"ContainerDied","Data":"1a0f11c0a8848c8f7977a3cd12cf018fd6860a8791eee0a0da846b21fad72f67"}
Sep 16 18:47:19.871420 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:19.871402    2578 generic.go:356] "Generic (PLEG): container finished" podID="82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a" containerID="172e2f92f43ede0cae53665756a63a3c8c07161c5f3fad7aa901f3d59cc8cfb6" exitCode=0
Sep 16 18:47:19.871483 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:19.871465    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mkp7p" event={"ID":"82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a","Type":"ContainerDied","Data":"172e2f92f43ede0cae53665756a63a3c8c07161c5f3fad7aa901f3d59cc8cfb6"}
Sep 16 18:47:19.873773 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:19.873754    2578 generic.go:356] "Generic (PLEG): container finished" podID="2c6baccf-3ec0-4e97-97b1-a60b5c937778" containerID="f6b8b9243795153b054032f923c0e69f585a1392377ed7502333a5d877cd9fc3" exitCode=0
Sep 16 18:47:19.873845 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:19.873791    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jv8c" event={"ID":"2c6baccf-3ec0-4e97-97b1-a60b5c937778","Type":"ContainerDied","Data":"f6b8b9243795153b054032f923c0e69f585a1392377ed7502333a5d877cd9fc3"}
Sep 16 18:47:20.880759 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:20.880727    2578 generic.go:356] "Generic (PLEG): container finished" podID="a4c9444a-1f29-4f6d-bda7-d7b6824ba147" containerID="8b34c8738383d3d2decf3a8622980f7a5ef488e7815878ddfa33a0beabceb805" exitCode=0
Sep 16 18:47:20.881204 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:20.880810    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apmtmb" event={"ID":"a4c9444a-1f29-4f6d-bda7-d7b6824ba147","Type":"ContainerDied","Data":"8b34c8738383d3d2decf3a8622980f7a5ef488e7815878ddfa33a0beabceb805"}
Sep 16 18:47:20.882913 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:20.882893    2578 generic.go:356] "Generic (PLEG): container finished" podID="9282b531-722b-429a-9dca-b8d98c874dd6" containerID="d7bf1a54d1e8878edef30ecd0229793ae1aff54d0c104fa5ccd284ee4bd9840f" exitCode=0
Sep 16 18:47:20.883022 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:20.882973    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck84gf" event={"ID":"9282b531-722b-429a-9dca-b8d98c874dd6","Type":"ContainerDied","Data":"d7bf1a54d1e8878edef30ecd0229793ae1aff54d0c104fa5ccd284ee4bd9840f"}
Sep 16 18:47:20.884914 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:20.884895    2578 generic.go:356] "Generic (PLEG): container finished" podID="82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a" containerID="11c3e460fb8862f018e43d9ab1dbcd601d532636a692770bc774a7958dc477c5" exitCode=0
Sep 16 18:47:20.884999 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:20.884959    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mkp7p" event={"ID":"82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a","Type":"ContainerDied","Data":"11c3e460fb8862f018e43d9ab1dbcd601d532636a692770bc774a7958dc477c5"}
Sep 16 18:47:20.886963 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:20.886947    2578 generic.go:356] "Generic (PLEG): container finished" podID="2c6baccf-3ec0-4e97-97b1-a60b5c937778" containerID="77f11efc2ac048b5e8ace825d500eb1ad34194f7f0e14e1151b2f01e82ea1716" exitCode=0
Sep 16 18:47:20.887030 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:20.887017    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jv8c" event={"ID":"2c6baccf-3ec0-4e97-97b1-a60b5c937778","Type":"ContainerDied","Data":"77f11efc2ac048b5e8ace825d500eb1ad34194f7f0e14e1151b2f01e82ea1716"}
Sep 16 18:47:22.022323 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.022300    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apmtmb"
Sep 16 18:47:22.061848 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.061824    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jv8c"
Sep 16 18:47:22.098289 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.098264    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mkp7p"
Sep 16 18:47:22.101215 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.101193    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck84gf"
Sep 16 18:47:22.169165 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.169105    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4c9444a-1f29-4f6d-bda7-d7b6824ba147-bundle\") pod \"a4c9444a-1f29-4f6d-bda7-d7b6824ba147\" (UID: \"a4c9444a-1f29-4f6d-bda7-d7b6824ba147\") "
Sep 16 18:47:22.169266 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.169184    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bftzx\" (UniqueName: \"kubernetes.io/projected/2c6baccf-3ec0-4e97-97b1-a60b5c937778-kube-api-access-bftzx\") pod \"2c6baccf-3ec0-4e97-97b1-a60b5c937778\" (UID: \"2c6baccf-3ec0-4e97-97b1-a60b5c937778\") "
Sep 16 18:47:22.169266 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.169226    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c6baccf-3ec0-4e97-97b1-a60b5c937778-bundle\") pod \"2c6baccf-3ec0-4e97-97b1-a60b5c937778\" (UID: \"2c6baccf-3ec0-4e97-97b1-a60b5c937778\") "
Sep 16 18:47:22.169266 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.169262    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4c9444a-1f29-4f6d-bda7-d7b6824ba147-util\") pod \"a4c9444a-1f29-4f6d-bda7-d7b6824ba147\" (UID: \"a4c9444a-1f29-4f6d-bda7-d7b6824ba147\") "
Sep 16 18:47:22.169424 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.169287    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9282b531-722b-429a-9dca-b8d98c874dd6-bundle\") pod \"9282b531-722b-429a-9dca-b8d98c874dd6\" (UID: \"9282b531-722b-429a-9dca-b8d98c874dd6\") "
Sep 16 18:47:22.169424 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.169310    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x46r2\" (UniqueName: \"kubernetes.io/projected/a4c9444a-1f29-4f6d-bda7-d7b6824ba147-kube-api-access-x46r2\") pod \"a4c9444a-1f29-4f6d-bda7-d7b6824ba147\" (UID: \"a4c9444a-1f29-4f6d-bda7-d7b6824ba147\") "
Sep 16 18:47:22.169424 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.169334    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c6baccf-3ec0-4e97-97b1-a60b5c937778-util\") pod \"2c6baccf-3ec0-4e97-97b1-a60b5c937778\" (UID: \"2c6baccf-3ec0-4e97-97b1-a60b5c937778\") "
Sep 16 18:47:22.169791 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.169766    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c6baccf-3ec0-4e97-97b1-a60b5c937778-bundle" (OuterVolumeSpecName: "bundle") pod "2c6baccf-3ec0-4e97-97b1-a60b5c937778" (UID: "2c6baccf-3ec0-4e97-97b1-a60b5c937778"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:22.169984 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.169956    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9282b531-722b-429a-9dca-b8d98c874dd6-bundle" (OuterVolumeSpecName: "bundle") pod "9282b531-722b-429a-9dca-b8d98c874dd6" (UID: "9282b531-722b-429a-9dca-b8d98c874dd6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:22.170468 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.170446    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c9444a-1f29-4f6d-bda7-d7b6824ba147-bundle" (OuterVolumeSpecName: "bundle") pod "a4c9444a-1f29-4f6d-bda7-d7b6824ba147" (UID: "a4c9444a-1f29-4f6d-bda7-d7b6824ba147"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:22.171340 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.171317    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c6baccf-3ec0-4e97-97b1-a60b5c937778-kube-api-access-bftzx" (OuterVolumeSpecName: "kube-api-access-bftzx") pod "2c6baccf-3ec0-4e97-97b1-a60b5c937778" (UID: "2c6baccf-3ec0-4e97-97b1-a60b5c937778"). InnerVolumeSpecName "kube-api-access-bftzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:22.171481 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.171466    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4c9444a-1f29-4f6d-bda7-d7b6824ba147-kube-api-access-x46r2" (OuterVolumeSpecName: "kube-api-access-x46r2") pod "a4c9444a-1f29-4f6d-bda7-d7b6824ba147" (UID: "a4c9444a-1f29-4f6d-bda7-d7b6824ba147"). InnerVolumeSpecName "kube-api-access-x46r2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:22.177872 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.177843    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c9444a-1f29-4f6d-bda7-d7b6824ba147-util" (OuterVolumeSpecName: "util") pod "a4c9444a-1f29-4f6d-bda7-d7b6824ba147" (UID: "a4c9444a-1f29-4f6d-bda7-d7b6824ba147"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:22.178287 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.178266    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c6baccf-3ec0-4e97-97b1-a60b5c937778-util" (OuterVolumeSpecName: "util") pod "2c6baccf-3ec0-4e97-97b1-a60b5c937778" (UID: "2c6baccf-3ec0-4e97-97b1-a60b5c937778"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:22.270184 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.270165    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a-bundle\") pod \"82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a\" (UID: \"82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a\") "
Sep 16 18:47:22.270275 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.270194    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmd96\" (UniqueName: \"kubernetes.io/projected/82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a-kube-api-access-fmd96\") pod \"82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a\" (UID: \"82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a\") "
Sep 16 18:47:22.270275 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.270230    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrwfb\" (UniqueName: \"kubernetes.io/projected/9282b531-722b-429a-9dca-b8d98c874dd6-kube-api-access-vrwfb\") pod \"9282b531-722b-429a-9dca-b8d98c874dd6\" (UID: \"9282b531-722b-429a-9dca-b8d98c874dd6\") "
Sep 16 18:47:22.270275 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.270250    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a-util\") pod \"82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a\" (UID: \"82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a\") "
Sep 16 18:47:22.270389 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.270281    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9282b531-722b-429a-9dca-b8d98c874dd6-util\") pod \"9282b531-722b-429a-9dca-b8d98c874dd6\" (UID: \"9282b531-722b-429a-9dca-b8d98c874dd6\") "
Sep 16 18:47:22.270434 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.270392    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bftzx\" (UniqueName: \"kubernetes.io/projected/2c6baccf-3ec0-4e97-97b1-a60b5c937778-kube-api-access-bftzx\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:22.270434 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.270402    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c6baccf-3ec0-4e97-97b1-a60b5c937778-bundle\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:22.270434 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.270414    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4c9444a-1f29-4f6d-bda7-d7b6824ba147-util\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:22.270434 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.270426    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9282b531-722b-429a-9dca-b8d98c874dd6-bundle\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:22.270582 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.270439    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x46r2\" (UniqueName: \"kubernetes.io/projected/a4c9444a-1f29-4f6d-bda7-d7b6824ba147-kube-api-access-x46r2\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:22.270582 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.270453    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c6baccf-3ec0-4e97-97b1-a60b5c937778-util\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:22.270582 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.270466    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4c9444a-1f29-4f6d-bda7-d7b6824ba147-bundle\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:22.270729 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.270702    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a-bundle" (OuterVolumeSpecName: "bundle") pod "82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a" (UID: "82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:22.272160 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.272131    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9282b531-722b-429a-9dca-b8d98c874dd6-kube-api-access-vrwfb" (OuterVolumeSpecName: "kube-api-access-vrwfb") pod "9282b531-722b-429a-9dca-b8d98c874dd6" (UID: "9282b531-722b-429a-9dca-b8d98c874dd6"). InnerVolumeSpecName "kube-api-access-vrwfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:22.272591 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.272574    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a-kube-api-access-fmd96" (OuterVolumeSpecName: "kube-api-access-fmd96") pod "82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a" (UID: "82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a"). InnerVolumeSpecName "kube-api-access-fmd96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:22.275825 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.275805    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a-util" (OuterVolumeSpecName: "util") pod "82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a" (UID: "82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:22.275982 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.275964    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9282b531-722b-429a-9dca-b8d98c874dd6-util" (OuterVolumeSpecName: "util") pod "9282b531-722b-429a-9dca-b8d98c874dd6" (UID: "9282b531-722b-429a-9dca-b8d98c874dd6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:22.370951 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.370926    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrwfb\" (UniqueName: \"kubernetes.io/projected/9282b531-722b-429a-9dca-b8d98c874dd6-kube-api-access-vrwfb\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:22.370951 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.370948    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a-util\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:22.371079 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.370958    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9282b531-722b-429a-9dca-b8d98c874dd6-util\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:22.371079 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.370965    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a-bundle\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:22.371079 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.370973    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmd96\" (UniqueName: \"kubernetes.io/projected/82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a-kube-api-access-fmd96\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:47:22.898652 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.898627    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apmtmb"
Sep 16 18:47:22.898809 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.898643    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apmtmb" event={"ID":"a4c9444a-1f29-4f6d-bda7-d7b6824ba147","Type":"ContainerDied","Data":"409bd704444ae08eea1f3e1de233a880de7c6922f92a6f2c83bbba48e5bfceb0"}
Sep 16 18:47:22.898809 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.898691    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="409bd704444ae08eea1f3e1de233a880de7c6922f92a6f2c83bbba48e5bfceb0"
Sep 16 18:47:22.900776 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.900756    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck84gf" event={"ID":"9282b531-722b-429a-9dca-b8d98c874dd6","Type":"ContainerDied","Data":"86c5d954aeaab070fe5398d67b2a4e6b9b0e75256c2e62a0c6dfa3a397c6e6a0"}
Sep 16 18:47:22.900954 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.900931    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86c5d954aeaab070fe5398d67b2a4e6b9b0e75256c2e62a0c6dfa3a397c6e6a0"
Sep 16 18:47:22.900954 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.900780    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ck84gf"
Sep 16 18:47:22.902782 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.902764    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mkp7p"
Sep 16 18:47:22.902892 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.902762    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mkp7p" event={"ID":"82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a","Type":"ContainerDied","Data":"424d6d4c136a73da74f0b14972669e91a110060a546d8f7bcf13528e3bc628b4"}
Sep 16 18:47:22.902892 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.902880    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="424d6d4c136a73da74f0b14972669e91a110060a546d8f7bcf13528e3bc628b4"
Sep 16 18:47:22.904701 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.904686    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jv8c"
Sep 16 18:47:22.904701 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.904688    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69jv8c" event={"ID":"2c6baccf-3ec0-4e97-97b1-a60b5c937778","Type":"ContainerDied","Data":"c31af485f058da734314c4de45e1f6c6b68c8f2e197d03dc9fc39b851dba8a6d"}
Sep 16 18:47:22.904835 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:22.904724    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c31af485f058da734314c4de45e1f6c6b68c8f2e197d03dc9fc39b851dba8a6d"
Sep 16 18:47:34.248565 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.248531    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-lnwj8"]
Sep 16 18:47:34.249033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.248818    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a" containerName="pull"
Sep 16 18:47:34.249033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.248831    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a" containerName="pull"
Sep 16 18:47:34.249033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.248840    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a" containerName="util"
Sep 16 18:47:34.249033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.248845    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a" containerName="util"
Sep 16 18:47:34.249033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.248857    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4c9444a-1f29-4f6d-bda7-d7b6824ba147" containerName="util"
Sep 16 18:47:34.249033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.248862    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4c9444a-1f29-4f6d-bda7-d7b6824ba147" containerName="util"
Sep 16 18:47:34.249033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.248868    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4c9444a-1f29-4f6d-bda7-d7b6824ba147" containerName="pull"
Sep 16 18:47:34.249033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.248872    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4c9444a-1f29-4f6d-bda7-d7b6824ba147" containerName="pull"
Sep 16 18:47:34.249033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.248878    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c6baccf-3ec0-4e97-97b1-a60b5c937778" containerName="extract"
Sep 16 18:47:34.249033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.248884    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c6baccf-3ec0-4e97-97b1-a60b5c937778" containerName="extract"
Sep 16 18:47:34.249033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.248890    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9282b531-722b-429a-9dca-b8d98c874dd6" containerName="pull"
Sep 16 18:47:34.249033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.248894    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9282b531-722b-429a-9dca-b8d98c874dd6" containerName="pull"
Sep 16 18:47:34.249033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.248904    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c6baccf-3ec0-4e97-97b1-a60b5c937778" containerName="util"
Sep 16 18:47:34.249033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.248908    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c6baccf-3ec0-4e97-97b1-a60b5c937778" containerName="util"
Sep 16 18:47:34.249033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.248916    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4c9444a-1f29-4f6d-bda7-d7b6824ba147" containerName="extract"
Sep 16 18:47:34.249033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.248920    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4c9444a-1f29-4f6d-bda7-d7b6824ba147" containerName="extract"
Sep 16 18:47:34.249033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.248927    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9282b531-722b-429a-9dca-b8d98c874dd6" containerName="util"
Sep 16 18:47:34.249033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.248932    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9282b531-722b-429a-9dca-b8d98c874dd6" containerName="util"
Sep 16 18:47:34.249033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.248937    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c6baccf-3ec0-4e97-97b1-a60b5c937778" containerName="pull"
Sep 16 18:47:34.249033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.248942    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c6baccf-3ec0-4e97-97b1-a60b5c937778" containerName="pull"
Sep 16 18:47:34.249033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.248949    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9282b531-722b-429a-9dca-b8d98c874dd6" containerName="extract"
Sep 16 18:47:34.249033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.248954    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9282b531-722b-429a-9dca-b8d98c874dd6" containerName="extract"
Sep 16 18:47:34.249033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.248960    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a" containerName="extract"
Sep 16 18:47:34.249033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.248965    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a" containerName="extract"
Sep 16 18:47:34.249033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.249009    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="82f9c4e7-0e07-4ba7-bdfb-6abdcc61505a" containerName="extract"
Sep 16 18:47:34.249033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.249015    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9282b531-722b-429a-9dca-b8d98c874dd6" containerName="extract"
Sep 16 18:47:34.249033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.249020    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4c9444a-1f29-4f6d-bda7-d7b6824ba147" containerName="extract"
Sep 16 18:47:34.249033 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.249027    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c6baccf-3ec0-4e97-97b1-a60b5c937778" containerName="extract"
Sep 16 18:47:34.258748 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.258728    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-lnwj8"
Sep 16 18:47:34.263646 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.263629    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 18:47:34.263892 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.263872    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ncgsb\""
Sep 16 18:47:34.264431 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.264410    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 18:47:34.269657 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.269635    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-lnwj8"]
Sep 16 18:47:34.343405 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.343366    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7l6t\" (UniqueName: \"kubernetes.io/projected/eed3ce9f-0c0b-4559-8eef-71ac47313c8d-kube-api-access-x7l6t\") pod \"kuadrant-operator-controller-manager-6d4cb79c-lnwj8\" (UID: \"eed3ce9f-0c0b-4559-8eef-71ac47313c8d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-lnwj8"
Sep 16 18:47:34.343573 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.343410    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eed3ce9f-0c0b-4559-8eef-71ac47313c8d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-lnwj8\" (UID: \"eed3ce9f-0c0b-4559-8eef-71ac47313c8d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-lnwj8"
Sep 16 18:47:34.444014 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.443981    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7l6t\" (UniqueName: \"kubernetes.io/projected/eed3ce9f-0c0b-4559-8eef-71ac47313c8d-kube-api-access-x7l6t\") pod \"kuadrant-operator-controller-manager-6d4cb79c-lnwj8\" (UID: \"eed3ce9f-0c0b-4559-8eef-71ac47313c8d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-lnwj8"
Sep 16 18:47:34.444189 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.444019    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eed3ce9f-0c0b-4559-8eef-71ac47313c8d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-lnwj8\" (UID: \"eed3ce9f-0c0b-4559-8eef-71ac47313c8d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-lnwj8"
Sep 16 18:47:34.444369 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.444349    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eed3ce9f-0c0b-4559-8eef-71ac47313c8d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-lnwj8\" (UID: \"eed3ce9f-0c0b-4559-8eef-71ac47313c8d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-lnwj8"
Sep 16 18:47:34.463167 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.463137    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7l6t\" (UniqueName: \"kubernetes.io/projected/eed3ce9f-0c0b-4559-8eef-71ac47313c8d-kube-api-access-x7l6t\") pod \"kuadrant-operator-controller-manager-6d4cb79c-lnwj8\" (UID: \"eed3ce9f-0c0b-4559-8eef-71ac47313c8d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-lnwj8"
Sep 16 18:47:34.569546 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.569469    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-lnwj8"
Sep 16 18:47:34.901581 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.901511    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-lnwj8"]
Sep 16 18:47:34.915977 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:47:34.915949    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeed3ce9f_0c0b_4559_8eef_71ac47313c8d.slice/crio-b43a9703c5f8412a02d5cd191df31e2fdca376049c6c4b38a3bac71d8b31499f WatchSource:0}: Error finding container b43a9703c5f8412a02d5cd191df31e2fdca376049c6c4b38a3bac71d8b31499f: Status 404 returned error can't find the container with id b43a9703c5f8412a02d5cd191df31e2fdca376049c6c4b38a3bac71d8b31499f
Sep 16 18:47:34.962976 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:34.962948    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-lnwj8" event={"ID":"eed3ce9f-0c0b-4559-8eef-71ac47313c8d","Type":"ContainerStarted","Data":"b43a9703c5f8412a02d5cd191df31e2fdca376049c6c4b38a3bac71d8b31499f"}
Sep 16 18:47:36.909700 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:36.909665    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-24fl9"]
Sep 16 18:47:36.913375 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:36.913354    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-24fl9"
Sep 16 18:47:36.916532 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:36.916511    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-d998v\""
Sep 16 18:47:36.937077 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:36.937052    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-24fl9"]
Sep 16 18:47:36.967701 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:36.967677    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dlsx\" (UniqueName: \"kubernetes.io/projected/4af900d0-1ca8-4ee6-9d51-2774904b4ac1-kube-api-access-5dlsx\") pod \"authorino-operator-6fc8f864f4-24fl9\" (UID: \"4af900d0-1ca8-4ee6-9d51-2774904b4ac1\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-24fl9"
Sep 16 18:47:37.068526 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:37.068502    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dlsx\" (UniqueName: \"kubernetes.io/projected/4af900d0-1ca8-4ee6-9d51-2774904b4ac1-kube-api-access-5dlsx\") pod \"authorino-operator-6fc8f864f4-24fl9\" (UID: \"4af900d0-1ca8-4ee6-9d51-2774904b4ac1\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-24fl9"
Sep 16 18:47:37.081714 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:37.081690    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dlsx\" (UniqueName: \"kubernetes.io/projected/4af900d0-1ca8-4ee6-9d51-2774904b4ac1-kube-api-access-5dlsx\") pod \"authorino-operator-6fc8f864f4-24fl9\" (UID: \"4af900d0-1ca8-4ee6-9d51-2774904b4ac1\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-24fl9"
Sep 16 18:47:37.225389 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:37.225361    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-24fl9"
Sep 16 18:47:37.518833 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:37.518802    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-24fl9"]
Sep 16 18:47:38.591590 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:47:38.591532    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af900d0_1ca8_4ee6_9d51_2774904b4ac1.slice/crio-f20edabbd07fa094a81ad37524e803dfad39ce43df77c697a14eedd9c3da230e WatchSource:0}: Error finding container f20edabbd07fa094a81ad37524e803dfad39ce43df77c697a14eedd9c3da230e: Status 404 returned error can't find the container with id f20edabbd07fa094a81ad37524e803dfad39ce43df77c697a14eedd9c3da230e
Sep 16 18:47:38.985323 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:38.985290    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-lnwj8" event={"ID":"eed3ce9f-0c0b-4559-8eef-71ac47313c8d","Type":"ContainerStarted","Data":"8da5048de388851dc8942206d66dcd2ed3663bbe0a62914df415500e79e63127"}
Sep 16 18:47:38.985501 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:38.985347    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-lnwj8"
Sep 16 18:47:38.986664 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:38.986639    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-24fl9" event={"ID":"4af900d0-1ca8-4ee6-9d51-2774904b4ac1","Type":"ContainerStarted","Data":"f20edabbd07fa094a81ad37524e803dfad39ce43df77c697a14eedd9c3da230e"}
Sep 16 18:47:39.016524 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:39.016481    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-lnwj8" podStartSLOduration=1.260118561 podStartE2EDuration="5.01646732s" totalImagesPullingTime="3.756348759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:39.013114555 +0000 UTC m=+427.248047998" watchObservedRunningTime="2026-09-16 18:47:39.01646732 +0000 UTC m=+427.251400764"
Sep 16 18:47:39.922283 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:39.921862    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5597478dbc-8t5zp"]
Sep 16 18:47:39.925984 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:39.925847    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5597478dbc-8t5zp"
Sep 16 18:47:39.938518 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:39.938493    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5597478dbc-8t5zp"]
Sep 16 18:47:39.996404 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:39.996370    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35cda191-1b89-423c-8366-a88a210a6ffb-console-oauth-config\") pod \"console-5597478dbc-8t5zp\" (UID: \"35cda191-1b89-423c-8366-a88a210a6ffb\") " pod="openshift-console/console-5597478dbc-8t5zp"
Sep 16 18:47:39.996404 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:39.996410    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35cda191-1b89-423c-8366-a88a210a6ffb-console-serving-cert\") pod \"console-5597478dbc-8t5zp\" (UID: \"35cda191-1b89-423c-8366-a88a210a6ffb\") " pod="openshift-console/console-5597478dbc-8t5zp"
Sep 16 18:47:39.996649 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:39.996518    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35cda191-1b89-423c-8366-a88a210a6ffb-service-ca\") pod \"console-5597478dbc-8t5zp\" (UID: \"35cda191-1b89-423c-8366-a88a210a6ffb\") " pod="openshift-console/console-5597478dbc-8t5zp"
Sep 16 18:47:39.996649 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:39.996562    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jhlh\" (UniqueName: \"kubernetes.io/projected/35cda191-1b89-423c-8366-a88a210a6ffb-kube-api-access-4jhlh\") pod \"console-5597478dbc-8t5zp\" (UID: \"35cda191-1b89-423c-8366-a88a210a6ffb\") " pod="openshift-console/console-5597478dbc-8t5zp"
Sep 16 18:47:39.996649 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:39.996586    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35cda191-1b89-423c-8366-a88a210a6ffb-trusted-ca-bundle\") pod \"console-5597478dbc-8t5zp\" (UID: \"35cda191-1b89-423c-8366-a88a210a6ffb\") " pod="openshift-console/console-5597478dbc-8t5zp"
Sep 16 18:47:39.996649 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:39.996637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35cda191-1b89-423c-8366-a88a210a6ffb-oauth-serving-cert\") pod \"console-5597478dbc-8t5zp\" (UID: \"35cda191-1b89-423c-8366-a88a210a6ffb\") " pod="openshift-console/console-5597478dbc-8t5zp"
Sep 16 18:47:39.996824 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:39.996703    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35cda191-1b89-423c-8366-a88a210a6ffb-console-config\") pod \"console-5597478dbc-8t5zp\" (UID: \"35cda191-1b89-423c-8366-a88a210a6ffb\") " pod="openshift-console/console-5597478dbc-8t5zp"
Sep 16 18:47:40.097764 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:40.097720    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35cda191-1b89-423c-8366-a88a210a6ffb-service-ca\") pod \"console-5597478dbc-8t5zp\" (UID: \"35cda191-1b89-423c-8366-a88a210a6ffb\") " pod="openshift-console/console-5597478dbc-8t5zp"
Sep 16 18:47:40.097953 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:40.097875    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jhlh\" (UniqueName: \"kubernetes.io/projected/35cda191-1b89-423c-8366-a88a210a6ffb-kube-api-access-4jhlh\") pod \"console-5597478dbc-8t5zp\" (UID: \"35cda191-1b89-423c-8366-a88a210a6ffb\") " pod="openshift-console/console-5597478dbc-8t5zp"
Sep 16 18:47:40.097953 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:40.097925    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35cda191-1b89-423c-8366-a88a210a6ffb-trusted-ca-bundle\") pod \"console-5597478dbc-8t5zp\" (UID: \"35cda191-1b89-423c-8366-a88a210a6ffb\") " pod="openshift-console/console-5597478dbc-8t5zp"
Sep 16 18:47:40.098117 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:40.097958    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35cda191-1b89-423c-8366-a88a210a6ffb-oauth-serving-cert\") pod \"console-5597478dbc-8t5zp\" (UID: \"35cda191-1b89-423c-8366-a88a210a6ffb\") " pod="openshift-console/console-5597478dbc-8t5zp"
Sep 16 18:47:40.098117 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:40.098034    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35cda191-1b89-423c-8366-a88a210a6ffb-console-config\") pod \"console-5597478dbc-8t5zp\" (UID: \"35cda191-1b89-423c-8366-a88a210a6ffb\") " pod="openshift-console/console-5597478dbc-8t5zp"
Sep 16 18:47:40.098117 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:40.098084    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35cda191-1b89-423c-8366-a88a210a6ffb-console-oauth-config\") pod \"console-5597478dbc-8t5zp\" (UID: \"35cda191-1b89-423c-8366-a88a210a6ffb\") " pod="openshift-console/console-5597478dbc-8t5zp"
Sep 16 18:47:40.098285 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:40.098122    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35cda191-1b89-423c-8366-a88a210a6ffb-console-serving-cert\") pod \"console-5597478dbc-8t5zp\" (UID: \"35cda191-1b89-423c-8366-a88a210a6ffb\") " pod="openshift-console/console-5597478dbc-8t5zp"
Sep 16 18:47:40.099084 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:40.099020    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35cda191-1b89-423c-8366-a88a210a6ffb-console-config\") pod \"console-5597478dbc-8t5zp\" (UID: \"35cda191-1b89-423c-8366-a88a210a6ffb\") " pod="openshift-console/console-5597478dbc-8t5zp"
Sep 16 18:47:40.099215 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:40.099096    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35cda191-1b89-423c-8366-a88a210a6ffb-trusted-ca-bundle\") pod \"console-5597478dbc-8t5zp\" (UID: \"35cda191-1b89-423c-8366-a88a210a6ffb\") " pod="openshift-console/console-5597478dbc-8t5zp"
Sep 16 18:47:40.099215 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:40.099127    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35cda191-1b89-423c-8366-a88a210a6ffb-service-ca\") pod \"console-5597478dbc-8t5zp\" (UID: \"35cda191-1b89-423c-8366-a88a210a6ffb\") " pod="openshift-console/console-5597478dbc-8t5zp"
Sep 16 18:47:40.100080 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:40.099622    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35cda191-1b89-423c-8366-a88a210a6ffb-oauth-serving-cert\") pod \"console-5597478dbc-8t5zp\" (UID: \"35cda191-1b89-423c-8366-a88a210a6ffb\") " pod="openshift-console/console-5597478dbc-8t5zp"
Sep 16 18:47:40.105403 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:40.102782    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35cda191-1b89-423c-8366-a88a210a6ffb-console-serving-cert\") pod \"console-5597478dbc-8t5zp\" (UID: \"35cda191-1b89-423c-8366-a88a210a6ffb\") " pod="openshift-console/console-5597478dbc-8t5zp"
Sep 16 18:47:40.106531 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:40.106449    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35cda191-1b89-423c-8366-a88a210a6ffb-console-oauth-config\") pod \"console-5597478dbc-8t5zp\" (UID: \"35cda191-1b89-423c-8366-a88a210a6ffb\") " pod="openshift-console/console-5597478dbc-8t5zp"
Sep 16 18:47:40.109314 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:40.109292    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jhlh\" (UniqueName: \"kubernetes.io/projected/35cda191-1b89-423c-8366-a88a210a6ffb-kube-api-access-4jhlh\") pod \"console-5597478dbc-8t5zp\" (UID: \"35cda191-1b89-423c-8366-a88a210a6ffb\") " pod="openshift-console/console-5597478dbc-8t5zp"
Sep 16 18:47:40.237694 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:40.237653    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5597478dbc-8t5zp"
Sep 16 18:47:40.545990 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:40.545962    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5597478dbc-8t5zp"]
Sep 16 18:47:40.548022 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:47:40.547992    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35cda191_1b89_423c_8366_a88a210a6ffb.slice/crio-7a3e7dc4b1af8574aafe80bda536f13e8e9ef75010603902185b858bc3f0c208 WatchSource:0}: Error finding container 7a3e7dc4b1af8574aafe80bda536f13e8e9ef75010603902185b858bc3f0c208: Status 404 returned error can't find the container with id 7a3e7dc4b1af8574aafe80bda536f13e8e9ef75010603902185b858bc3f0c208
Sep 16 18:47:40.999409 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:40.999374    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-24fl9" event={"ID":"4af900d0-1ca8-4ee6-9d51-2774904b4ac1","Type":"ContainerStarted","Data":"21411ba91e2f45237fcbb1f30417d3c7a579ae83320b9ea3afff7fb1cebf8f45"}
Sep 16 18:47:40.999924 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:40.999499    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-24fl9"
Sep 16 18:47:41.001068 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:41.001023    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5597478dbc-8t5zp" event={"ID":"35cda191-1b89-423c-8366-a88a210a6ffb","Type":"ContainerStarted","Data":"4dc942c3e645b8321bc90d74baa53c605728ac8ebf95b3e155b9eea0c580478e"}
Sep 16 18:47:41.001165 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:41.001079    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5597478dbc-8t5zp" event={"ID":"35cda191-1b89-423c-8366-a88a210a6ffb","Type":"ContainerStarted","Data":"7a3e7dc4b1af8574aafe80bda536f13e8e9ef75010603902185b858bc3f0c208"}
Sep 16 18:47:41.025066 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:41.025011    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6fc8f864f4-24fl9" podStartSLOduration=3.348297616 podStartE2EDuration="5.024997989s" totalImagesPullingTime="1.676700373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:41.024703873 +0000 UTC m=+429.259637355" watchObservedRunningTime="2026-09-16 18:47:41.024997989 +0000 UTC m=+429.259931445"
Sep 16 18:47:41.050297 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:41.050257    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5597478dbc-8t5zp" podStartSLOduration=2.050241694 podStartE2EDuration="2.050241694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:41.048912577 +0000 UTC m=+429.283846021" watchObservedRunningTime="2026-09-16 18:47:41.050241694 +0000 UTC m=+429.285175148"
Sep 16 18:47:49.995645 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:49.995604    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-lnwj8"
Sep 16 18:47:50.238681 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:50.238655    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5597478dbc-8t5zp"
Sep 16 18:47:50.238846 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:50.238704    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5597478dbc-8t5zp"
Sep 16 18:47:50.243194 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:50.243171    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5597478dbc-8t5zp"
Sep 16 18:47:51.055218 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:51.055189    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5597478dbc-8t5zp"
Sep 16 18:47:51.138873 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:51.138846    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67d786cbdf-vcw8m"]
Sep 16 18:47:52.008442 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:47:52.008404    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-24fl9"
Sep 16 18:48:16.158202 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:16.158129    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-67d786cbdf-vcw8m" podUID="48e92cab-8ddd-4b46-9e79-c75c475399d9" containerName="console" containerID="cri-o://e2d3bd559e20339dafbfa1d5cd0bc7667d1a37f032ec46e6a26843b6b03c57bb" gracePeriod=15
Sep 16 18:48:16.400015 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:16.399992    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67d786cbdf-vcw8m"
Sep 16 18:48:16.456076 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:16.456033    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/48e92cab-8ddd-4b46-9e79-c75c475399d9-console-serving-cert\") pod \"48e92cab-8ddd-4b46-9e79-c75c475399d9\" (UID: \"48e92cab-8ddd-4b46-9e79-c75c475399d9\") "
Sep 16 18:48:16.456215 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:16.456092    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/48e92cab-8ddd-4b46-9e79-c75c475399d9-oauth-serving-cert\") pod \"48e92cab-8ddd-4b46-9e79-c75c475399d9\" (UID: \"48e92cab-8ddd-4b46-9e79-c75c475399d9\") "
Sep 16 18:48:16.456215 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:16.456123    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bs2d\" (UniqueName: \"kubernetes.io/projected/48e92cab-8ddd-4b46-9e79-c75c475399d9-kube-api-access-7bs2d\") pod \"48e92cab-8ddd-4b46-9e79-c75c475399d9\" (UID: \"48e92cab-8ddd-4b46-9e79-c75c475399d9\") "
Sep 16 18:48:16.456215 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:16.456152    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/48e92cab-8ddd-4b46-9e79-c75c475399d9-service-ca\") pod \"48e92cab-8ddd-4b46-9e79-c75c475399d9\" (UID: \"48e92cab-8ddd-4b46-9e79-c75c475399d9\") "
Sep 16 18:48:16.456215 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:16.456193    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/48e92cab-8ddd-4b46-9e79-c75c475399d9-console-oauth-config\") pod \"48e92cab-8ddd-4b46-9e79-c75c475399d9\" (UID: \"48e92cab-8ddd-4b46-9e79-c75c475399d9\") "
Sep 16 18:48:16.456421 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:16.456215    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48e92cab-8ddd-4b46-9e79-c75c475399d9-trusted-ca-bundle\") pod \"48e92cab-8ddd-4b46-9e79-c75c475399d9\" (UID: \"48e92cab-8ddd-4b46-9e79-c75c475399d9\") "
Sep 16 18:48:16.456421 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:16.456253    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/48e92cab-8ddd-4b46-9e79-c75c475399d9-console-config\") pod \"48e92cab-8ddd-4b46-9e79-c75c475399d9\" (UID: \"48e92cab-8ddd-4b46-9e79-c75c475399d9\") "
Sep 16 18:48:16.456671 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:16.456634    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48e92cab-8ddd-4b46-9e79-c75c475399d9-service-ca" (OuterVolumeSpecName: "service-ca") pod "48e92cab-8ddd-4b46-9e79-c75c475399d9" (UID: "48e92cab-8ddd-4b46-9e79-c75c475399d9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:16.456671 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:16.456650    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48e92cab-8ddd-4b46-9e79-c75c475399d9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "48e92cab-8ddd-4b46-9e79-c75c475399d9" (UID: "48e92cab-8ddd-4b46-9e79-c75c475399d9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:16.456871 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:16.456702    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48e92cab-8ddd-4b46-9e79-c75c475399d9-console-config" (OuterVolumeSpecName: "console-config") pod "48e92cab-8ddd-4b46-9e79-c75c475399d9" (UID: "48e92cab-8ddd-4b46-9e79-c75c475399d9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:16.456871 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:16.456710    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48e92cab-8ddd-4b46-9e79-c75c475399d9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "48e92cab-8ddd-4b46-9e79-c75c475399d9" (UID: "48e92cab-8ddd-4b46-9e79-c75c475399d9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:16.458076 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:16.458054    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48e92cab-8ddd-4b46-9e79-c75c475399d9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "48e92cab-8ddd-4b46-9e79-c75c475399d9" (UID: "48e92cab-8ddd-4b46-9e79-c75c475399d9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:16.458651 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:16.458632    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48e92cab-8ddd-4b46-9e79-c75c475399d9-kube-api-access-7bs2d" (OuterVolumeSpecName: "kube-api-access-7bs2d") pod "48e92cab-8ddd-4b46-9e79-c75c475399d9" (UID: "48e92cab-8ddd-4b46-9e79-c75c475399d9"). InnerVolumeSpecName "kube-api-access-7bs2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:48:16.458702 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:16.458643    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48e92cab-8ddd-4b46-9e79-c75c475399d9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "48e92cab-8ddd-4b46-9e79-c75c475399d9" (UID: "48e92cab-8ddd-4b46-9e79-c75c475399d9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:16.556933 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:16.556905    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/48e92cab-8ddd-4b46-9e79-c75c475399d9-console-serving-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:48:16.556933 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:16.556929    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/48e92cab-8ddd-4b46-9e79-c75c475399d9-oauth-serving-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:48:16.557091 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:16.556938    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bs2d\" (UniqueName: \"kubernetes.io/projected/48e92cab-8ddd-4b46-9e79-c75c475399d9-kube-api-access-7bs2d\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:48:16.557091 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:16.556947    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/48e92cab-8ddd-4b46-9e79-c75c475399d9-service-ca\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:48:16.557091 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:16.556957    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/48e92cab-8ddd-4b46-9e79-c75c475399d9-console-oauth-config\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:48:16.557091 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:16.556965    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48e92cab-8ddd-4b46-9e79-c75c475399d9-trusted-ca-bundle\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:48:16.557091 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:16.556973    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/48e92cab-8ddd-4b46-9e79-c75c475399d9-console-config\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:48:17.192130 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:17.192088    2578 generic.go:356] "Generic (PLEG): container finished" podID="48e92cab-8ddd-4b46-9e79-c75c475399d9" containerID="e2d3bd559e20339dafbfa1d5cd0bc7667d1a37f032ec46e6a26843b6b03c57bb" exitCode=0
Sep 16 18:48:17.192525 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:17.192151    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67d786cbdf-vcw8m"
Sep 16 18:48:17.192525 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:17.192181    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67d786cbdf-vcw8m" event={"ID":"48e92cab-8ddd-4b46-9e79-c75c475399d9","Type":"ContainerDied","Data":"e2d3bd559e20339dafbfa1d5cd0bc7667d1a37f032ec46e6a26843b6b03c57bb"}
Sep 16 18:48:17.192525 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:17.192253    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67d786cbdf-vcw8m" event={"ID":"48e92cab-8ddd-4b46-9e79-c75c475399d9","Type":"ContainerDied","Data":"49bdcce1bc201e87a86d0413cf976299de9d9353ffdb7af34d72f38117c27b64"}
Sep 16 18:48:17.192525 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:17.192277    2578 scope.go:117] "RemoveContainer" containerID="e2d3bd559e20339dafbfa1d5cd0bc7667d1a37f032ec46e6a26843b6b03c57bb"
Sep 16 18:48:17.204604 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:17.204584    2578 scope.go:117] "RemoveContainer" containerID="e2d3bd559e20339dafbfa1d5cd0bc7667d1a37f032ec46e6a26843b6b03c57bb"
Sep 16 18:48:17.204980 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:48:17.204953    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2d3bd559e20339dafbfa1d5cd0bc7667d1a37f032ec46e6a26843b6b03c57bb\": container with ID starting with e2d3bd559e20339dafbfa1d5cd0bc7667d1a37f032ec46e6a26843b6b03c57bb not found: ID does not exist" containerID="e2d3bd559e20339dafbfa1d5cd0bc7667d1a37f032ec46e6a26843b6b03c57bb"
Sep 16 18:48:17.205136 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:17.204987    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2d3bd559e20339dafbfa1d5cd0bc7667d1a37f032ec46e6a26843b6b03c57bb"} err="failed to get container status \"e2d3bd559e20339dafbfa1d5cd0bc7667d1a37f032ec46e6a26843b6b03c57bb\": rpc error: code = NotFound desc = could not find container \"e2d3bd559e20339dafbfa1d5cd0bc7667d1a37f032ec46e6a26843b6b03c57bb\": container with ID starting with e2d3bd559e20339dafbfa1d5cd0bc7667d1a37f032ec46e6a26843b6b03c57bb not found: ID does not exist"
Sep 16 18:48:17.231461 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:17.231438    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67d786cbdf-vcw8m"]
Sep 16 18:48:17.265805 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:17.265780    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67d786cbdf-vcw8m"]
Sep 16 18:48:18.217005 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:18.216966    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48e92cab-8ddd-4b46-9e79-c75c475399d9" path="/var/lib/kubelet/pods/48e92cab-8ddd-4b46-9e79-c75c475399d9/volumes"
Sep 16 18:48:22.775383 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:22.775349    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-ppc76"]
Sep 16 18:48:22.775856 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:22.775773    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48e92cab-8ddd-4b46-9e79-c75c475399d9" containerName="console"
Sep 16 18:48:22.775856 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:22.775792    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="48e92cab-8ddd-4b46-9e79-c75c475399d9" containerName="console"
Sep 16 18:48:22.775973 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:22.775885    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="48e92cab-8ddd-4b46-9e79-c75c475399d9" containerName="console"
Sep 16 18:48:22.778313 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:22.778293    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-ppc76"
Sep 16 18:48:22.781401 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:22.781374    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pnvqr\""
Sep 16 18:48:22.781518 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:22.781379    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 18:48:22.790612 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:22.790588    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-ppc76"]
Sep 16 18:48:22.874903 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:22.874875    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-ppc76"]
Sep 16 18:48:22.904954 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:22.904928    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f8fdc6f-3490-4d02-80fc-5622bf776be7-config-file\") pod \"limitador-limitador-8588749977-ppc76\" (UID: \"8f8fdc6f-3490-4d02-80fc-5622bf776be7\") " pod="kuadrant-system/limitador-limitador-8588749977-ppc76"
Sep 16 18:48:22.905101 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:22.904997    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r56hv\" (UniqueName: \"kubernetes.io/projected/8f8fdc6f-3490-4d02-80fc-5622bf776be7-kube-api-access-r56hv\") pod \"limitador-limitador-8588749977-ppc76\" (UID: \"8f8fdc6f-3490-4d02-80fc-5622bf776be7\") " pod="kuadrant-system/limitador-limitador-8588749977-ppc76"
Sep 16 18:48:23.006332 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:23.006298    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r56hv\" (UniqueName: \"kubernetes.io/projected/8f8fdc6f-3490-4d02-80fc-5622bf776be7-kube-api-access-r56hv\") pod \"limitador-limitador-8588749977-ppc76\" (UID: \"8f8fdc6f-3490-4d02-80fc-5622bf776be7\") " pod="kuadrant-system/limitador-limitador-8588749977-ppc76"
Sep 16 18:48:23.006544 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:23.006353    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f8fdc6f-3490-4d02-80fc-5622bf776be7-config-file\") pod \"limitador-limitador-8588749977-ppc76\" (UID: \"8f8fdc6f-3490-4d02-80fc-5622bf776be7\") " pod="kuadrant-system/limitador-limitador-8588749977-ppc76"
Sep 16 18:48:23.006892 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:23.006872    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f8fdc6f-3490-4d02-80fc-5622bf776be7-config-file\") pod \"limitador-limitador-8588749977-ppc76\" (UID: \"8f8fdc6f-3490-4d02-80fc-5622bf776be7\") " pod="kuadrant-system/limitador-limitador-8588749977-ppc76"
Sep 16 18:48:23.030704 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:23.030643    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r56hv\" (UniqueName: \"kubernetes.io/projected/8f8fdc6f-3490-4d02-80fc-5622bf776be7-kube-api-access-r56hv\") pod \"limitador-limitador-8588749977-ppc76\" (UID: \"8f8fdc6f-3490-4d02-80fc-5622bf776be7\") " pod="kuadrant-system/limitador-limitador-8588749977-ppc76"
Sep 16 18:48:23.087624 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:23.087596    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-ppc76"
Sep 16 18:48:23.361378 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:23.361357    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-ppc76"]
Sep 16 18:48:23.363702 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:48:23.363675    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8fdc6f_3490_4d02_80fc_5622bf776be7.slice/crio-d83d3b1f6a35f90a8da3215b9b3f6f7d5bf8594f4a948cd8f0c4f7b4a3d79691 WatchSource:0}: Error finding container d83d3b1f6a35f90a8da3215b9b3f6f7d5bf8594f4a948cd8f0c4f7b4a3d79691: Status 404 returned error can't find the container with id d83d3b1f6a35f90a8da3215b9b3f6f7d5bf8594f4a948cd8f0c4f7b4a3d79691
Sep 16 18:48:24.229533 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:24.229499    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-ppc76" event={"ID":"8f8fdc6f-3490-4d02-80fc-5622bf776be7","Type":"ContainerStarted","Data":"d83d3b1f6a35f90a8da3215b9b3f6f7d5bf8594f4a948cd8f0c4f7b4a3d79691"}
Sep 16 18:48:25.237130 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:25.237091    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-ppc76" event={"ID":"8f8fdc6f-3490-4d02-80fc-5622bf776be7","Type":"ContainerStarted","Data":"8a5db6aab5d43dda39645ea45b9ab94879e6d81de89668b1a1b0c18dfc2139aa"}
Sep 16 18:48:25.237565 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:25.237243    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-ppc76"
Sep 16 18:48:25.260073 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:25.257243    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-ppc76" podStartSLOduration=2.265147459 podStartE2EDuration="3.257219998s" totalImagesPullingTime="992.072539ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:25.253425156 +0000 UTC m=+473.488358602" watchObservedRunningTime="2026-09-16 18:48:25.257219998 +0000 UTC m=+473.492153443"
Sep 16 18:48:36.243152 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:36.243123    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-ppc76"
Sep 16 18:48:37.168412 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:37.168375    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-ppc76"]
Sep 16 18:48:37.168615 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:37.168592    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-ppc76" podUID="8f8fdc6f-3490-4d02-80fc-5622bf776be7" containerName="limitador" containerID="cri-o://8a5db6aab5d43dda39645ea45b9ab94879e6d81de89668b1a1b0c18dfc2139aa" gracePeriod=30
Sep 16 18:48:37.716667 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:37.716639    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-ppc76"
Sep 16 18:48:37.816260 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:37.816199    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r56hv\" (UniqueName: \"kubernetes.io/projected/8f8fdc6f-3490-4d02-80fc-5622bf776be7-kube-api-access-r56hv\") pod \"8f8fdc6f-3490-4d02-80fc-5622bf776be7\" (UID: \"8f8fdc6f-3490-4d02-80fc-5622bf776be7\") "
Sep 16 18:48:37.816377 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:37.816314    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f8fdc6f-3490-4d02-80fc-5622bf776be7-config-file\") pod \"8f8fdc6f-3490-4d02-80fc-5622bf776be7\" (UID: \"8f8fdc6f-3490-4d02-80fc-5622bf776be7\") "
Sep 16 18:48:37.816670 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:37.816644    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f8fdc6f-3490-4d02-80fc-5622bf776be7-config-file" (OuterVolumeSpecName: "config-file") pod "8f8fdc6f-3490-4d02-80fc-5622bf776be7" (UID: "8f8fdc6f-3490-4d02-80fc-5622bf776be7"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:37.818347 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:37.818326    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f8fdc6f-3490-4d02-80fc-5622bf776be7-kube-api-access-r56hv" (OuterVolumeSpecName: "kube-api-access-r56hv") pod "8f8fdc6f-3490-4d02-80fc-5622bf776be7" (UID: "8f8fdc6f-3490-4d02-80fc-5622bf776be7"). InnerVolumeSpecName "kube-api-access-r56hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:48:37.917820 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:37.917797    2578 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f8fdc6f-3490-4d02-80fc-5622bf776be7-config-file\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:48:37.917820 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:37.917818    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r56hv\" (UniqueName: \"kubernetes.io/projected/8f8fdc6f-3490-4d02-80fc-5622bf776be7-kube-api-access-r56hv\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:48:38.301096 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:38.301064    2578 generic.go:356] "Generic (PLEG): container finished" podID="8f8fdc6f-3490-4d02-80fc-5622bf776be7" containerID="8a5db6aab5d43dda39645ea45b9ab94879e6d81de89668b1a1b0c18dfc2139aa" exitCode=0
Sep 16 18:48:38.301213 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:38.301106    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-ppc76"
Sep 16 18:48:38.301213 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:38.301150    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-ppc76" event={"ID":"8f8fdc6f-3490-4d02-80fc-5622bf776be7","Type":"ContainerDied","Data":"8a5db6aab5d43dda39645ea45b9ab94879e6d81de89668b1a1b0c18dfc2139aa"}
Sep 16 18:48:38.301213 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:38.301199    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-ppc76" event={"ID":"8f8fdc6f-3490-4d02-80fc-5622bf776be7","Type":"ContainerDied","Data":"d83d3b1f6a35f90a8da3215b9b3f6f7d5bf8594f4a948cd8f0c4f7b4a3d79691"}
Sep 16 18:48:38.301313 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:38.301224    2578 scope.go:117] "RemoveContainer" containerID="8a5db6aab5d43dda39645ea45b9ab94879e6d81de89668b1a1b0c18dfc2139aa"
Sep 16 18:48:38.309718 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:38.309691    2578 scope.go:117] "RemoveContainer" containerID="8a5db6aab5d43dda39645ea45b9ab94879e6d81de89668b1a1b0c18dfc2139aa"
Sep 16 18:48:38.309985 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:48:38.309968    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a5db6aab5d43dda39645ea45b9ab94879e6d81de89668b1a1b0c18dfc2139aa\": container with ID starting with 8a5db6aab5d43dda39645ea45b9ab94879e6d81de89668b1a1b0c18dfc2139aa not found: ID does not exist" containerID="8a5db6aab5d43dda39645ea45b9ab94879e6d81de89668b1a1b0c18dfc2139aa"
Sep 16 18:48:38.310031 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:38.309992    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a5db6aab5d43dda39645ea45b9ab94879e6d81de89668b1a1b0c18dfc2139aa"} err="failed to get container status \"8a5db6aab5d43dda39645ea45b9ab94879e6d81de89668b1a1b0c18dfc2139aa\": rpc error: code = NotFound desc = could not find container \"8a5db6aab5d43dda39645ea45b9ab94879e6d81de89668b1a1b0c18dfc2139aa\": container with ID starting with 8a5db6aab5d43dda39645ea45b9ab94879e6d81de89668b1a1b0c18dfc2139aa not found: ID does not exist"
Sep 16 18:48:38.323156 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:38.323130    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-ppc76"]
Sep 16 18:48:38.328620 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:38.328600    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-ppc76"]
Sep 16 18:48:40.217600 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:40.217553    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f8fdc6f-3490-4d02-80fc-5622bf776be7" path="/var/lib/kubelet/pods/8f8fdc6f-3490-4d02-80fc-5622bf776be7/volumes"
Sep 16 18:48:46.592893 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:46.592856    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d9898bd99-jlvf7"]
Sep 16 18:48:46.593402 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:46.593223    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f8fdc6f-3490-4d02-80fc-5622bf776be7" containerName="limitador"
Sep 16 18:48:46.593402 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:46.593242    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f8fdc6f-3490-4d02-80fc-5622bf776be7" containerName="limitador"
Sep 16 18:48:46.593402 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:46.593349    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f8fdc6f-3490-4d02-80fc-5622bf776be7" containerName="limitador"
Sep 16 18:48:46.597542 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:46.597523    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-jlvf7"
Sep 16 18:48:46.600094 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:46.600075    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jf5d4\""
Sep 16 18:48:46.601313 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:46.601295    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 16 18:48:46.608155 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:46.608132    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-jlvf7"]
Sep 16 18:48:46.683297 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:46.683268    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9g69\" (UniqueName: \"kubernetes.io/projected/5f868101-ea7a-4685-8c91-d67140498fd3-kube-api-access-x9g69\") pod \"authorino-6d9898bd99-jlvf7\" (UID: \"5f868101-ea7a-4685-8c91-d67140498fd3\") " pod="kuadrant-system/authorino-6d9898bd99-jlvf7"
Sep 16 18:48:46.683397 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:46.683300    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5f868101-ea7a-4685-8c91-d67140498fd3-tls-cert\") pod \"authorino-6d9898bd99-jlvf7\" (UID: \"5f868101-ea7a-4685-8c91-d67140498fd3\") " pod="kuadrant-system/authorino-6d9898bd99-jlvf7"
Sep 16 18:48:46.783609 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:46.783583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9g69\" (UniqueName: \"kubernetes.io/projected/5f868101-ea7a-4685-8c91-d67140498fd3-kube-api-access-x9g69\") pod \"authorino-6d9898bd99-jlvf7\" (UID: \"5f868101-ea7a-4685-8c91-d67140498fd3\") " pod="kuadrant-system/authorino-6d9898bd99-jlvf7"
Sep 16 18:48:46.783724 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:46.783615    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5f868101-ea7a-4685-8c91-d67140498fd3-tls-cert\") pod \"authorino-6d9898bd99-jlvf7\" (UID: \"5f868101-ea7a-4685-8c91-d67140498fd3\") " pod="kuadrant-system/authorino-6d9898bd99-jlvf7"
Sep 16 18:48:46.786029 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:46.786010    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5f868101-ea7a-4685-8c91-d67140498fd3-tls-cert\") pod \"authorino-6d9898bd99-jlvf7\" (UID: \"5f868101-ea7a-4685-8c91-d67140498fd3\") " pod="kuadrant-system/authorino-6d9898bd99-jlvf7"
Sep 16 18:48:46.793916 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:46.793895    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9g69\" (UniqueName: \"kubernetes.io/projected/5f868101-ea7a-4685-8c91-d67140498fd3-kube-api-access-x9g69\") pod \"authorino-6d9898bd99-jlvf7\" (UID: \"5f868101-ea7a-4685-8c91-d67140498fd3\") " pod="kuadrant-system/authorino-6d9898bd99-jlvf7"
Sep 16 18:48:46.906239 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:46.906187    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-jlvf7"
Sep 16 18:48:47.182429 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:47.182407    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-jlvf7"]
Sep 16 18:48:47.184386 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:48:47.184361    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f868101_ea7a_4685_8c91_d67140498fd3.slice/crio-73bcb2be54dd244d524027d121b78f45a279fa5c535e197bac98501a2d96ac7a WatchSource:0}: Error finding container 73bcb2be54dd244d524027d121b78f45a279fa5c535e197bac98501a2d96ac7a: Status 404 returned error can't find the container with id 73bcb2be54dd244d524027d121b78f45a279fa5c535e197bac98501a2d96ac7a
Sep 16 18:48:47.348750 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:47.348720    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-jlvf7" event={"ID":"5f868101-ea7a-4685-8c91-d67140498fd3","Type":"ContainerStarted","Data":"73bcb2be54dd244d524027d121b78f45a279fa5c535e197bac98501a2d96ac7a"}
Sep 16 18:48:49.360699 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:49.360658    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-jlvf7" event={"ID":"5f868101-ea7a-4685-8c91-d67140498fd3","Type":"ContainerStarted","Data":"8811ab6c645bcac0b2c99bbbeffd1c208155557eb4129d2fb3d4b34265a98823"}
Sep 16 18:48:49.378641 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:48:49.378589    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d9898bd99-jlvf7" podStartSLOduration=1.79812613 podStartE2EDuration="3.378576272s" totalImagesPullingTime="1.580450142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:49.378187791 +0000 UTC m=+497.613121236" watchObservedRunningTime="2026-09-16 18:48:49.378576272 +0000 UTC m=+497.613509729"
Sep 16 18:49:09.026959 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:09.026928    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-4bmk9"]
Sep 16 18:49:09.030073 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:09.030056    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4bmk9"
Sep 16 18:49:09.046369 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:09.046345    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f68303c7-a114-497f-be72-c30f89d7dd1b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-4bmk9\" (UID: \"f68303c7-a114-497f-be72-c30f89d7dd1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4bmk9"
Sep 16 18:49:09.046471 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:09.046381    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f68303c7-a114-497f-be72-c30f89d7dd1b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-4bmk9\" (UID: \"f68303c7-a114-497f-be72-c30f89d7dd1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4bmk9"
Sep 16 18:49:09.046471 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:09.046408    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f68303c7-a114-497f-be72-c30f89d7dd1b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-4bmk9\" (UID: \"f68303c7-a114-497f-be72-c30f89d7dd1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4bmk9"
Sep 16 18:49:09.046471 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:09.046444    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f68303c7-a114-497f-be72-c30f89d7dd1b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-4bmk9\" (UID: \"f68303c7-a114-497f-be72-c30f89d7dd1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4bmk9"
Sep 16 18:49:09.046471 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:09.046458    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbl6t\" (UniqueName: \"kubernetes.io/projected/f68303c7-a114-497f-be72-c30f89d7dd1b-kube-api-access-jbl6t\") pod \"istiod-openshift-gateway-5677ccb49b-4bmk9\" (UID: \"f68303c7-a114-497f-be72-c30f89d7dd1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4bmk9"
Sep 16 18:49:09.046616 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:09.046483    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f68303c7-a114-497f-be72-c30f89d7dd1b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-4bmk9\" (UID: \"f68303c7-a114-497f-be72-c30f89d7dd1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4bmk9"
Sep 16 18:49:09.046616 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:09.046500    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f68303c7-a114-497f-be72-c30f89d7dd1b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-4bmk9\" (UID: \"f68303c7-a114-497f-be72-c30f89d7dd1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4bmk9"
Sep 16 18:49:09.057096 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:09.057074    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-4bmk9"]
Sep 16 18:49:09.147385 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:09.147359    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f68303c7-a114-497f-be72-c30f89d7dd1b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-4bmk9\" (UID: \"f68303c7-a114-497f-be72-c30f89d7dd1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4bmk9"
Sep 16 18:49:09.147491 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:09.147395    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f68303c7-a114-497f-be72-c30f89d7dd1b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-4bmk9\" (UID: \"f68303c7-a114-497f-be72-c30f89d7dd1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4bmk9"
Sep 16 18:49:09.147491 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:09.147418    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f68303c7-a114-497f-be72-c30f89d7dd1b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-4bmk9\" (UID: \"f68303c7-a114-497f-be72-c30f89d7dd1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4bmk9"
Sep 16 18:49:09.147491 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:09.147445    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f68303c7-a114-497f-be72-c30f89d7dd1b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-4bmk9\" (UID: \"f68303c7-a114-497f-be72-c30f89d7dd1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4bmk9"
Sep 16 18:49:09.147593 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:09.147561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbl6t\" (UniqueName: \"kubernetes.io/projected/f68303c7-a114-497f-be72-c30f89d7dd1b-kube-api-access-jbl6t\") pod \"istiod-openshift-gateway-5677ccb49b-4bmk9\" (UID: \"f68303c7-a114-497f-be72-c30f89d7dd1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4bmk9"
Sep 16 18:49:09.147637 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:09.147600    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f68303c7-a114-497f-be72-c30f89d7dd1b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-4bmk9\" (UID: \"f68303c7-a114-497f-be72-c30f89d7dd1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4bmk9"
Sep 16 18:49:09.147637 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:09.147625    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f68303c7-a114-497f-be72-c30f89d7dd1b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-4bmk9\" (UID: \"f68303c7-a114-497f-be72-c30f89d7dd1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4bmk9"
Sep 16 18:49:09.148158 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:09.148131    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f68303c7-a114-497f-be72-c30f89d7dd1b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-4bmk9\" (UID: \"f68303c7-a114-497f-be72-c30f89d7dd1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4bmk9"
Sep 16 18:49:09.149705 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:09.149685    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f68303c7-a114-497f-be72-c30f89d7dd1b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-4bmk9\" (UID: \"f68303c7-a114-497f-be72-c30f89d7dd1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4bmk9"
Sep 16 18:49:09.149874 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:09.149848    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f68303c7-a114-497f-be72-c30f89d7dd1b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-4bmk9\" (UID: \"f68303c7-a114-497f-be72-c30f89d7dd1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4bmk9"
Sep 16 18:49:09.150012 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:09.149997    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f68303c7-a114-497f-be72-c30f89d7dd1b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-4bmk9\" (UID: \"f68303c7-a114-497f-be72-c30f89d7dd1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4bmk9"
Sep 16 18:49:09.150093 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:09.150076    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f68303c7-a114-497f-be72-c30f89d7dd1b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-4bmk9\" (UID: \"f68303c7-a114-497f-be72-c30f89d7dd1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4bmk9"
Sep 16 18:49:09.161366 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:09.161347    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f68303c7-a114-497f-be72-c30f89d7dd1b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-4bmk9\" (UID: \"f68303c7-a114-497f-be72-c30f89d7dd1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4bmk9"
Sep 16 18:49:09.163946 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:09.163916    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbl6t\" (UniqueName: \"kubernetes.io/projected/f68303c7-a114-497f-be72-c30f89d7dd1b-kube-api-access-jbl6t\") pod \"istiod-openshift-gateway-5677ccb49b-4bmk9\" (UID: \"f68303c7-a114-497f-be72-c30f89d7dd1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4bmk9"
Sep 16 18:49:09.338266 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:09.338220    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4bmk9"
Sep 16 18:49:09.615819 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:09.615745    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-4bmk9"]
Sep 16 18:49:09.618743 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:49:09.618715    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf68303c7_a114_497f_be72_c30f89d7dd1b.slice/crio-63d310ad40ba73994f4ffb40724ecf58a00f57823d4c6049f3da2cf052483988 WatchSource:0}: Error finding container 63d310ad40ba73994f4ffb40724ecf58a00f57823d4c6049f3da2cf052483988: Status 404 returned error can't find the container with id 63d310ad40ba73994f4ffb40724ecf58a00f57823d4c6049f3da2cf052483988
Sep 16 18:49:09.620629 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:09.620597    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:49:09.620691 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:09.620658    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:49:10.465241 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:10.465207    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4bmk9" event={"ID":"f68303c7-a114-497f-be72-c30f89d7dd1b","Type":"ContainerStarted","Data":"24446e247383b4ae8b8f60d93ed834cb9ff9f2eecabf13dc5943f99d64e7e0b9"}
Sep 16 18:49:10.465621 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:10.465257    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4bmk9" event={"ID":"f68303c7-a114-497f-be72-c30f89d7dd1b","Type":"ContainerStarted","Data":"63d310ad40ba73994f4ffb40724ecf58a00f57823d4c6049f3da2cf052483988"}
Sep 16 18:49:10.465621 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:10.465384    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4bmk9"
Sep 16 18:49:10.466791 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:10.466769    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4bmk9"
Sep 16 18:49:10.515293 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:10.515255    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-4bmk9" podStartSLOduration=2.515244347 podStartE2EDuration="2.515244347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:10.512593189 +0000 UTC m=+518.747526632" watchObservedRunningTime="2026-09-16 18:49:10.515244347 +0000 UTC m=+518.750177789"
Sep 16 18:49:10.649879 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:10.649853    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"]
Sep 16 18:49:10.650077 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:10.650014    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46" podUID="37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f" containerName="discovery" containerID="cri-o://6c115d4295a63271f01027e78b1b0636af17e8cf1de37ed4bb86b87d79e80b85" gracePeriod=30
Sep 16 18:49:10.899532 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:10.899507    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"
Sep 16 18:49:10.962102 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:10.962071    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-istio-csr-ca-configmap\") pod \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\" (UID: \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\") "
Sep 16 18:49:10.962246 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:10.962109    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-istio-token\") pod \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\" (UID: \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\") "
Sep 16 18:49:10.962246 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:10.962152    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n47dg\" (UniqueName: \"kubernetes.io/projected/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-kube-api-access-n47dg\") pod \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\" (UID: \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\") "
Sep 16 18:49:10.962246 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:10.962180    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-istio-kubeconfig\") pod \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\" (UID: \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\") "
Sep 16 18:49:10.962246 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:10.962204    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-istio-csr-dns-cert\") pod \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\" (UID: \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\") "
Sep 16 18:49:10.962246 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:10.962233    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-cacerts\") pod \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\" (UID: \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\") "
Sep 16 18:49:10.962501 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:10.962337    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-local-certs\") pod \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\" (UID: \"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f\") "
Sep 16 18:49:10.962501 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:10.962487    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f" (UID: "37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:49:10.962644 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:10.962624    2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-istio-csr-ca-configmap\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:49:10.964493 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:10.964449    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-local-certs" (OuterVolumeSpecName: "local-certs") pod "37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f" (UID: "37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:10.964600 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:10.964505    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f" (UID: "37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:10.964722 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:10.964703    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-istio-token" (OuterVolumeSpecName: "istio-token") pod "37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f" (UID: "37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:10.964774 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:10.964751    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-cacerts" (OuterVolumeSpecName: "cacerts") pod "37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f" (UID: "37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:10.964839 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:10.964822    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f" (UID: "37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:49:10.964839 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:10.964828    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-kube-api-access-n47dg" (OuterVolumeSpecName: "kube-api-access-n47dg") pod "37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f" (UID: "37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f"). InnerVolumeSpecName "kube-api-access-n47dg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:11.062969 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:11.062911    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n47dg\" (UniqueName: \"kubernetes.io/projected/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-kube-api-access-n47dg\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:49:11.062969 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:11.062932    2578 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-istio-kubeconfig\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:49:11.062969 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:11.062941    2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-istio-csr-dns-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:49:11.062969 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:11.062949    2578 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-cacerts\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:49:11.062969 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:11.062958    2578 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-local-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:49:11.062969 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:11.062966    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f-istio-token\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:49:11.475902 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:11.475873    2578 generic.go:356] "Generic (PLEG): container finished" podID="37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f" containerID="6c115d4295a63271f01027e78b1b0636af17e8cf1de37ed4bb86b87d79e80b85" exitCode=0
Sep 16 18:49:11.476330 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:11.475921    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46" event={"ID":"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f","Type":"ContainerDied","Data":"6c115d4295a63271f01027e78b1b0636af17e8cf1de37ed4bb86b87d79e80b85"}
Sep 16 18:49:11.476330 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:11.475938    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"
Sep 16 18:49:11.476330 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:11.476021    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46" event={"ID":"37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f","Type":"ContainerDied","Data":"28200fad8e764abd61d1299bc148a57a5aacf4cf55c8e4c3317e8ce81e557a4d"}
Sep 16 18:49:11.476330 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:11.476064    2578 scope.go:117] "RemoveContainer" containerID="6c115d4295a63271f01027e78b1b0636af17e8cf1de37ed4bb86b87d79e80b85"
Sep 16 18:49:11.486020 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:11.485999    2578 scope.go:117] "RemoveContainer" containerID="6c115d4295a63271f01027e78b1b0636af17e8cf1de37ed4bb86b87d79e80b85"
Sep 16 18:49:11.486431 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:49:11.486371    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c115d4295a63271f01027e78b1b0636af17e8cf1de37ed4bb86b87d79e80b85\": container with ID starting with 6c115d4295a63271f01027e78b1b0636af17e8cf1de37ed4bb86b87d79e80b85 not found: ID does not exist" containerID="6c115d4295a63271f01027e78b1b0636af17e8cf1de37ed4bb86b87d79e80b85"
Sep 16 18:49:11.486431 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:11.486402    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c115d4295a63271f01027e78b1b0636af17e8cf1de37ed4bb86b87d79e80b85"} err="failed to get container status \"6c115d4295a63271f01027e78b1b0636af17e8cf1de37ed4bb86b87d79e80b85\": rpc error: code = NotFound desc = could not find container \"6c115d4295a63271f01027e78b1b0636af17e8cf1de37ed4bb86b87d79e80b85\": container with ID starting with 6c115d4295a63271f01027e78b1b0636af17e8cf1de37ed4bb86b87d79e80b85 not found: ID does not exist"
Sep 16 18:49:11.499982 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:11.499960    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"]
Sep 16 18:49:11.507537 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:11.507516    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-2ch46"]
Sep 16 18:49:12.217958 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:12.217922    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f" path="/var/lib/kubelet/pods/37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f/volumes"
Sep 16 18:49:21.928440 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:21.928409    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-599b79d756-g5ztn"]
Sep 16 18:49:21.928891 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:21.928693    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f" containerName="discovery"
Sep 16 18:49:21.928891 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:21.928704    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f" containerName="discovery"
Sep 16 18:49:21.928891 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:21.928756    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="37b44e19-36c7-4ce8-b2fa-3f997d3dbd2f" containerName="discovery"
Sep 16 18:49:21.933237 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:21.933220    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-599b79d756-g5ztn"
Sep 16 18:49:21.935731 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:21.935712    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 16 18:49:21.935846 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:21.935775    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-6sh5h\""
Sep 16 18:49:21.935846 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:21.935778    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:49:21.937009 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:21.936991    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:49:21.943423 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:21.943403    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-599b79d756-g5ztn"]
Sep 16 18:49:22.034417 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:22.034386    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dttp\" (UniqueName: \"kubernetes.io/projected/abd6f47b-a02a-4790-85ce-0e0cb5da5af7-kube-api-access-2dttp\") pod \"kserve-controller-manager-599b79d756-g5ztn\" (UID: \"abd6f47b-a02a-4790-85ce-0e0cb5da5af7\") " pod="kserve/kserve-controller-manager-599b79d756-g5ztn"
Sep 16 18:49:22.034567 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:22.034451    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/abd6f47b-a02a-4790-85ce-0e0cb5da5af7-cert\") pod \"kserve-controller-manager-599b79d756-g5ztn\" (UID: \"abd6f47b-a02a-4790-85ce-0e0cb5da5af7\") " pod="kserve/kserve-controller-manager-599b79d756-g5ztn"
Sep 16 18:49:22.135494 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:22.135463    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/abd6f47b-a02a-4790-85ce-0e0cb5da5af7-cert\") pod \"kserve-controller-manager-599b79d756-g5ztn\" (UID: \"abd6f47b-a02a-4790-85ce-0e0cb5da5af7\") " pod="kserve/kserve-controller-manager-599b79d756-g5ztn"
Sep 16 18:49:22.135639 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:22.135524    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dttp\" (UniqueName: \"kubernetes.io/projected/abd6f47b-a02a-4790-85ce-0e0cb5da5af7-kube-api-access-2dttp\") pod \"kserve-controller-manager-599b79d756-g5ztn\" (UID: \"abd6f47b-a02a-4790-85ce-0e0cb5da5af7\") " pod="kserve/kserve-controller-manager-599b79d756-g5ztn"
Sep 16 18:49:22.137975 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:22.137947    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/abd6f47b-a02a-4790-85ce-0e0cb5da5af7-cert\") pod \"kserve-controller-manager-599b79d756-g5ztn\" (UID: \"abd6f47b-a02a-4790-85ce-0e0cb5da5af7\") " pod="kserve/kserve-controller-manager-599b79d756-g5ztn"
Sep 16 18:49:22.144944 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:22.144919    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dttp\" (UniqueName: \"kubernetes.io/projected/abd6f47b-a02a-4790-85ce-0e0cb5da5af7-kube-api-access-2dttp\") pod \"kserve-controller-manager-599b79d756-g5ztn\" (UID: \"abd6f47b-a02a-4790-85ce-0e0cb5da5af7\") " pod="kserve/kserve-controller-manager-599b79d756-g5ztn"
Sep 16 18:49:22.243756 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:22.243699    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-599b79d756-g5ztn"
Sep 16 18:49:22.518795 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:22.518758    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-599b79d756-g5ztn"]
Sep 16 18:49:22.520541 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:49:22.520517    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd6f47b_a02a_4790_85ce_0e0cb5da5af7.slice/crio-325243384e716e0221f539ef60ffee3236b64a5bca4773d92651ae16af52052c WatchSource:0}: Error finding container 325243384e716e0221f539ef60ffee3236b64a5bca4773d92651ae16af52052c: Status 404 returned error can't find the container with id 325243384e716e0221f539ef60ffee3236b64a5bca4773d92651ae16af52052c
Sep 16 18:49:22.528598 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:22.528570    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-599b79d756-g5ztn" event={"ID":"abd6f47b-a02a-4790-85ce-0e0cb5da5af7","Type":"ContainerStarted","Data":"325243384e716e0221f539ef60ffee3236b64a5bca4773d92651ae16af52052c"}
Sep 16 18:49:25.550074 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:25.550013    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-599b79d756-g5ztn" event={"ID":"abd6f47b-a02a-4790-85ce-0e0cb5da5af7","Type":"ContainerStarted","Data":"20a32c3d376e4e42776cba6c5d17a9ec8135b8d38e85d6c81697aa46068ffc66"}
Sep 16 18:49:25.550592 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:25.550196    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-599b79d756-g5ztn"
Sep 16 18:49:25.568102 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:25.568062    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-599b79d756-g5ztn" podStartSLOduration=2.184184138 podStartE2EDuration="4.568029429s" totalImagesPullingTime="2.383845291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:25.567129729 +0000 UTC m=+533.802063198" watchObservedRunningTime="2026-09-16 18:49:25.568029429 +0000 UTC m=+533.802962870"
Sep 16 18:49:56.560982 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:49:56.560932    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-599b79d756-g5ztn"
Sep 16 18:50:05.303387 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:05.303356    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-599b79d756-g5ztn"]
Sep 16 18:50:05.303807 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:05.303550    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-599b79d756-g5ztn" podUID="abd6f47b-a02a-4790-85ce-0e0cb5da5af7" containerName="manager" containerID="cri-o://20a32c3d376e4e42776cba6c5d17a9ec8135b8d38e85d6c81697aa46068ffc66" gracePeriod=10
Sep 16 18:50:05.327304 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:05.327276    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-599b79d756-47w5x"]
Sep 16 18:50:05.329602 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:05.329585    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-599b79d756-47w5x"
Sep 16 18:50:05.339104 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:05.339078    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-599b79d756-47w5x"]
Sep 16 18:50:05.444836 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:05.444809    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gt4z\" (UniqueName: \"kubernetes.io/projected/cfec1bc7-76c3-4d11-90ea-268fa89169bc-kube-api-access-9gt4z\") pod \"kserve-controller-manager-599b79d756-47w5x\" (UID: \"cfec1bc7-76c3-4d11-90ea-268fa89169bc\") " pod="kserve/kserve-controller-manager-599b79d756-47w5x"
Sep 16 18:50:05.444950 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:05.444907    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cfec1bc7-76c3-4d11-90ea-268fa89169bc-cert\") pod \"kserve-controller-manager-599b79d756-47w5x\" (UID: \"cfec1bc7-76c3-4d11-90ea-268fa89169bc\") " pod="kserve/kserve-controller-manager-599b79d756-47w5x"
Sep 16 18:50:05.541121 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:05.541097    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-599b79d756-g5ztn"
Sep 16 18:50:05.545315 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:05.545294    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cfec1bc7-76c3-4d11-90ea-268fa89169bc-cert\") pod \"kserve-controller-manager-599b79d756-47w5x\" (UID: \"cfec1bc7-76c3-4d11-90ea-268fa89169bc\") " pod="kserve/kserve-controller-manager-599b79d756-47w5x"
Sep 16 18:50:05.545426 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:05.545347    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gt4z\" (UniqueName: \"kubernetes.io/projected/cfec1bc7-76c3-4d11-90ea-268fa89169bc-kube-api-access-9gt4z\") pod \"kserve-controller-manager-599b79d756-47w5x\" (UID: \"cfec1bc7-76c3-4d11-90ea-268fa89169bc\") " pod="kserve/kserve-controller-manager-599b79d756-47w5x"
Sep 16 18:50:05.547369 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:05.547334    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cfec1bc7-76c3-4d11-90ea-268fa89169bc-cert\") pod \"kserve-controller-manager-599b79d756-47w5x\" (UID: \"cfec1bc7-76c3-4d11-90ea-268fa89169bc\") " pod="kserve/kserve-controller-manager-599b79d756-47w5x"
Sep 16 18:50:05.553826 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:05.553776    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gt4z\" (UniqueName: \"kubernetes.io/projected/cfec1bc7-76c3-4d11-90ea-268fa89169bc-kube-api-access-9gt4z\") pod \"kserve-controller-manager-599b79d756-47w5x\" (UID: \"cfec1bc7-76c3-4d11-90ea-268fa89169bc\") " pod="kserve/kserve-controller-manager-599b79d756-47w5x"
Sep 16 18:50:05.646056 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:05.646016    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dttp\" (UniqueName: \"kubernetes.io/projected/abd6f47b-a02a-4790-85ce-0e0cb5da5af7-kube-api-access-2dttp\") pod \"abd6f47b-a02a-4790-85ce-0e0cb5da5af7\" (UID: \"abd6f47b-a02a-4790-85ce-0e0cb5da5af7\") "
Sep 16 18:50:05.646167 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:05.646080    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/abd6f47b-a02a-4790-85ce-0e0cb5da5af7-cert\") pod \"abd6f47b-a02a-4790-85ce-0e0cb5da5af7\" (UID: \"abd6f47b-a02a-4790-85ce-0e0cb5da5af7\") "
Sep 16 18:50:05.647919 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:05.647892    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd6f47b-a02a-4790-85ce-0e0cb5da5af7-kube-api-access-2dttp" (OuterVolumeSpecName: "kube-api-access-2dttp") pod "abd6f47b-a02a-4790-85ce-0e0cb5da5af7" (UID: "abd6f47b-a02a-4790-85ce-0e0cb5da5af7"). InnerVolumeSpecName "kube-api-access-2dttp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:50:05.648003 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:05.647931    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abd6f47b-a02a-4790-85ce-0e0cb5da5af7-cert" (OuterVolumeSpecName: "cert") pod "abd6f47b-a02a-4790-85ce-0e0cb5da5af7" (UID: "abd6f47b-a02a-4790-85ce-0e0cb5da5af7"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:50:05.672124 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:05.672100    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-599b79d756-47w5x"
Sep 16 18:50:05.746568 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:05.746546    2578 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/abd6f47b-a02a-4790-85ce-0e0cb5da5af7-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:50:05.746677 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:05.746571    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dttp\" (UniqueName: \"kubernetes.io/projected/abd6f47b-a02a-4790-85ce-0e0cb5da5af7-kube-api-access-2dttp\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:50:05.748604 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:05.748580    2578 generic.go:356] "Generic (PLEG): container finished" podID="abd6f47b-a02a-4790-85ce-0e0cb5da5af7" containerID="20a32c3d376e4e42776cba6c5d17a9ec8135b8d38e85d6c81697aa46068ffc66" exitCode=0
Sep 16 18:50:05.748713 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:05.748623    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-599b79d756-g5ztn"
Sep 16 18:50:05.748713 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:05.748666    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-599b79d756-g5ztn" event={"ID":"abd6f47b-a02a-4790-85ce-0e0cb5da5af7","Type":"ContainerDied","Data":"20a32c3d376e4e42776cba6c5d17a9ec8135b8d38e85d6c81697aa46068ffc66"}
Sep 16 18:50:05.748791 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:05.748727    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-599b79d756-g5ztn" event={"ID":"abd6f47b-a02a-4790-85ce-0e0cb5da5af7","Type":"ContainerDied","Data":"325243384e716e0221f539ef60ffee3236b64a5bca4773d92651ae16af52052c"}
Sep 16 18:50:05.748791 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:05.748761    2578 scope.go:117] "RemoveContainer" containerID="20a32c3d376e4e42776cba6c5d17a9ec8135b8d38e85d6c81697aa46068ffc66"
Sep 16 18:50:05.757470 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:05.757454    2578 scope.go:117] "RemoveContainer" containerID="20a32c3d376e4e42776cba6c5d17a9ec8135b8d38e85d6c81697aa46068ffc66"
Sep 16 18:50:05.757731 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:50:05.757711    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20a32c3d376e4e42776cba6c5d17a9ec8135b8d38e85d6c81697aa46068ffc66\": container with ID starting with 20a32c3d376e4e42776cba6c5d17a9ec8135b8d38e85d6c81697aa46068ffc66 not found: ID does not exist" containerID="20a32c3d376e4e42776cba6c5d17a9ec8135b8d38e85d6c81697aa46068ffc66"
Sep 16 18:50:05.757783 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:05.757740    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20a32c3d376e4e42776cba6c5d17a9ec8135b8d38e85d6c81697aa46068ffc66"} err="failed to get container status \"20a32c3d376e4e42776cba6c5d17a9ec8135b8d38e85d6c81697aa46068ffc66\": rpc error: code = NotFound desc = could not find container \"20a32c3d376e4e42776cba6c5d17a9ec8135b8d38e85d6c81697aa46068ffc66\": container with ID starting with 20a32c3d376e4e42776cba6c5d17a9ec8135b8d38e85d6c81697aa46068ffc66 not found: ID does not exist"
Sep 16 18:50:05.772425 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:05.772405    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-599b79d756-g5ztn"]
Sep 16 18:50:05.776869 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:05.776848    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-599b79d756-g5ztn"]
Sep 16 18:50:05.938232 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:05.938178    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-599b79d756-47w5x"]
Sep 16 18:50:05.940484 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:50:05.940459    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfec1bc7_76c3_4d11_90ea_268fa89169bc.slice/crio-095a90943e3af4fd94d504b1ae08662b7f83ceca345a188c9580895948d33229 WatchSource:0}: Error finding container 095a90943e3af4fd94d504b1ae08662b7f83ceca345a188c9580895948d33229: Status 404 returned error can't find the container with id 095a90943e3af4fd94d504b1ae08662b7f83ceca345a188c9580895948d33229
Sep 16 18:50:06.216568 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:06.216500    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abd6f47b-a02a-4790-85ce-0e0cb5da5af7" path="/var/lib/kubelet/pods/abd6f47b-a02a-4790-85ce-0e0cb5da5af7/volumes"
Sep 16 18:50:06.755333 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:06.755251    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-599b79d756-47w5x" event={"ID":"cfec1bc7-76c3-4d11-90ea-268fa89169bc","Type":"ContainerStarted","Data":"3481fe1a5a2048713b0c7eeb097ea0743bd1221afce7ee494a2fd7cb619d1003"}
Sep 16 18:50:06.755333 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:06.755317    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-599b79d756-47w5x"
Sep 16 18:50:06.755752 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:06.755341    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-599b79d756-47w5x" event={"ID":"cfec1bc7-76c3-4d11-90ea-268fa89169bc","Type":"ContainerStarted","Data":"095a90943e3af4fd94d504b1ae08662b7f83ceca345a188c9580895948d33229"}
Sep 16 18:50:06.777511 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:06.777464    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-599b79d756-47w5x" podStartSLOduration=1.251028813 podStartE2EDuration="1.77744865s" totalImagesPullingTime="526.419837ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:06.774488639 +0000 UTC m=+575.009422083" watchObservedRunningTime="2026-09-16 18:50:06.77744865 +0000 UTC m=+575.012382093"
Sep 16 18:50:37.765921 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:37.765886    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-599b79d756-47w5x"
Sep 16 18:50:39.125819 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:39.125784    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-jv92k"]
Sep 16 18:50:39.126278 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:39.126244    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abd6f47b-a02a-4790-85ce-0e0cb5da5af7" containerName="manager"
Sep 16 18:50:39.126278 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:39.126263    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="abd6f47b-a02a-4790-85ce-0e0cb5da5af7" containerName="manager"
Sep 16 18:50:39.126387 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:39.126336    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="abd6f47b-a02a-4790-85ce-0e0cb5da5af7" containerName="manager"
Sep 16 18:50:39.130526 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:39.130504    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-jv92k"
Sep 16 18:50:39.133151 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:39.133125    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 16 18:50:39.133270 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:39.133127    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-sdsk8\""
Sep 16 18:50:39.139564 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:39.139544    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-jv92k"]
Sep 16 18:50:39.291401 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:39.291370    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6lc6\" (UniqueName: \"kubernetes.io/projected/f3ffdc74-be37-40a0-9f72-be40cfe0896b-kube-api-access-j6lc6\") pod \"model-serving-api-7b954ffc95-jv92k\" (UID: \"f3ffdc74-be37-40a0-9f72-be40cfe0896b\") " pod="kserve/model-serving-api-7b954ffc95-jv92k"
Sep 16 18:50:39.291401 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:39.291403    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3ffdc74-be37-40a0-9f72-be40cfe0896b-tls-certs\") pod \"model-serving-api-7b954ffc95-jv92k\" (UID: \"f3ffdc74-be37-40a0-9f72-be40cfe0896b\") " pod="kserve/model-serving-api-7b954ffc95-jv92k"
Sep 16 18:50:39.392717 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:39.392656    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6lc6\" (UniqueName: \"kubernetes.io/projected/f3ffdc74-be37-40a0-9f72-be40cfe0896b-kube-api-access-j6lc6\") pod \"model-serving-api-7b954ffc95-jv92k\" (UID: \"f3ffdc74-be37-40a0-9f72-be40cfe0896b\") " pod="kserve/model-serving-api-7b954ffc95-jv92k"
Sep 16 18:50:39.392717 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:39.392689    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3ffdc74-be37-40a0-9f72-be40cfe0896b-tls-certs\") pod \"model-serving-api-7b954ffc95-jv92k\" (UID: \"f3ffdc74-be37-40a0-9f72-be40cfe0896b\") " pod="kserve/model-serving-api-7b954ffc95-jv92k"
Sep 16 18:50:39.394964 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:39.394937    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3ffdc74-be37-40a0-9f72-be40cfe0896b-tls-certs\") pod \"model-serving-api-7b954ffc95-jv92k\" (UID: \"f3ffdc74-be37-40a0-9f72-be40cfe0896b\") " pod="kserve/model-serving-api-7b954ffc95-jv92k"
Sep 16 18:50:39.402259 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:39.402236    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6lc6\" (UniqueName: \"kubernetes.io/projected/f3ffdc74-be37-40a0-9f72-be40cfe0896b-kube-api-access-j6lc6\") pod \"model-serving-api-7b954ffc95-jv92k\" (UID: \"f3ffdc74-be37-40a0-9f72-be40cfe0896b\") " pod="kserve/model-serving-api-7b954ffc95-jv92k"
Sep 16 18:50:39.440711 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:39.440687    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-jv92k"
Sep 16 18:50:39.712675 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:39.712652    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-jv92k"]
Sep 16 18:50:39.714596 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:50:39.714564    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3ffdc74_be37_40a0_9f72_be40cfe0896b.slice/crio-7e7f9c9c1ba52dfbaed10408f54eafa8e64bdd19b11608d0ba73e8bcd6ab8e1a WatchSource:0}: Error finding container 7e7f9c9c1ba52dfbaed10408f54eafa8e64bdd19b11608d0ba73e8bcd6ab8e1a: Status 404 returned error can't find the container with id 7e7f9c9c1ba52dfbaed10408f54eafa8e64bdd19b11608d0ba73e8bcd6ab8e1a
Sep 16 18:50:39.919008 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:39.918979    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-jv92k" event={"ID":"f3ffdc74-be37-40a0-9f72-be40cfe0896b","Type":"ContainerStarted","Data":"7e7f9c9c1ba52dfbaed10408f54eafa8e64bdd19b11608d0ba73e8bcd6ab8e1a"}
Sep 16 18:50:41.931277 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:41.931237    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-jv92k" event={"ID":"f3ffdc74-be37-40a0-9f72-be40cfe0896b","Type":"ContainerStarted","Data":"f3ad890e21858e9f658adf6721de053cf33cee59db6772cbea3f10a3fffdfa82"}
Sep 16 18:50:41.931717 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:41.931395    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-jv92k"
Sep 16 18:50:41.949696 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:41.949651    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-jv92k" podStartSLOduration=0.891298909 podStartE2EDuration="2.949637186s" totalImagesPullingTime="2.058338277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:41.948946832 +0000 UTC m=+610.183880300" watchObservedRunningTime="2026-09-16 18:50:41.949637186 +0000 UTC m=+610.184570629"
Sep 16 18:50:52.940871 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:50:52.940838    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-jv92k"
Sep 16 18:52:39.274962 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.274924    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 18:52:39.278251 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.278234    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:52:39.281082 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.281058    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 16 18:52:39.282204 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.282185    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:52:39.282419 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.282397    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 16 18:52:39.282499 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.282406    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-zw8kr\""
Sep 16 18:52:39.282499 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.282397    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-xnzpq\""
Sep 16 18:52:39.282499 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.282448    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 16 18:52:39.282499 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.282465    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 16 18:52:39.291179 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.291157    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 18:52:39.376145 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.376114    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:52:39.376334 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.376163    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j62d2\" (UniqueName: \"kubernetes.io/projected/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-kube-api-access-j62d2\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:52:39.376334 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.376204    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:52:39.376334 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.376249    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:52:39.376334 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.376274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:52:39.376334 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.376323    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:52:39.376512 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.376419    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:52:39.376512 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.376446    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:52:39.476914 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.476881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:52:39.477128 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.476937    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:52:39.477128 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.476977    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:52:39.477128 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.477022    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j62d2\" (UniqueName: \"kubernetes.io/projected/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-kube-api-access-j62d2\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:52:39.477128 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.477077    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:52:39.477128 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.477114    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:52:39.477398 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.477165    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:52:39.477398 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.477233    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:52:39.477542 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.477522    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:52:39.477633 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.477540    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:52:39.477633 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.477573    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:52:39.477791 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.477768    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:52:39.477846 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.477784    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:52:39.479457 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.479438    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:52:39.479538 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.479468    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:52:39.486587 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.486562    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j62d2\" (UniqueName: \"kubernetes.io/projected/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-kube-api-access-j62d2\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:52:39.589060 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.588972    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:52:39.891933 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.891905    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 18:52:39.894519 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:52:39.894489    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4beb9e_22dc_4b7b_929e_46c804fe02e1.slice/crio-6ff6532133491be3d43c9d76e3eb97b53cb8fc341314690003360773c17348d7 WatchSource:0}: Error finding container 6ff6532133491be3d43c9d76e3eb97b53cb8fc341314690003360773c17348d7: Status 404 returned error can't find the container with id 6ff6532133491be3d43c9d76e3eb97b53cb8fc341314690003360773c17348d7
Sep 16 18:52:39.896663 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:39.896640    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:52:40.537006 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:40.536957    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"ab4beb9e-22dc-4b7b-929e-46c804fe02e1","Type":"ContainerStarted","Data":"6ff6532133491be3d43c9d76e3eb97b53cb8fc341314690003360773c17348d7"}
Sep 16 18:52:43.556298 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:43.556271    2578 generic.go:356] "Generic (PLEG): container finished" podID="79e2c88c-45e8-4432-b974-05ea559e2be6" containerID="5d88ec55edceeaad007a9a1cbf16d86e5820eefc1ed485285e959583fc9c647d" exitCode=0
Sep 16 18:52:43.556629 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:43.556341    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7" event={"ID":"79e2c88c-45e8-4432-b974-05ea559e2be6","Type":"ContainerDied","Data":"5d88ec55edceeaad007a9a1cbf16d86e5820eefc1ed485285e959583fc9c647d"}
Sep 16 18:52:43.556801 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:43.556775    2578 scope.go:117] "RemoveContainer" containerID="5d88ec55edceeaad007a9a1cbf16d86e5820eefc1ed485285e959583fc9c647d"
Sep 16 18:52:43.557914 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:43.557880    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:52:43.558074 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:43.557957    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:52:43.558074 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:43.558004    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:52:43.558074 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:43.558060    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"ab4beb9e-22dc-4b7b-929e-46c804fe02e1","Type":"ContainerStarted","Data":"b10fccdffbdb63def8f4d25172c754f3a95a30f4c2a936e8749b53276142ba3f"}
Sep 16 18:52:44.564761 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:44.564726    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7" event={"ID":"79e2c88c-45e8-4432-b974-05ea559e2be6","Type":"ContainerStarted","Data":"fbb846374394787e67acad494b57e99ec67e73db94d46c155dec913434264ec0"}
Sep 16 18:52:44.888246 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:44.888155    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:52:44.888395 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:44.888367    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 16 18:52:44.888438 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:44.888415    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7" podUID="79e2c88c-45e8-4432-b974-05ea559e2be6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 16 18:52:45.888328 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:45.888286    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 16 18:52:45.888718 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:45.888348    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7" podUID="79e2c88c-45e8-4432-b974-05ea559e2be6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 16 18:52:46.577554 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:46.577521    2578 generic.go:356] "Generic (PLEG): container finished" podID="ab4beb9e-22dc-4b7b-929e-46c804fe02e1" containerID="b10fccdffbdb63def8f4d25172c754f3a95a30f4c2a936e8749b53276142ba3f" exitCode=0
Sep 16 18:52:46.577716 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:46.577594    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"ab4beb9e-22dc-4b7b-929e-46c804fe02e1","Type":"ContainerDied","Data":"b10fccdffbdb63def8f4d25172c754f3a95a30f4c2a936e8749b53276142ba3f"}
Sep 16 18:52:46.888719 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:46.888639    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 16 18:52:46.888719 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:46.888688    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7" podUID="79e2c88c-45e8-4432-b974-05ea559e2be6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 16 18:52:47.888792 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:47.888326    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 16 18:52:47.888792 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:47.888391    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7" podUID="79e2c88c-45e8-4432-b974-05ea559e2be6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 16 18:52:48.888843 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:48.888404    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 16 18:52:48.888843 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:48.888469    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7" podUID="79e2c88c-45e8-4432-b974-05ea559e2be6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 16 18:52:50.160901 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:50.160864    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:52:50.161387 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:50.161210    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:52:50.162235 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:52:50.162210    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7"
Sep 16 18:53:13.749470 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:53:13.749432    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"ab4beb9e-22dc-4b7b-929e-46c804fe02e1","Type":"ContainerStarted","Data":"815e05f2b07b512b2880ab2cf6082538fdd3a5bf523f425141ebfcffa4dd0b17"}
Sep 16 18:53:13.779731 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:53:13.779677    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=1.8010665449999999 podStartE2EDuration="34.779656531s" totalImagesPullingTime="30.595857608s" totalInitContainerRuntime="2.382732378s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:13.775520815 +0000 UTC m=+762.010454269" watchObservedRunningTime="2026-09-16 18:53:13.779656531 +0000 UTC m=+762.014589976"
Sep 16 18:55:06.900489 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:06.900452    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 18:55:21.921954 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:21.921878    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="ab4beb9e-22dc-4b7b-929e-46c804fe02e1" containerName="main" containerID="cri-o://815e05f2b07b512b2880ab2cf6082538fdd3a5bf523f425141ebfcffa4dd0b17" gracePeriod=45
Sep 16 18:55:22.568002 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:22.567979    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:22.683354 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:22.683333    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-model-cache\") pod \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\" (UID: \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\") "
Sep 16 18:55:22.683473 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:22.683368    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-tls-certs\") pod \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\" (UID: \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\") "
Sep 16 18:55:22.683473 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:22.683415    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-dshm\") pod \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\" (UID: \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\") "
Sep 16 18:55:22.683473 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:22.683430    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-home\") pod \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\" (UID: \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\") "
Sep 16 18:55:22.683473 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:22.683446    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j62d2\" (UniqueName: \"kubernetes.io/projected/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-kube-api-access-j62d2\") pod \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\" (UID: \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\") "
Sep 16 18:55:22.683473 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:22.683461    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-kserve-provision-location\") pod \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\" (UID: \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\") "
Sep 16 18:55:22.683727 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:22.683493    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-cabundle-cert\") pod \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\" (UID: \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\") "
Sep 16 18:55:22.683727 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:22.683584    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-tmp-dir\") pod \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\" (UID: \"ab4beb9e-22dc-4b7b-929e-46c804fe02e1\") "
Sep 16 18:55:22.683727 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:22.683603    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-model-cache" (OuterVolumeSpecName: "model-cache") pod "ab4beb9e-22dc-4b7b-929e-46c804fe02e1" (UID: "ab4beb9e-22dc-4b7b-929e-46c804fe02e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:22.683874 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:22.683858    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:55:22.683926 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:22.683893    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ab4beb9e-22dc-4b7b-929e-46c804fe02e1" (UID: "ab4beb9e-22dc-4b7b-929e-46c804fe02e1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:55:22.684157 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:22.684134    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-home" (OuterVolumeSpecName: "home") pod "ab4beb9e-22dc-4b7b-929e-46c804fe02e1" (UID: "ab4beb9e-22dc-4b7b-929e-46c804fe02e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:22.685854 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:22.685828    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-dshm" (OuterVolumeSpecName: "dshm") pod "ab4beb9e-22dc-4b7b-929e-46c804fe02e1" (UID: "ab4beb9e-22dc-4b7b-929e-46c804fe02e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:22.686014 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:22.685995    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab4beb9e-22dc-4b7b-929e-46c804fe02e1" (UID: "ab4beb9e-22dc-4b7b-929e-46c804fe02e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:55:22.686100 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:22.686015    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-kube-api-access-j62d2" (OuterVolumeSpecName: "kube-api-access-j62d2") pod "ab4beb9e-22dc-4b7b-929e-46c804fe02e1" (UID: "ab4beb9e-22dc-4b7b-929e-46c804fe02e1"). InnerVolumeSpecName "kube-api-access-j62d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:22.695659 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:22.695634    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ab4beb9e-22dc-4b7b-929e-46c804fe02e1" (UID: "ab4beb9e-22dc-4b7b-929e-46c804fe02e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:22.703108 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:22.703084    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ab4beb9e-22dc-4b7b-929e-46c804fe02e1" (UID: "ab4beb9e-22dc-4b7b-929e-46c804fe02e1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:22.784813 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:22.784790    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-cabundle-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:55:22.784813 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:22.784811    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:55:22.784948 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:22.784819    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:55:22.784948 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:22.784826    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:55:22.784948 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:22.784834    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:55:22.784948 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:22.784841    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j62d2\" (UniqueName: \"kubernetes.io/projected/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-kube-api-access-j62d2\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:55:22.784948 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:22.784848    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab4beb9e-22dc-4b7b-929e-46c804fe02e1-kserve-provision-location\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:55:23.437425 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:23.437374    2578 generic.go:356] "Generic (PLEG): container finished" podID="ab4beb9e-22dc-4b7b-929e-46c804fe02e1" containerID="815e05f2b07b512b2880ab2cf6082538fdd3a5bf523f425141ebfcffa4dd0b17" exitCode=0
Sep 16 18:55:23.437835 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:23.437436    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 18:55:23.437835 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:23.437466    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"ab4beb9e-22dc-4b7b-929e-46c804fe02e1","Type":"ContainerDied","Data":"815e05f2b07b512b2880ab2cf6082538fdd3a5bf523f425141ebfcffa4dd0b17"}
Sep 16 18:55:23.437835 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:23.437545    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"ab4beb9e-22dc-4b7b-929e-46c804fe02e1","Type":"ContainerDied","Data":"6ff6532133491be3d43c9d76e3eb97b53cb8fc341314690003360773c17348d7"}
Sep 16 18:55:23.437835 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:23.437580    2578 scope.go:117] "RemoveContainer" containerID="815e05f2b07b512b2880ab2cf6082538fdd3a5bf523f425141ebfcffa4dd0b17"
Sep 16 18:55:23.447010 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:23.446992    2578 scope.go:117] "RemoveContainer" containerID="b10fccdffbdb63def8f4d25172c754f3a95a30f4c2a936e8749b53276142ba3f"
Sep 16 18:55:23.454825 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:23.454809    2578 scope.go:117] "RemoveContainer" containerID="815e05f2b07b512b2880ab2cf6082538fdd3a5bf523f425141ebfcffa4dd0b17"
Sep 16 18:55:23.455111 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:55:23.455092    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"815e05f2b07b512b2880ab2cf6082538fdd3a5bf523f425141ebfcffa4dd0b17\": container with ID starting with 815e05f2b07b512b2880ab2cf6082538fdd3a5bf523f425141ebfcffa4dd0b17 not found: ID does not exist" containerID="815e05f2b07b512b2880ab2cf6082538fdd3a5bf523f425141ebfcffa4dd0b17"
Sep 16 18:55:23.455184 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:23.455118    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"815e05f2b07b512b2880ab2cf6082538fdd3a5bf523f425141ebfcffa4dd0b17"} err="failed to get container status \"815e05f2b07b512b2880ab2cf6082538fdd3a5bf523f425141ebfcffa4dd0b17\": rpc error: code = NotFound desc = could not find container \"815e05f2b07b512b2880ab2cf6082538fdd3a5bf523f425141ebfcffa4dd0b17\": container with ID starting with 815e05f2b07b512b2880ab2cf6082538fdd3a5bf523f425141ebfcffa4dd0b17 not found: ID does not exist"
Sep 16 18:55:23.455184 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:23.455136    2578 scope.go:117] "RemoveContainer" containerID="b10fccdffbdb63def8f4d25172c754f3a95a30f4c2a936e8749b53276142ba3f"
Sep 16 18:55:23.455398 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:55:23.455383    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b10fccdffbdb63def8f4d25172c754f3a95a30f4c2a936e8749b53276142ba3f\": container with ID starting with b10fccdffbdb63def8f4d25172c754f3a95a30f4c2a936e8749b53276142ba3f not found: ID does not exist" containerID="b10fccdffbdb63def8f4d25172c754f3a95a30f4c2a936e8749b53276142ba3f"
Sep 16 18:55:23.455440 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:23.455401    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b10fccdffbdb63def8f4d25172c754f3a95a30f4c2a936e8749b53276142ba3f"} err="failed to get container status \"b10fccdffbdb63def8f4d25172c754f3a95a30f4c2a936e8749b53276142ba3f\": rpc error: code = NotFound desc = could not find container \"b10fccdffbdb63def8f4d25172c754f3a95a30f4c2a936e8749b53276142ba3f\": container with ID starting with b10fccdffbdb63def8f4d25172c754f3a95a30f4c2a936e8749b53276142ba3f not found: ID does not exist"
Sep 16 18:55:23.463029 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:23.463007    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 18:55:23.470047 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:23.470014    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 18:55:24.218137 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:24.218107    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab4beb9e-22dc-4b7b-929e-46c804fe02e1" path="/var/lib/kubelet/pods/ab4beb9e-22dc-4b7b-929e-46c804fe02e1/volumes"
Sep 16 18:55:46.880359 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:46.880326    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"]
Sep 16 18:55:46.880813 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:46.880724    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab4beb9e-22dc-4b7b-929e-46c804fe02e1" containerName="main"
Sep 16 18:55:46.880813 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:46.880744    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab4beb9e-22dc-4b7b-929e-46c804fe02e1" containerName="main"
Sep 16 18:55:46.880813 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:46.880757    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab4beb9e-22dc-4b7b-929e-46c804fe02e1" containerName="storage-initializer"
Sep 16 18:55:46.880813 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:46.880763    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab4beb9e-22dc-4b7b-929e-46c804fe02e1" containerName="storage-initializer"
Sep 16 18:55:46.881035 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:46.880841    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab4beb9e-22dc-4b7b-929e-46c804fe02e1" containerName="main"
Sep 16 18:55:46.883358 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:46.883331    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:55:46.886610 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:46.886587    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 16 18:55:46.886726 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:46.886652    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 16 18:55:46.886726 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:46.886671    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-pc29c\""
Sep 16 18:55:46.886826 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:46.886780    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Sep 16 18:55:46.897940 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:46.897914    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"]
Sep 16 18:55:46.953670 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:46.953640    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bc39931-995e-46b0-821c-49e4d0622109-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-7z24n\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:55:46.953763 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:46.953675    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdcv8\" (UniqueName: \"kubernetes.io/projected/5bc39931-995e-46b0-821c-49e4d0622109-kube-api-access-pdcv8\") pod \"router-gateway-1-openshift-default-6b46477c5b-7z24n\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:55:46.953763 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:46.953712    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc39931-995e-46b0-821c-49e4d0622109-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-7z24n\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:55:46.953763 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:46.953737    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bc39931-995e-46b0-821c-49e4d0622109-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-7z24n\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:55:46.953763 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:46.953754    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc39931-995e-46b0-821c-49e4d0622109-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-7z24n\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:55:46.953895 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:46.953783    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bc39931-995e-46b0-821c-49e4d0622109-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-7z24n\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:55:46.953895 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:46.953804    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bc39931-995e-46b0-821c-49e4d0622109-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-7z24n\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:55:46.953895 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:46.953824    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bc39931-995e-46b0-821c-49e4d0622109-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-7z24n\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:55:46.953895 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:46.953838    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bc39931-995e-46b0-821c-49e4d0622109-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-7z24n\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:55:47.054235 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:47.054205    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc39931-995e-46b0-821c-49e4d0622109-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-7z24n\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:55:47.054375 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:47.054239    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bc39931-995e-46b0-821c-49e4d0622109-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-7z24n\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:55:47.054375 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:47.054257    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc39931-995e-46b0-821c-49e4d0622109-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-7z24n\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:55:47.054375 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:47.054281    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bc39931-995e-46b0-821c-49e4d0622109-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-7z24n\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:55:47.054375 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:47.054299    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bc39931-995e-46b0-821c-49e4d0622109-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-7z24n\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:55:47.054375 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:47.054324    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bc39931-995e-46b0-821c-49e4d0622109-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-7z24n\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:55:47.054375 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:47.054347    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bc39931-995e-46b0-821c-49e4d0622109-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-7z24n\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:55:47.054667 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:47.054383    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bc39931-995e-46b0-821c-49e4d0622109-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-7z24n\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:55:47.054667 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:47.054417    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdcv8\" (UniqueName: \"kubernetes.io/projected/5bc39931-995e-46b0-821c-49e4d0622109-kube-api-access-pdcv8\") pod \"router-gateway-1-openshift-default-6b46477c5b-7z24n\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:55:47.054667 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:47.054628    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc39931-995e-46b0-821c-49e4d0622109-credential-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-7z24n\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:55:47.054805 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:47.054680    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bc39931-995e-46b0-821c-49e4d0622109-istio-data\") pod \"router-gateway-1-openshift-default-6b46477c5b-7z24n\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:55:47.054805 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:47.054724    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc39931-995e-46b0-821c-49e4d0622109-workload-socket\") pod \"router-gateway-1-openshift-default-6b46477c5b-7z24n\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:55:47.054869 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:47.054833    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bc39931-995e-46b0-821c-49e4d0622109-workload-certs\") pod \"router-gateway-1-openshift-default-6b46477c5b-7z24n\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:55:47.055275 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:47.055256    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bc39931-995e-46b0-821c-49e4d0622109-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6b46477c5b-7z24n\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:55:47.056542 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:47.056519    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bc39931-995e-46b0-821c-49e4d0622109-istio-envoy\") pod \"router-gateway-1-openshift-default-6b46477c5b-7z24n\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:55:47.056634 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:47.056573    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bc39931-995e-46b0-821c-49e4d0622109-istio-podinfo\") pod \"router-gateway-1-openshift-default-6b46477c5b-7z24n\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:55:47.064568 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:47.064549    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdcv8\" (UniqueName: \"kubernetes.io/projected/5bc39931-995e-46b0-821c-49e4d0622109-kube-api-access-pdcv8\") pod \"router-gateway-1-openshift-default-6b46477c5b-7z24n\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:55:47.064651 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:47.064618    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bc39931-995e-46b0-821c-49e4d0622109-istio-token\") pod \"router-gateway-1-openshift-default-6b46477c5b-7z24n\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:55:47.194183 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:47.194160    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:55:47.470459 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:47.470433    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"]
Sep 16 18:55:47.471903 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:55:47.471878    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bc39931_995e_46b0_821c_49e4d0622109.slice/crio-0a2ad4742752d27d93fb2868a8d557d6f463b9191cfaa6a21324eb8a8916da3b WatchSource:0}: Error finding container 0a2ad4742752d27d93fb2868a8d557d6f463b9191cfaa6a21324eb8a8916da3b: Status 404 returned error can't find the container with id 0a2ad4742752d27d93fb2868a8d557d6f463b9191cfaa6a21324eb8a8916da3b
Sep 16 18:55:47.473811 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:47.473779    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:55:47.473896 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:47.473854    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:55:47.473957 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:47.473896    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:55:47.561589 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:47.561545    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n" event={"ID":"5bc39931-995e-46b0-821c-49e4d0622109","Type":"ContainerStarted","Data":"e4e1a9939cc6c912caf08b4ceb01a65395ce38e8a29436f3acd23e9e0b4f97d0"}
Sep 16 18:55:47.561735 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:47.561606    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n" event={"ID":"5bc39931-995e-46b0-821c-49e4d0622109","Type":"ContainerStarted","Data":"0a2ad4742752d27d93fb2868a8d557d6f463b9191cfaa6a21324eb8a8916da3b"}
Sep 16 18:55:47.585104 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:47.585031    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n" podStartSLOduration=1.5850139269999999 podStartE2EDuration="1.585013927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:47.583621769 +0000 UTC m=+915.818555212" watchObservedRunningTime="2026-09-16 18:55:47.585013927 +0000 UTC m=+915.819947376"
Sep 16 18:55:48.194326 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:48.194297    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:55:48.198553 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:48.198529    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:55:48.566413 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:48.566330    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:55:48.567298 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:55:48.567272    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:56:17.480129 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.480096    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"]
Sep 16 18:56:17.482365 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.482345    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:17.485021 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.484940    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Sep 16 18:56:17.485021 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.484943    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 16 18:56:17.485021 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.484993    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 16 18:56:17.485262 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.485057    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-lsjvc\""
Sep 16 18:56:17.497894 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.497874    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"]
Sep 16 18:56:17.578641 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.578606    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb59514-7a60-4153-ac2d-2c757ee85ddd-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j5lvw\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:17.578641 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.578646    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb59514-7a60-4153-ac2d-2c757ee85ddd-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j5lvw\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:17.579061 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.578722    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb59514-7a60-4153-ac2d-2c757ee85ddd-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j5lvw\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:17.579061 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.578757    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swtq4\" (UniqueName: \"kubernetes.io/projected/afb59514-7a60-4153-ac2d-2c757ee85ddd-kube-api-access-swtq4\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j5lvw\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:17.579061 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.578781    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb59514-7a60-4153-ac2d-2c757ee85ddd-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j5lvw\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:17.579061 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.578820    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb59514-7a60-4153-ac2d-2c757ee85ddd-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j5lvw\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:17.579061 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.578844    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb59514-7a60-4153-ac2d-2c757ee85ddd-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j5lvw\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:17.579061 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.578867    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb59514-7a60-4153-ac2d-2c757ee85ddd-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j5lvw\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:17.579061 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.578905    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb59514-7a60-4153-ac2d-2c757ee85ddd-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j5lvw\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:17.679532 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.679500    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb59514-7a60-4153-ac2d-2c757ee85ddd-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j5lvw\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:17.679675 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.679536    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb59514-7a60-4153-ac2d-2c757ee85ddd-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j5lvw\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:17.679675 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.679591    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb59514-7a60-4153-ac2d-2c757ee85ddd-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j5lvw\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:17.679675 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.679638    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb59514-7a60-4153-ac2d-2c757ee85ddd-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j5lvw\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:17.679675 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.679661    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb59514-7a60-4153-ac2d-2c757ee85ddd-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j5lvw\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:17.679890 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.679681    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb59514-7a60-4153-ac2d-2c757ee85ddd-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j5lvw\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:17.679890 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.679728    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb59514-7a60-4153-ac2d-2c757ee85ddd-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j5lvw\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:17.679890 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.679751    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swtq4\" (UniqueName: \"kubernetes.io/projected/afb59514-7a60-4153-ac2d-2c757ee85ddd-kube-api-access-swtq4\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j5lvw\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:17.679890 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.679779    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb59514-7a60-4153-ac2d-2c757ee85ddd-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j5lvw\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:17.680269 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.680067    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb59514-7a60-4153-ac2d-2c757ee85ddd-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j5lvw\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:17.680269 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.680134    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb59514-7a60-4153-ac2d-2c757ee85ddd-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j5lvw\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:17.680269 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.680164    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb59514-7a60-4153-ac2d-2c757ee85ddd-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j5lvw\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:17.680427 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.680357    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb59514-7a60-4153-ac2d-2c757ee85ddd-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j5lvw\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:17.680475 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.680431    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb59514-7a60-4153-ac2d-2c757ee85ddd-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j5lvw\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:17.681968 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.681947    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb59514-7a60-4153-ac2d-2c757ee85ddd-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j5lvw\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:17.682252 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.682236    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb59514-7a60-4153-ac2d-2c757ee85ddd-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j5lvw\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:17.695001 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.694969    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb59514-7a60-4153-ac2d-2c757ee85ddd-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j5lvw\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:17.698055 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.698023    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swtq4\" (UniqueName: \"kubernetes.io/projected/afb59514-7a60-4153-ac2d-2c757ee85ddd-kube-api-access-swtq4\") pod \"router-gateway-1-openshift-default-7dcbd489bd-j5lvw\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:17.793191 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:17.793121    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:18.106511 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:18.106439    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"]
Sep 16 18:56:18.109431 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:56:18.109399    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafb59514_7a60_4153_ac2d_2c757ee85ddd.slice/crio-f66d17820231c8e0b3f6be89191b64d28b549beae151b0e1d273da0b02c81a52 WatchSource:0}: Error finding container f66d17820231c8e0b3f6be89191b64d28b549beae151b0e1d273da0b02c81a52: Status 404 returned error can't find the container with id f66d17820231c8e0b3f6be89191b64d28b549beae151b0e1d273da0b02c81a52
Sep 16 18:56:18.111442 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:18.111414    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:56:18.111509 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:18.111477    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:56:18.111509 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:18.111505    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:56:18.732560 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:18.732523    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw" event={"ID":"afb59514-7a60-4153-ac2d-2c757ee85ddd","Type":"ContainerStarted","Data":"0444fc9a6c70bf797413b0333b41402b0990b23553f33eba91f49481af545520"}
Sep 16 18:56:18.733238 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:18.733196    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw" event={"ID":"afb59514-7a60-4153-ac2d-2c757ee85ddd","Type":"ContainerStarted","Data":"f66d17820231c8e0b3f6be89191b64d28b549beae151b0e1d273da0b02c81a52"}
Sep 16 18:56:18.760485 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:18.760441    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw" podStartSLOduration=1.760428573 podStartE2EDuration="1.760428573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:18.758238232 +0000 UTC m=+946.993171685" watchObservedRunningTime="2026-09-16 18:56:18.760428573 +0000 UTC m=+946.995362015"
Sep 16 18:56:18.793792 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:18.793770    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:19.798487 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:19.798456    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:20.748199 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:20.748170    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:20.749162 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:20.749139    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:22.163507 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:22.163470    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"]
Sep 16 18:56:22.163943 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:22.163706    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n" podUID="5bc39931-995e-46b0-821c-49e4d0622109" containerName="istio-proxy" containerID="cri-o://e4e1a9939cc6c912caf08b4ceb01a65395ce38e8a29436f3acd23e9e0b4f97d0" gracePeriod=30
Sep 16 18:56:24.595690 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.595667    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:56:24.639405 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.639377    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bc39931-995e-46b0-821c-49e4d0622109-istio-data\") pod \"5bc39931-995e-46b0-821c-49e4d0622109\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") "
Sep 16 18:56:24.639560 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.639415    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc39931-995e-46b0-821c-49e4d0622109-credential-socket\") pod \"5bc39931-995e-46b0-821c-49e4d0622109\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") "
Sep 16 18:56:24.639560 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.639487    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdcv8\" (UniqueName: \"kubernetes.io/projected/5bc39931-995e-46b0-821c-49e4d0622109-kube-api-access-pdcv8\") pod \"5bc39931-995e-46b0-821c-49e4d0622109\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") "
Sep 16 18:56:24.639560 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.639536    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bc39931-995e-46b0-821c-49e4d0622109-istio-podinfo\") pod \"5bc39931-995e-46b0-821c-49e4d0622109\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") "
Sep 16 18:56:24.639734 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.639573    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bc39931-995e-46b0-821c-49e4d0622109-workload-certs\") pod \"5bc39931-995e-46b0-821c-49e4d0622109\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") "
Sep 16 18:56:24.639734 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.639623    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bc39931-995e-46b0-821c-49e4d0622109-istio-token\") pod \"5bc39931-995e-46b0-821c-49e4d0622109\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") "
Sep 16 18:56:24.639734 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.639674    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc39931-995e-46b0-821c-49e4d0622109-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5bc39931-995e-46b0-821c-49e4d0622109" (UID: "5bc39931-995e-46b0-821c-49e4d0622109"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:24.639734 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.639686    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc39931-995e-46b0-821c-49e4d0622109-istio-data" (OuterVolumeSpecName: "istio-data") pod "5bc39931-995e-46b0-821c-49e4d0622109" (UID: "5bc39931-995e-46b0-821c-49e4d0622109"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:24.639734 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.639678    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bc39931-995e-46b0-821c-49e4d0622109-istiod-ca-cert\") pod \"5bc39931-995e-46b0-821c-49e4d0622109\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") "
Sep 16 18:56:24.639986 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.639749    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bc39931-995e-46b0-821c-49e4d0622109-istio-envoy\") pod \"5bc39931-995e-46b0-821c-49e4d0622109\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") "
Sep 16 18:56:24.639986 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.639796    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc39931-995e-46b0-821c-49e4d0622109-workload-socket\") pod \"5bc39931-995e-46b0-821c-49e4d0622109\" (UID: \"5bc39931-995e-46b0-821c-49e4d0622109\") "
Sep 16 18:56:24.639986 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.639882    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc39931-995e-46b0-821c-49e4d0622109-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5bc39931-995e-46b0-821c-49e4d0622109" (UID: "5bc39931-995e-46b0-821c-49e4d0622109"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:24.640163 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.640090    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc39931-995e-46b0-821c-49e4d0622109-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5bc39931-995e-46b0-821c-49e4d0622109" (UID: "5bc39931-995e-46b0-821c-49e4d0622109"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:24.640163 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.640112    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bc39931-995e-46b0-821c-49e4d0622109-istio-data\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:56:24.640163 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.640117    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bc39931-995e-46b0-821c-49e4d0622109-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5bc39931-995e-46b0-821c-49e4d0622109" (UID: "5bc39931-995e-46b0-821c-49e4d0622109"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:56:24.640163 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.640129    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc39931-995e-46b0-821c-49e4d0622109-credential-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:56:24.640361 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.640164    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bc39931-995e-46b0-821c-49e4d0622109-workload-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:56:24.642295 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.642248    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bc39931-995e-46b0-821c-49e4d0622109-istio-token" (OuterVolumeSpecName: "istio-token") pod "5bc39931-995e-46b0-821c-49e4d0622109" (UID: "5bc39931-995e-46b0-821c-49e4d0622109"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:56:24.642295 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.642267    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bc39931-995e-46b0-821c-49e4d0622109-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5bc39931-995e-46b0-821c-49e4d0622109" (UID: "5bc39931-995e-46b0-821c-49e4d0622109"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:24.642443 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.642352    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bc39931-995e-46b0-821c-49e4d0622109-kube-api-access-pdcv8" (OuterVolumeSpecName: "kube-api-access-pdcv8") pod "5bc39931-995e-46b0-821c-49e4d0622109" (UID: "5bc39931-995e-46b0-821c-49e4d0622109"). InnerVolumeSpecName "kube-api-access-pdcv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:56:24.642443 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.642348    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bc39931-995e-46b0-821c-49e4d0622109-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5bc39931-995e-46b0-821c-49e4d0622109" (UID: "5bc39931-995e-46b0-821c-49e4d0622109"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:56:24.740732 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.740706    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bc39931-995e-46b0-821c-49e4d0622109-istio-envoy\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:56:24.740732 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.740730    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bc39931-995e-46b0-821c-49e4d0622109-workload-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:56:24.740866 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.740740    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdcv8\" (UniqueName: \"kubernetes.io/projected/5bc39931-995e-46b0-821c-49e4d0622109-kube-api-access-pdcv8\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:56:24.740866 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.740750    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bc39931-995e-46b0-821c-49e4d0622109-istio-podinfo\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:56:24.740866 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.740757    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bc39931-995e-46b0-821c-49e4d0622109-istio-token\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:56:24.740866 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.740765    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bc39931-995e-46b0-821c-49e4d0622109-istiod-ca-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:56:24.768674 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.768649    2578 generic.go:356] "Generic (PLEG): container finished" podID="5bc39931-995e-46b0-821c-49e4d0622109" containerID="e4e1a9939cc6c912caf08b4ceb01a65395ce38e8a29436f3acd23e9e0b4f97d0" exitCode=0
Sep 16 18:56:24.768771 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.768691    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"
Sep 16 18:56:24.768771 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.768698    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n" event={"ID":"5bc39931-995e-46b0-821c-49e4d0622109","Type":"ContainerDied","Data":"e4e1a9939cc6c912caf08b4ceb01a65395ce38e8a29436f3acd23e9e0b4f97d0"}
Sep 16 18:56:24.768771 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.768739    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n" event={"ID":"5bc39931-995e-46b0-821c-49e4d0622109","Type":"ContainerDied","Data":"0a2ad4742752d27d93fb2868a8d557d6f463b9191cfaa6a21324eb8a8916da3b"}
Sep 16 18:56:24.768771 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.768759    2578 scope.go:117] "RemoveContainer" containerID="e4e1a9939cc6c912caf08b4ceb01a65395ce38e8a29436f3acd23e9e0b4f97d0"
Sep 16 18:56:24.777512 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.777497    2578 scope.go:117] "RemoveContainer" containerID="e4e1a9939cc6c912caf08b4ceb01a65395ce38e8a29436f3acd23e9e0b4f97d0"
Sep 16 18:56:24.777756 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:56:24.777739    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4e1a9939cc6c912caf08b4ceb01a65395ce38e8a29436f3acd23e9e0b4f97d0\": container with ID starting with e4e1a9939cc6c912caf08b4ceb01a65395ce38e8a29436f3acd23e9e0b4f97d0 not found: ID does not exist" containerID="e4e1a9939cc6c912caf08b4ceb01a65395ce38e8a29436f3acd23e9e0b4f97d0"
Sep 16 18:56:24.777796 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.777763    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4e1a9939cc6c912caf08b4ceb01a65395ce38e8a29436f3acd23e9e0b4f97d0"} err="failed to get container status \"e4e1a9939cc6c912caf08b4ceb01a65395ce38e8a29436f3acd23e9e0b4f97d0\": rpc error: code = NotFound desc = could not find container \"e4e1a9939cc6c912caf08b4ceb01a65395ce38e8a29436f3acd23e9e0b4f97d0\": container with ID starting with e4e1a9939cc6c912caf08b4ceb01a65395ce38e8a29436f3acd23e9e0b4f97d0 not found: ID does not exist"
Sep 16 18:56:24.791483 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.791463    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"]
Sep 16 18:56:24.796586 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:24.796566    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6b46477c5b-7z24n"]
Sep 16 18:56:26.217290 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:26.217255    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bc39931-995e-46b0-821c-49e4d0622109" path="/var/lib/kubelet/pods/5bc39931-995e-46b0-821c-49e4d0622109/volumes"
Sep 16 18:56:49.753597 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:49.753562    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"]
Sep 16 18:56:49.754004 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:49.753783    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw" podUID="afb59514-7a60-4153-ac2d-2c757ee85ddd" containerName="istio-proxy" containerID="cri-o://0444fc9a6c70bf797413b0333b41402b0990b23553f33eba91f49481af545520" gracePeriod=30
Sep 16 18:56:51.790166 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:51.790132    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw" podUID="afb59514-7a60-4153-ac2d-2c757ee85ddd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 18:56:52.499326 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.499302    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:52.552901 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.552815    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb59514-7a60-4153-ac2d-2c757ee85ddd-workload-certs\") pod \"afb59514-7a60-4153-ac2d-2c757ee85ddd\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") "
Sep 16 18:56:52.552901 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.552865    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb59514-7a60-4153-ac2d-2c757ee85ddd-istio-data\") pod \"afb59514-7a60-4153-ac2d-2c757ee85ddd\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") "
Sep 16 18:56:52.553150 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.552901    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swtq4\" (UniqueName: \"kubernetes.io/projected/afb59514-7a60-4153-ac2d-2c757ee85ddd-kube-api-access-swtq4\") pod \"afb59514-7a60-4153-ac2d-2c757ee85ddd\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") "
Sep 16 18:56:52.553150 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.552933    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb59514-7a60-4153-ac2d-2c757ee85ddd-credential-socket\") pod \"afb59514-7a60-4153-ac2d-2c757ee85ddd\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") "
Sep 16 18:56:52.553150 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.552977    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb59514-7a60-4153-ac2d-2c757ee85ddd-istio-token\") pod \"afb59514-7a60-4153-ac2d-2c757ee85ddd\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") "
Sep 16 18:56:52.553150 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.553019    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb59514-7a60-4153-ac2d-2c757ee85ddd-istio-envoy\") pod \"afb59514-7a60-4153-ac2d-2c757ee85ddd\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") "
Sep 16 18:56:52.553150 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.553075    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb59514-7a60-4153-ac2d-2c757ee85ddd-istiod-ca-cert\") pod \"afb59514-7a60-4153-ac2d-2c757ee85ddd\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") "
Sep 16 18:56:52.553150 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.553102    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb59514-7a60-4153-ac2d-2c757ee85ddd-workload-socket\") pod \"afb59514-7a60-4153-ac2d-2c757ee85ddd\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") "
Sep 16 18:56:52.553482 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.553150    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb59514-7a60-4153-ac2d-2c757ee85ddd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "afb59514-7a60-4153-ac2d-2c757ee85ddd" (UID: "afb59514-7a60-4153-ac2d-2c757ee85ddd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:52.553482 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.553171    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb59514-7a60-4153-ac2d-2c757ee85ddd-istio-podinfo\") pod \"afb59514-7a60-4153-ac2d-2c757ee85ddd\" (UID: \"afb59514-7a60-4153-ac2d-2c757ee85ddd\") "
Sep 16 18:56:52.553482 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.553200    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb59514-7a60-4153-ac2d-2c757ee85ddd-istio-data" (OuterVolumeSpecName: "istio-data") pod "afb59514-7a60-4153-ac2d-2c757ee85ddd" (UID: "afb59514-7a60-4153-ac2d-2c757ee85ddd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:52.553482 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.553372    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afb59514-7a60-4153-ac2d-2c757ee85ddd-workload-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:56:52.553482 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.553389    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afb59514-7a60-4153-ac2d-2c757ee85ddd-istio-data\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:56:52.553482 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.553393    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb59514-7a60-4153-ac2d-2c757ee85ddd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "afb59514-7a60-4153-ac2d-2c757ee85ddd" (UID: "afb59514-7a60-4153-ac2d-2c757ee85ddd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:52.553794 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.553602    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb59514-7a60-4153-ac2d-2c757ee85ddd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "afb59514-7a60-4153-ac2d-2c757ee85ddd" (UID: "afb59514-7a60-4153-ac2d-2c757ee85ddd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:52.553794 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.553627    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afb59514-7a60-4153-ac2d-2c757ee85ddd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "afb59514-7a60-4153-ac2d-2c757ee85ddd" (UID: "afb59514-7a60-4153-ac2d-2c757ee85ddd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:56:52.555956 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.555823    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb59514-7a60-4153-ac2d-2c757ee85ddd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "afb59514-7a60-4153-ac2d-2c757ee85ddd" (UID: "afb59514-7a60-4153-ac2d-2c757ee85ddd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:52.555956 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.555841    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afb59514-7a60-4153-ac2d-2c757ee85ddd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "afb59514-7a60-4153-ac2d-2c757ee85ddd" (UID: "afb59514-7a60-4153-ac2d-2c757ee85ddd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 18:56:52.555956 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.555928    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb59514-7a60-4153-ac2d-2c757ee85ddd-kube-api-access-swtq4" (OuterVolumeSpecName: "kube-api-access-swtq4") pod "afb59514-7a60-4153-ac2d-2c757ee85ddd" (UID: "afb59514-7a60-4153-ac2d-2c757ee85ddd"). InnerVolumeSpecName "kube-api-access-swtq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:56:52.555956 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.555938    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb59514-7a60-4153-ac2d-2c757ee85ddd-istio-token" (OuterVolumeSpecName: "istio-token") pod "afb59514-7a60-4153-ac2d-2c757ee85ddd" (UID: "afb59514-7a60-4153-ac2d-2c757ee85ddd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:56:52.654045 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.654012    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swtq4\" (UniqueName: \"kubernetes.io/projected/afb59514-7a60-4153-ac2d-2c757ee85ddd-kube-api-access-swtq4\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:56:52.654155 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.654062    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb59514-7a60-4153-ac2d-2c757ee85ddd-credential-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:56:52.654155 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.654073    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afb59514-7a60-4153-ac2d-2c757ee85ddd-istio-token\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:56:52.654155 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.654082    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afb59514-7a60-4153-ac2d-2c757ee85ddd-istio-envoy\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:56:52.654155 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.654089    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afb59514-7a60-4153-ac2d-2c757ee85ddd-istiod-ca-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:56:52.654155 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.654096    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afb59514-7a60-4153-ac2d-2c757ee85ddd-workload-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:56:52.654155 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.654103    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afb59514-7a60-4153-ac2d-2c757ee85ddd-istio-podinfo\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:56:52.922647 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.922575    2578 generic.go:356] "Generic (PLEG): container finished" podID="afb59514-7a60-4153-ac2d-2c757ee85ddd" containerID="0444fc9a6c70bf797413b0333b41402b0990b23553f33eba91f49481af545520" exitCode=0
Sep 16 18:56:52.922647 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.922636    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"
Sep 16 18:56:52.923102 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.922650    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw" event={"ID":"afb59514-7a60-4153-ac2d-2c757ee85ddd","Type":"ContainerDied","Data":"0444fc9a6c70bf797413b0333b41402b0990b23553f33eba91f49481af545520"}
Sep 16 18:56:52.923102 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.922737    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw" event={"ID":"afb59514-7a60-4153-ac2d-2c757ee85ddd","Type":"ContainerDied","Data":"f66d17820231c8e0b3f6be89191b64d28b549beae151b0e1d273da0b02c81a52"}
Sep 16 18:56:52.923102 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.922762    2578 scope.go:117] "RemoveContainer" containerID="0444fc9a6c70bf797413b0333b41402b0990b23553f33eba91f49481af545520"
Sep 16 18:56:52.932758 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.932738    2578 scope.go:117] "RemoveContainer" containerID="0444fc9a6c70bf797413b0333b41402b0990b23553f33eba91f49481af545520"
Sep 16 18:56:52.933062 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:56:52.933025    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0444fc9a6c70bf797413b0333b41402b0990b23553f33eba91f49481af545520\": container with ID starting with 0444fc9a6c70bf797413b0333b41402b0990b23553f33eba91f49481af545520 not found: ID does not exist" containerID="0444fc9a6c70bf797413b0333b41402b0990b23553f33eba91f49481af545520"
Sep 16 18:56:52.933138 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.933072    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0444fc9a6c70bf797413b0333b41402b0990b23553f33eba91f49481af545520"} err="failed to get container status \"0444fc9a6c70bf797413b0333b41402b0990b23553f33eba91f49481af545520\": rpc error: code = NotFound desc = could not find container \"0444fc9a6c70bf797413b0333b41402b0990b23553f33eba91f49481af545520\": container with ID starting with 0444fc9a6c70bf797413b0333b41402b0990b23553f33eba91f49481af545520 not found: ID does not exist"
Sep 16 18:56:52.947295 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.947270    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"]
Sep 16 18:56:52.952462 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:52.952439    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-j5lvw"]
Sep 16 18:56:54.217273 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:56:54.217237    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afb59514-7a60-4153-ac2d-2c757ee85ddd" path="/var/lib/kubelet/pods/afb59514-7a60-4153-ac2d-2c757ee85ddd/volumes"
Sep 16 18:57:00.641947 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:57:00.641910    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx"]
Sep 16 18:57:00.642341 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:57:00.642294    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afb59514-7a60-4153-ac2d-2c757ee85ddd" containerName="istio-proxy"
Sep 16 18:57:00.642341 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:57:00.642307    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="afb59514-7a60-4153-ac2d-2c757ee85ddd" containerName="istio-proxy"
Sep 16 18:57:00.642341 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:57:00.642322    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bc39931-995e-46b0-821c-49e4d0622109" containerName="istio-proxy"
Sep 16 18:57:00.642341 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:57:00.642328    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bc39931-995e-46b0-821c-49e4d0622109" containerName="istio-proxy"
Sep 16 18:57:00.642499 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:57:00.642390    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bc39931-995e-46b0-821c-49e4d0622109" containerName="istio-proxy"
Sep 16 18:57:00.642499 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:57:00.642397    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="afb59514-7a60-4153-ac2d-2c757ee85ddd" containerName="istio-proxy"
Sep 16 18:57:00.644228 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:57:00.644212    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx"
Sep 16 18:57:00.647953 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:57:00.647927    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-s8d7x\""
Sep 16 18:57:00.648141 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:57:00.647927    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-f4rfs\""
Sep 16 18:57:00.648141 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:57:00.647994    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 16 18:57:00.648141 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:57:00.648068    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 16 18:57:00.648319 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:57:00.648301    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 16 18:57:00.656626 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:57:00.656605    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx"]
Sep 16 18:57:00.716622 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:57:00.716595    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv9vl\" (UniqueName: \"kubernetes.io/projected/1eb8f8d7-eb54-409e-8c60-17bd9473f1df-kube-api-access-fv9vl\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx\" (UID: \"1eb8f8d7-eb54-409e-8c60-17bd9473f1df\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx"
Sep 16 18:57:00.716736 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:57:00.716635    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eb8f8d7-eb54-409e-8c60-17bd9473f1df-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx\" (UID: \"1eb8f8d7-eb54-409e-8c60-17bd9473f1df\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx"
Sep 16 18:57:00.817356 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:57:00.817327    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv9vl\" (UniqueName: \"kubernetes.io/projected/1eb8f8d7-eb54-409e-8c60-17bd9473f1df-kube-api-access-fv9vl\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx\" (UID: \"1eb8f8d7-eb54-409e-8c60-17bd9473f1df\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx"
Sep 16 18:57:00.817477 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:57:00.817367    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eb8f8d7-eb54-409e-8c60-17bd9473f1df-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx\" (UID: \"1eb8f8d7-eb54-409e-8c60-17bd9473f1df\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx"
Sep 16 18:57:00.819854 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:57:00.819829    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eb8f8d7-eb54-409e-8c60-17bd9473f1df-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx\" (UID: \"1eb8f8d7-eb54-409e-8c60-17bd9473f1df\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx"
Sep 16 18:57:00.830591 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:57:00.830565    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv9vl\" (UniqueName: \"kubernetes.io/projected/1eb8f8d7-eb54-409e-8c60-17bd9473f1df-kube-api-access-fv9vl\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx\" (UID: \"1eb8f8d7-eb54-409e-8c60-17bd9473f1df\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx"
Sep 16 18:57:00.954198 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:57:00.954175    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx"
Sep 16 18:57:01.230214 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:57:01.230180    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx"]
Sep 16 18:57:01.234851 ip-10-0-142-80 kubenswrapper[2578]: W0916 18:57:01.234829    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb8f8d7_eb54_409e_8c60_17bd9473f1df.slice/crio-0e012353894067ade514efe26dad479bc40a7f5b79666664b21f86f8d040e3fb WatchSource:0}: Error finding container 0e012353894067ade514efe26dad479bc40a7f5b79666664b21f86f8d040e3fb: Status 404 returned error can't find the container with id 0e012353894067ade514efe26dad479bc40a7f5b79666664b21f86f8d040e3fb
Sep 16 18:57:01.970361 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:57:01.970321    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx" event={"ID":"1eb8f8d7-eb54-409e-8c60-17bd9473f1df","Type":"ContainerStarted","Data":"0e012353894067ade514efe26dad479bc40a7f5b79666664b21f86f8d040e3fb"}
Sep 16 18:57:03.983570 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:57:03.983538    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx" event={"ID":"1eb8f8d7-eb54-409e-8c60-17bd9473f1df","Type":"ContainerStarted","Data":"3f324d51c697d68d8dffc69b3514059532ef81f60f9403460a44960414d78f63"}
Sep 16 18:57:03.984004 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:57:03.983591    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx"
Sep 16 18:57:04.003225 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:57:04.003185    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx" podStartSLOduration=1.800169661 podStartE2EDuration="4.00316943s" totalImagesPullingTime="2.202999769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:57:04.001808057 +0000 UTC m=+992.236741501" watchObservedRunningTime="2026-09-16 18:57:04.00316943 +0000 UTC m=+992.238102872"
Sep 16 18:57:34.991551 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:57:34.991511    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx"
Sep 16 18:58:55.991167 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:58:55.991137    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx"]
Sep 16 18:59:04.992116 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:04.992070    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx" podUID="1eb8f8d7-eb54-409e-8c60-17bd9473f1df" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:59:11.012723 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:11.012657    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx" podUID="1eb8f8d7-eb54-409e-8c60-17bd9473f1df" containerName="main" containerID="cri-o://3f324d51c697d68d8dffc69b3514059532ef81f60f9403460a44960414d78f63" gracePeriod=45
Sep 16 18:59:14.991630 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:14.991587    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx" podUID="1eb8f8d7-eb54-409e-8c60-17bd9473f1df" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:59:24.991743 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:24.991697    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx" podUID="1eb8f8d7-eb54-409e-8c60-17bd9473f1df" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:59:24.992166 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:24.991820    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx"
Sep 16 18:59:34.991625 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:34.991573    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx" podUID="1eb8f8d7-eb54-409e-8c60-17bd9473f1df" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:59:41.248355 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.248327    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx"
Sep 16 18:59:41.300947 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.300881    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fv9vl\" (UniqueName: \"kubernetes.io/projected/1eb8f8d7-eb54-409e-8c60-17bd9473f1df-kube-api-access-fv9vl\") pod \"1eb8f8d7-eb54-409e-8c60-17bd9473f1df\" (UID: \"1eb8f8d7-eb54-409e-8c60-17bd9473f1df\") "
Sep 16 18:59:41.300947 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.300929    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eb8f8d7-eb54-409e-8c60-17bd9473f1df-tls-certs\") pod \"1eb8f8d7-eb54-409e-8c60-17bd9473f1df\" (UID: \"1eb8f8d7-eb54-409e-8c60-17bd9473f1df\") "
Sep 16 18:59:41.303076 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.303027    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eb8f8d7-eb54-409e-8c60-17bd9473f1df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1eb8f8d7-eb54-409e-8c60-17bd9473f1df" (UID: "1eb8f8d7-eb54-409e-8c60-17bd9473f1df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:59:41.303179 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.303080    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb8f8d7-eb54-409e-8c60-17bd9473f1df-kube-api-access-fv9vl" (OuterVolumeSpecName: "kube-api-access-fv9vl") pod "1eb8f8d7-eb54-409e-8c60-17bd9473f1df" (UID: "1eb8f8d7-eb54-409e-8c60-17bd9473f1df"). InnerVolumeSpecName "kube-api-access-fv9vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:59:41.401782 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.401759    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv9vl\" (UniqueName: \"kubernetes.io/projected/1eb8f8d7-eb54-409e-8c60-17bd9473f1df-kube-api-access-fv9vl\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:59:41.401782 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.401780    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eb8f8d7-eb54-409e-8c60-17bd9473f1df-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 18:59:41.690921 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.690894    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t"]
Sep 16 18:59:41.691230 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.691216    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eb8f8d7-eb54-409e-8c60-17bd9473f1df" containerName="main"
Sep 16 18:59:41.691283 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.691232    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eb8f8d7-eb54-409e-8c60-17bd9473f1df" containerName="main"
Sep 16 18:59:41.691328 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.691318    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1eb8f8d7-eb54-409e-8c60-17bd9473f1df" containerName="main"
Sep 16 18:59:41.694267 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.694244    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t"
Sep 16 18:59:41.696868 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.696847    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-hk6wx\""
Sep 16 18:59:41.696986 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.696911    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 16 18:59:41.698003 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.697978    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 16 18:59:41.698114 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.698020    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-mbrg9\""
Sep 16 18:59:41.698114 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.698104    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 16 18:59:41.707806 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.707776    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t"]
Sep 16 18:59:41.796869 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.796839    2578 generic.go:356] "Generic (PLEG): container finished" podID="1eb8f8d7-eb54-409e-8c60-17bd9473f1df" containerID="3f324d51c697d68d8dffc69b3514059532ef81f60f9403460a44960414d78f63" exitCode=0
Sep 16 18:59:41.797026 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.796892    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx"
Sep 16 18:59:41.797026 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.796925    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx" event={"ID":"1eb8f8d7-eb54-409e-8c60-17bd9473f1df","Type":"ContainerDied","Data":"3f324d51c697d68d8dffc69b3514059532ef81f60f9403460a44960414d78f63"}
Sep 16 18:59:41.797026 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.797000    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx" event={"ID":"1eb8f8d7-eb54-409e-8c60-17bd9473f1df","Type":"ContainerDied","Data":"0e012353894067ade514efe26dad479bc40a7f5b79666664b21f86f8d040e3fb"}
Sep 16 18:59:41.797182 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.797027    2578 scope.go:117] "RemoveContainer" containerID="3f324d51c697d68d8dffc69b3514059532ef81f60f9403460a44960414d78f63"
Sep 16 18:59:41.805337 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.805316    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52acd63d-b06a-454b-91b7-d185c38b70f9-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t\" (UID: \"52acd63d-b06a-454b-91b7-d185c38b70f9\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t"
Sep 16 18:59:41.805467 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.805353    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgtpn\" (UniqueName: \"kubernetes.io/projected/52acd63d-b06a-454b-91b7-d185c38b70f9-kube-api-access-tgtpn\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t\" (UID: \"52acd63d-b06a-454b-91b7-d185c38b70f9\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t"
Sep 16 18:59:41.806319 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.806294    2578 scope.go:117] "RemoveContainer" containerID="3f324d51c697d68d8dffc69b3514059532ef81f60f9403460a44960414d78f63"
Sep 16 18:59:41.806598 ip-10-0-142-80 kubenswrapper[2578]: E0916 18:59:41.806577    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f324d51c697d68d8dffc69b3514059532ef81f60f9403460a44960414d78f63\": container with ID starting with 3f324d51c697d68d8dffc69b3514059532ef81f60f9403460a44960414d78f63 not found: ID does not exist" containerID="3f324d51c697d68d8dffc69b3514059532ef81f60f9403460a44960414d78f63"
Sep 16 18:59:41.806686 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.806603    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f324d51c697d68d8dffc69b3514059532ef81f60f9403460a44960414d78f63"} err="failed to get container status \"3f324d51c697d68d8dffc69b3514059532ef81f60f9403460a44960414d78f63\": rpc error: code = NotFound desc = could not find container \"3f324d51c697d68d8dffc69b3514059532ef81f60f9403460a44960414d78f63\": container with ID starting with 3f324d51c697d68d8dffc69b3514059532ef81f60f9403460a44960414d78f63 not found: ID does not exist"
Sep 16 18:59:41.832427 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.832406    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx"]
Sep 16 18:59:41.837909 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.837888    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-6dzfx"]
Sep 16 18:59:41.906290 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.906266    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52acd63d-b06a-454b-91b7-d185c38b70f9-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t\" (UID: \"52acd63d-b06a-454b-91b7-d185c38b70f9\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t"
Sep 16 18:59:41.906381 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.906300    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgtpn\" (UniqueName: \"kubernetes.io/projected/52acd63d-b06a-454b-91b7-d185c38b70f9-kube-api-access-tgtpn\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t\" (UID: \"52acd63d-b06a-454b-91b7-d185c38b70f9\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t"
Sep 16 18:59:41.908726 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.908697    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52acd63d-b06a-454b-91b7-d185c38b70f9-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t\" (UID: \"52acd63d-b06a-454b-91b7-d185c38b70f9\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t"
Sep 16 18:59:41.915790 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:41.915767    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgtpn\" (UniqueName: \"kubernetes.io/projected/52acd63d-b06a-454b-91b7-d185c38b70f9-kube-api-access-tgtpn\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t\" (UID: \"52acd63d-b06a-454b-91b7-d185c38b70f9\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t"
Sep 16 18:59:42.006590 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:42.006529    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t"
Sep 16 18:59:42.217802 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:42.217766    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1eb8f8d7-eb54-409e-8c60-17bd9473f1df" path="/var/lib/kubelet/pods/1eb8f8d7-eb54-409e-8c60-17bd9473f1df/volumes"
Sep 16 18:59:42.286119 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:42.286092    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t"]
Sep 16 18:59:42.805903 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:42.805869    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t" event={"ID":"52acd63d-b06a-454b-91b7-d185c38b70f9","Type":"ContainerStarted","Data":"7a05a38bdc43eb4c921fa6d9025e1b9946b3b037c109bae29996affd1e4a01bf"}
Sep 16 18:59:42.806100 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:42.805918    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t" event={"ID":"52acd63d-b06a-454b-91b7-d185c38b70f9","Type":"ContainerStarted","Data":"9c068532f75f5ac7c9b5841836f059a8ef1285267618729d77d12cd4c64d6bf2"}
Sep 16 18:59:42.806523 ip-10-0-142-80 kubenswrapper[2578]: I0916 18:59:42.806504    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t"
Sep 16 19:00:14.817922 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:00:14.817885    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t"
Sep 16 19:00:14.860357 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:00:14.860296    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t" podStartSLOduration=33.860257266 podStartE2EDuration="33.860257266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:42.835618358 +0000 UTC m=+1151.070551804" watchObservedRunningTime="2026-09-16 19:00:14.860257266 +0000 UTC m=+1183.095190711"
Sep 16 19:00:30.633490 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:00:30.633455    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t"]
Sep 16 19:00:34.817991 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:00:34.817949    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t" podUID="52acd63d-b06a-454b-91b7-d185c38b70f9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:00:44.817948 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:00:44.817901    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t" podUID="52acd63d-b06a-454b-91b7-d185c38b70f9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:00:45.651928 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:00:45.651891    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t" podUID="52acd63d-b06a-454b-91b7-d185c38b70f9" containerName="main" containerID="cri-o://7a05a38bdc43eb4c921fa6d9025e1b9946b3b037c109bae29996affd1e4a01bf" gracePeriod=45
Sep 16 19:00:54.818031 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:00:54.817987    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t" podUID="52acd63d-b06a-454b-91b7-d185c38b70f9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:00:54.818458 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:00:54.818132    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t"
Sep 16 19:01:04.818149 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:04.818104    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t" podUID="52acd63d-b06a-454b-91b7-d185c38b70f9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:01:14.818377 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:14.818333    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t" podUID="52acd63d-b06a-454b-91b7-d185c38b70f9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:01:15.891552 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:15.891528    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t"
Sep 16 19:01:16.000724 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:16.000691    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52acd63d-b06a-454b-91b7-d185c38b70f9-tls-certs\") pod \"52acd63d-b06a-454b-91b7-d185c38b70f9\" (UID: \"52acd63d-b06a-454b-91b7-d185c38b70f9\") "
Sep 16 19:01:16.000878 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:16.000757    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgtpn\" (UniqueName: \"kubernetes.io/projected/52acd63d-b06a-454b-91b7-d185c38b70f9-kube-api-access-tgtpn\") pod \"52acd63d-b06a-454b-91b7-d185c38b70f9\" (UID: \"52acd63d-b06a-454b-91b7-d185c38b70f9\") "
Sep 16 19:01:16.002795 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:16.002768    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52acd63d-b06a-454b-91b7-d185c38b70f9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "52acd63d-b06a-454b-91b7-d185c38b70f9" (UID: "52acd63d-b06a-454b-91b7-d185c38b70f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:01:16.002965 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:16.002825    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52acd63d-b06a-454b-91b7-d185c38b70f9-kube-api-access-tgtpn" (OuterVolumeSpecName: "kube-api-access-tgtpn") pod "52acd63d-b06a-454b-91b7-d185c38b70f9" (UID: "52acd63d-b06a-454b-91b7-d185c38b70f9"). InnerVolumeSpecName "kube-api-access-tgtpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:16.102086 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:16.102057    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52acd63d-b06a-454b-91b7-d185c38b70f9-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:01:16.102086 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:16.102086    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgtpn\" (UniqueName: \"kubernetes.io/projected/52acd63d-b06a-454b-91b7-d185c38b70f9-kube-api-access-tgtpn\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:01:16.299553 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:16.299496    2578 generic.go:356] "Generic (PLEG): container finished" podID="52acd63d-b06a-454b-91b7-d185c38b70f9" containerID="7a05a38bdc43eb4c921fa6d9025e1b9946b3b037c109bae29996affd1e4a01bf" exitCode=0
Sep 16 19:01:16.299553 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:16.299537    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t"
Sep 16 19:01:16.299673 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:16.299580    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t" event={"ID":"52acd63d-b06a-454b-91b7-d185c38b70f9","Type":"ContainerDied","Data":"7a05a38bdc43eb4c921fa6d9025e1b9946b3b037c109bae29996affd1e4a01bf"}
Sep 16 19:01:16.299673 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:16.299626    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t" event={"ID":"52acd63d-b06a-454b-91b7-d185c38b70f9","Type":"ContainerDied","Data":"9c068532f75f5ac7c9b5841836f059a8ef1285267618729d77d12cd4c64d6bf2"}
Sep 16 19:01:16.299673 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:16.299650    2578 scope.go:117] "RemoveContainer" containerID="7a05a38bdc43eb4c921fa6d9025e1b9946b3b037c109bae29996affd1e4a01bf"
Sep 16 19:01:16.313681 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:16.313658    2578 scope.go:117] "RemoveContainer" containerID="7a05a38bdc43eb4c921fa6d9025e1b9946b3b037c109bae29996affd1e4a01bf"
Sep 16 19:01:16.314196 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:01:16.314149    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a05a38bdc43eb4c921fa6d9025e1b9946b3b037c109bae29996affd1e4a01bf\": container with ID starting with 7a05a38bdc43eb4c921fa6d9025e1b9946b3b037c109bae29996affd1e4a01bf not found: ID does not exist" containerID="7a05a38bdc43eb4c921fa6d9025e1b9946b3b037c109bae29996affd1e4a01bf"
Sep 16 19:01:16.314281 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:16.314226    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a05a38bdc43eb4c921fa6d9025e1b9946b3b037c109bae29996affd1e4a01bf"} err="failed to get container status \"7a05a38bdc43eb4c921fa6d9025e1b9946b3b037c109bae29996affd1e4a01bf\": rpc error: code = NotFound desc = could not find container \"7a05a38bdc43eb4c921fa6d9025e1b9946b3b037c109bae29996affd1e4a01bf\": container with ID starting with 7a05a38bdc43eb4c921fa6d9025e1b9946b3b037c109bae29996affd1e4a01bf not found: ID does not exist"
Sep 16 19:01:16.321917 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:16.321896    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t"]
Sep 16 19:01:16.327446 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:16.327425    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9lp57t"]
Sep 16 19:01:18.217808 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:18.217774    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52acd63d-b06a-454b-91b7-d185c38b70f9" path="/var/lib/kubelet/pods/52acd63d-b06a-454b-91b7-d185c38b70f9/volumes"
Sep 16 19:01:40.007284 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:40.007249    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj"]
Sep 16 19:01:40.007958 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:40.007928    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52acd63d-b06a-454b-91b7-d185c38b70f9" containerName="main"
Sep 16 19:01:40.007958 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:40.007956    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="52acd63d-b06a-454b-91b7-d185c38b70f9" containerName="main"
Sep 16 19:01:40.008135 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:40.008056    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="52acd63d-b06a-454b-91b7-d185c38b70f9" containerName="main"
Sep 16 19:01:40.012347 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:40.012327    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj"
Sep 16 19:01:40.015377 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:40.015352    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 16 19:01:40.016411 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:40.016391    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 16 19:01:40.016474 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:40.016403    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-7ns9g\""
Sep 16 19:01:40.016742 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:40.016722    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 16 19:01:40.016835 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:40.016797    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-5mr8r\""
Sep 16 19:01:40.032760 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:40.032737    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj"]
Sep 16 19:01:40.184057 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:40.184006    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pttk\" (UniqueName: \"kubernetes.io/projected/7f143187-23f4-4ed9-866d-f5519e373d0b-kube-api-access-9pttk\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj\" (UID: \"7f143187-23f4-4ed9-866d-f5519e373d0b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj"
Sep 16 19:01:40.184184 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:40.184075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f143187-23f4-4ed9-866d-f5519e373d0b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj\" (UID: \"7f143187-23f4-4ed9-866d-f5519e373d0b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj"
Sep 16 19:01:40.285365 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:40.285298    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pttk\" (UniqueName: \"kubernetes.io/projected/7f143187-23f4-4ed9-866d-f5519e373d0b-kube-api-access-9pttk\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj\" (UID: \"7f143187-23f4-4ed9-866d-f5519e373d0b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj"
Sep 16 19:01:40.285365 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:40.285347    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f143187-23f4-4ed9-866d-f5519e373d0b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj\" (UID: \"7f143187-23f4-4ed9-866d-f5519e373d0b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj"
Sep 16 19:01:40.287699 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:40.287677    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f143187-23f4-4ed9-866d-f5519e373d0b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj\" (UID: \"7f143187-23f4-4ed9-866d-f5519e373d0b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj"
Sep 16 19:01:40.297934 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:40.297912    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pttk\" (UniqueName: \"kubernetes.io/projected/7f143187-23f4-4ed9-866d-f5519e373d0b-kube-api-access-9pttk\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj\" (UID: \"7f143187-23f4-4ed9-866d-f5519e373d0b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj"
Sep 16 19:01:40.322874 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:40.322851    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj"
Sep 16 19:01:40.599513 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:40.599485    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj"]
Sep 16 19:01:40.601105 ip-10-0-142-80 kubenswrapper[2578]: W0916 19:01:40.601078    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f143187_23f4_4ed9_866d_f5519e373d0b.slice/crio-4cc0644ff314c0b6cb76437fe704794281df6f9829ef29ccef09e75dd98a58d2 WatchSource:0}: Error finding container 4cc0644ff314c0b6cb76437fe704794281df6f9829ef29ccef09e75dd98a58d2: Status 404 returned error can't find the container with id 4cc0644ff314c0b6cb76437fe704794281df6f9829ef29ccef09e75dd98a58d2
Sep 16 19:01:41.427593 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:41.427559    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj" event={"ID":"7f143187-23f4-4ed9-866d-f5519e373d0b","Type":"ContainerStarted","Data":"c15ced86bf84df97d243dd10a03e8a66cbb37bb6be588bcc6c7345c6d7e7617f"}
Sep 16 19:01:41.427987 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:41.427612    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj" event={"ID":"7f143187-23f4-4ed9-866d-f5519e373d0b","Type":"ContainerStarted","Data":"4cc0644ff314c0b6cb76437fe704794281df6f9829ef29ccef09e75dd98a58d2"}
Sep 16 19:01:41.427987 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:41.427661    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj"
Sep 16 19:01:41.448226 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:01:41.448182    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj" podStartSLOduration=2.448167747 podStartE2EDuration="2.448167747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:01:41.446202886 +0000 UTC m=+1269.681136328" watchObservedRunningTime="2026-09-16 19:01:41.448167747 +0000 UTC m=+1269.683101169"
Sep 16 19:02:12.433699 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:02:12.433667    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj"
Sep 16 19:02:15.265833 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:02:15.265800    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj"]
Sep 16 19:02:22.434065 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:02:22.434003    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj" podUID="7f143187-23f4-4ed9-866d-f5519e373d0b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:02:26.944487 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:02:26.944448    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 16 19:02:26.944487 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:02:26.944487    2578 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj" podUID="7f143187-23f4-4ed9-866d-f5519e373d0b" containerName="main" containerID="cri-o://c15ced86bf84df97d243dd10a03e8a66cbb37bb6be588bcc6c7345c6d7e7617f"
Sep 16 19:02:26.944984 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:02:26.944530    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj" podUID="7f143187-23f4-4ed9-866d-f5519e373d0b" containerName="main" containerID="cri-o://c15ced86bf84df97d243dd10a03e8a66cbb37bb6be588bcc6c7345c6d7e7617f" gracePeriod=49
Sep 16 19:02:27.075738 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:02:27.075718    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj_7f143187-23f4-4ed9-866d-f5519e373d0b/main/0.log"
Sep 16 19:02:27.077864 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:02:27.077850    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj"
Sep 16 19:02:27.134362 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:02:27.134337    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f143187-23f4-4ed9-866d-f5519e373d0b-tls-certs\") pod \"7f143187-23f4-4ed9-866d-f5519e373d0b\" (UID: \"7f143187-23f4-4ed9-866d-f5519e373d0b\") "
Sep 16 19:02:27.134476 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:02:27.134396    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pttk\" (UniqueName: \"kubernetes.io/projected/7f143187-23f4-4ed9-866d-f5519e373d0b-kube-api-access-9pttk\") pod \"7f143187-23f4-4ed9-866d-f5519e373d0b\" (UID: \"7f143187-23f4-4ed9-866d-f5519e373d0b\") "
Sep 16 19:02:27.136380 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:02:27.136350    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f143187-23f4-4ed9-866d-f5519e373d0b-kube-api-access-9pttk" (OuterVolumeSpecName: "kube-api-access-9pttk") pod "7f143187-23f4-4ed9-866d-f5519e373d0b" (UID: "7f143187-23f4-4ed9-866d-f5519e373d0b"). InnerVolumeSpecName "kube-api-access-9pttk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:02:27.136380 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:02:27.136366    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f143187-23f4-4ed9-866d-f5519e373d0b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f143187-23f4-4ed9-866d-f5519e373d0b" (UID: "7f143187-23f4-4ed9-866d-f5519e373d0b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:02:27.235102 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:02:27.235028    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f143187-23f4-4ed9-866d-f5519e373d0b-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:02:27.235102 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:02:27.235073    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pttk\" (UniqueName: \"kubernetes.io/projected/7f143187-23f4-4ed9-866d-f5519e373d0b-kube-api-access-9pttk\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:02:27.658845 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:02:27.658784    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj_7f143187-23f4-4ed9-866d-f5519e373d0b/main/0.log"
Sep 16 19:02:27.658845 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:02:27.658836    2578 generic.go:356] "Generic (PLEG): container finished" podID="7f143187-23f4-4ed9-866d-f5519e373d0b" containerID="c15ced86bf84df97d243dd10a03e8a66cbb37bb6be588bcc6c7345c6d7e7617f" exitCode=1
Sep 16 19:02:27.659004 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:02:27.658883    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj"
Sep 16 19:02:27.659004 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:02:27.658916    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj" event={"ID":"7f143187-23f4-4ed9-866d-f5519e373d0b","Type":"ContainerDied","Data":"c15ced86bf84df97d243dd10a03e8a66cbb37bb6be588bcc6c7345c6d7e7617f"}
Sep 16 19:02:27.659004 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:02:27.658973    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj" event={"ID":"7f143187-23f4-4ed9-866d-f5519e373d0b","Type":"ContainerDied","Data":"4cc0644ff314c0b6cb76437fe704794281df6f9829ef29ccef09e75dd98a58d2"}
Sep 16 19:02:27.659004 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:02:27.658995    2578 scope.go:117] "RemoveContainer" containerID="c15ced86bf84df97d243dd10a03e8a66cbb37bb6be588bcc6c7345c6d7e7617f"
Sep 16 19:02:27.670331 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:02:27.670315    2578 scope.go:117] "RemoveContainer" containerID="c15ced86bf84df97d243dd10a03e8a66cbb37bb6be588bcc6c7345c6d7e7617f"
Sep 16 19:02:27.670606 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:02:27.670587    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c15ced86bf84df97d243dd10a03e8a66cbb37bb6be588bcc6c7345c6d7e7617f\": container with ID starting with c15ced86bf84df97d243dd10a03e8a66cbb37bb6be588bcc6c7345c6d7e7617f not found: ID does not exist" containerID="c15ced86bf84df97d243dd10a03e8a66cbb37bb6be588bcc6c7345c6d7e7617f"
Sep 16 19:02:27.670658 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:02:27.670615    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c15ced86bf84df97d243dd10a03e8a66cbb37bb6be588bcc6c7345c6d7e7617f"} err="failed to get container status \"c15ced86bf84df97d243dd10a03e8a66cbb37bb6be588bcc6c7345c6d7e7617f\": rpc error: code = NotFound desc = could not find container \"c15ced86bf84df97d243dd10a03e8a66cbb37bb6be588bcc6c7345c6d7e7617f\": container with ID starting with c15ced86bf84df97d243dd10a03e8a66cbb37bb6be588bcc6c7345c6d7e7617f not found: ID does not exist"
Sep 16 19:02:27.684764 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:02:27.684742    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj"]
Sep 16 19:02:27.692011 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:02:27.691990    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5k8ckj"]
Sep 16 19:02:28.217368 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:02:28.217336    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f143187-23f4-4ed9-866d-f5519e373d0b" path="/var/lib/kubelet/pods/7f143187-23f4-4ed9-866d-f5519e373d0b/volumes"
Sep 16 19:03:53.647669 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.647631    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8"]
Sep 16 19:03:53.648325 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.648172    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f143187-23f4-4ed9-866d-f5519e373d0b" containerName="main"
Sep 16 19:03:53.648325 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.648196    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f143187-23f4-4ed9-866d-f5519e373d0b" containerName="main"
Sep 16 19:03:53.648325 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.648297    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f143187-23f4-4ed9-866d-f5519e373d0b" containerName="main"
Sep 16 19:03:53.651243 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.651218    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8"
Sep 16 19:03:53.655222 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.655194    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 16 19:03:53.655383 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.655227    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 16 19:03:53.655383 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.655197    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 16 19:03:53.655383 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.655343    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-q6fr8\""
Sep 16 19:03:53.660697 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.660667    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8"]
Sep 16 19:03:53.764197 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.764152    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02813cba-6a55-4181-b392-d504ad2ee4e0-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fbdq8\" (UID: \"02813cba-6a55-4181-b392-d504ad2ee4e0\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8"
Sep 16 19:03:53.764368 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.764204    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02813cba-6a55-4181-b392-d504ad2ee4e0-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fbdq8\" (UID: \"02813cba-6a55-4181-b392-d504ad2ee4e0\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8"
Sep 16 19:03:53.764368 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.764266    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02813cba-6a55-4181-b392-d504ad2ee4e0-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fbdq8\" (UID: \"02813cba-6a55-4181-b392-d504ad2ee4e0\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8"
Sep 16 19:03:53.764368 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.764291    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02813cba-6a55-4181-b392-d504ad2ee4e0-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fbdq8\" (UID: \"02813cba-6a55-4181-b392-d504ad2ee4e0\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8"
Sep 16 19:03:53.764368 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.764320    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgcv6\" (UniqueName: \"kubernetes.io/projected/02813cba-6a55-4181-b392-d504ad2ee4e0-kube-api-access-kgcv6\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fbdq8\" (UID: \"02813cba-6a55-4181-b392-d504ad2ee4e0\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8"
Sep 16 19:03:53.764368 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.764353    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02813cba-6a55-4181-b392-d504ad2ee4e0-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fbdq8\" (UID: \"02813cba-6a55-4181-b392-d504ad2ee4e0\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8"
Sep 16 19:03:53.865442 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.865405    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02813cba-6a55-4181-b392-d504ad2ee4e0-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fbdq8\" (UID: \"02813cba-6a55-4181-b392-d504ad2ee4e0\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8"
Sep 16 19:03:53.865442 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.865443    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02813cba-6a55-4181-b392-d504ad2ee4e0-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fbdq8\" (UID: \"02813cba-6a55-4181-b392-d504ad2ee4e0\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8"
Sep 16 19:03:53.865661 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.865471    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02813cba-6a55-4181-b392-d504ad2ee4e0-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fbdq8\" (UID: \"02813cba-6a55-4181-b392-d504ad2ee4e0\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8"
Sep 16 19:03:53.865661 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.865580    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02813cba-6a55-4181-b392-d504ad2ee4e0-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fbdq8\" (UID: \"02813cba-6a55-4181-b392-d504ad2ee4e0\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8"
Sep 16 19:03:53.865661 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.865619    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgcv6\" (UniqueName: \"kubernetes.io/projected/02813cba-6a55-4181-b392-d504ad2ee4e0-kube-api-access-kgcv6\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fbdq8\" (UID: \"02813cba-6a55-4181-b392-d504ad2ee4e0\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8"
Sep 16 19:03:53.865661 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.865653    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02813cba-6a55-4181-b392-d504ad2ee4e0-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fbdq8\" (UID: \"02813cba-6a55-4181-b392-d504ad2ee4e0\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8"
Sep 16 19:03:53.865877 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.865819    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02813cba-6a55-4181-b392-d504ad2ee4e0-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fbdq8\" (UID: \"02813cba-6a55-4181-b392-d504ad2ee4e0\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8"
Sep 16 19:03:53.865933 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.865895    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02813cba-6a55-4181-b392-d504ad2ee4e0-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fbdq8\" (UID: \"02813cba-6a55-4181-b392-d504ad2ee4e0\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8"
Sep 16 19:03:53.866012 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.865991    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02813cba-6a55-4181-b392-d504ad2ee4e0-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fbdq8\" (UID: \"02813cba-6a55-4181-b392-d504ad2ee4e0\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8"
Sep 16 19:03:53.867926 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.867895    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02813cba-6a55-4181-b392-d504ad2ee4e0-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fbdq8\" (UID: \"02813cba-6a55-4181-b392-d504ad2ee4e0\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8"
Sep 16 19:03:53.868376 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.868351    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02813cba-6a55-4181-b392-d504ad2ee4e0-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fbdq8\" (UID: \"02813cba-6a55-4181-b392-d504ad2ee4e0\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8"
Sep 16 19:03:53.876683 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.876662    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgcv6\" (UniqueName: \"kubernetes.io/projected/02813cba-6a55-4181-b392-d504ad2ee4e0-kube-api-access-kgcv6\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fbdq8\" (UID: \"02813cba-6a55-4181-b392-d504ad2ee4e0\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8"
Sep 16 19:03:53.906084 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.905999    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf"]
Sep 16 19:03:53.909445 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.909426    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf"
Sep 16 19:03:53.912507 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.912490    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-s2cnt\""
Sep 16 19:03:53.922706 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.922683    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf"]
Sep 16 19:03:53.964723 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.964698    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8"
Sep 16 19:03:53.966942 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.966920    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a8b15c3-7ee9-45d1-bc6c-54981e420e88-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf\" (UID: \"7a8b15c3-7ee9-45d1-bc6c-54981e420e88\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf"
Sep 16 19:03:53.967056 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.966961    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a8b15c3-7ee9-45d1-bc6c-54981e420e88-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf\" (UID: \"7a8b15c3-7ee9-45d1-bc6c-54981e420e88\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf"
Sep 16 19:03:53.967130 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.967087    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a8b15c3-7ee9-45d1-bc6c-54981e420e88-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf\" (UID: \"7a8b15c3-7ee9-45d1-bc6c-54981e420e88\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf"
Sep 16 19:03:53.967187 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:53.967129    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62ghj\" (UniqueName: \"kubernetes.io/projected/7a8b15c3-7ee9-45d1-bc6c-54981e420e88-kube-api-access-62ghj\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf\" (UID: \"7a8b15c3-7ee9-45d1-bc6c-54981e420e88\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf"
Sep 16 19:03:54.067834 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:54.067804    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a8b15c3-7ee9-45d1-bc6c-54981e420e88-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf\" (UID: \"7a8b15c3-7ee9-45d1-bc6c-54981e420e88\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf"
Sep 16 19:03:54.067979 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:54.067842    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62ghj\" (UniqueName: \"kubernetes.io/projected/7a8b15c3-7ee9-45d1-bc6c-54981e420e88-kube-api-access-62ghj\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf\" (UID: \"7a8b15c3-7ee9-45d1-bc6c-54981e420e88\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf"
Sep 16 19:03:54.067979 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:54.067872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a8b15c3-7ee9-45d1-bc6c-54981e420e88-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf\" (UID: \"7a8b15c3-7ee9-45d1-bc6c-54981e420e88\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf"
Sep 16 19:03:54.067979 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:54.067894    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a8b15c3-7ee9-45d1-bc6c-54981e420e88-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf\" (UID: \"7a8b15c3-7ee9-45d1-bc6c-54981e420e88\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf"
Sep 16 19:03:54.068304 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:54.068283    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a8b15c3-7ee9-45d1-bc6c-54981e420e88-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf\" (UID: \"7a8b15c3-7ee9-45d1-bc6c-54981e420e88\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf"
Sep 16 19:03:54.068344 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:54.068287    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a8b15c3-7ee9-45d1-bc6c-54981e420e88-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf\" (UID: \"7a8b15c3-7ee9-45d1-bc6c-54981e420e88\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf"
Sep 16 19:03:54.070181 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:54.070163    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a8b15c3-7ee9-45d1-bc6c-54981e420e88-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf\" (UID: \"7a8b15c3-7ee9-45d1-bc6c-54981e420e88\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf"
Sep 16 19:03:54.076670 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:54.076649    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62ghj\" (UniqueName: \"kubernetes.io/projected/7a8b15c3-7ee9-45d1-bc6c-54981e420e88-kube-api-access-62ghj\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf\" (UID: \"7a8b15c3-7ee9-45d1-bc6c-54981e420e88\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf"
Sep 16 19:03:54.220188 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:54.220163    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf"
Sep 16 19:03:54.236341 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:54.236317    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8"]
Sep 16 19:03:54.238161 ip-10-0-142-80 kubenswrapper[2578]: W0916 19:03:54.238130    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02813cba_6a55_4181_b392_d504ad2ee4e0.slice/crio-cf862954bd5871fcafeaf7ce521ba630bd0df5e7dbac3f6bf1fbdf19982a41e3 WatchSource:0}: Error finding container cf862954bd5871fcafeaf7ce521ba630bd0df5e7dbac3f6bf1fbdf19982a41e3: Status 404 returned error can't find the container with id cf862954bd5871fcafeaf7ce521ba630bd0df5e7dbac3f6bf1fbdf19982a41e3
Sep 16 19:03:54.239998 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:54.239980    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:03:54.493872 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:54.493845    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf"]
Sep 16 19:03:54.497755 ip-10-0-142-80 kubenswrapper[2578]: W0916 19:03:54.497724    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a8b15c3_7ee9_45d1_bc6c_54981e420e88.slice/crio-09814bc9211d10fc1d98ec5a2c88875d8b33c481347b183f39019ff0e696893f WatchSource:0}: Error finding container 09814bc9211d10fc1d98ec5a2c88875d8b33c481347b183f39019ff0e696893f: Status 404 returned error can't find the container with id 09814bc9211d10fc1d98ec5a2c88875d8b33c481347b183f39019ff0e696893f
Sep 16 19:03:55.107328 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:55.107290    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8" event={"ID":"02813cba-6a55-4181-b392-d504ad2ee4e0","Type":"ContainerStarted","Data":"cf862954bd5871fcafeaf7ce521ba630bd0df5e7dbac3f6bf1fbdf19982a41e3"}
Sep 16 19:03:55.109150 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:55.109121    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf" event={"ID":"7a8b15c3-7ee9-45d1-bc6c-54981e420e88","Type":"ContainerStarted","Data":"9bb416d5b1948d4acab35b8c8a7ddf6047b3f61c852a621b99799397a3401c71"}
Sep 16 19:03:55.109289 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:55.109167    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf" event={"ID":"7a8b15c3-7ee9-45d1-bc6c-54981e420e88","Type":"ContainerStarted","Data":"09814bc9211d10fc1d98ec5a2c88875d8b33c481347b183f39019ff0e696893f"}
Sep 16 19:03:55.109289 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:55.109271    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf"
Sep 16 19:03:55.133407 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:55.133356    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf" podStartSLOduration=2.133340408 podStartE2EDuration="2.133340408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:03:55.132187094 +0000 UTC m=+1403.367120550" watchObservedRunningTime="2026-09-16 19:03:55.133340408 +0000 UTC m=+1403.368273850"
Sep 16 19:03:56.115013 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:56.114972    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8" event={"ID":"02813cba-6a55-4181-b392-d504ad2ee4e0","Type":"ContainerStarted","Data":"2b4fdb59fd9477b4f2e74dbbeab1bbfb509668bc03d1ab713da7eec950363e01"}
Sep 16 19:03:56.135840 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:56.135787    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8" podStartSLOduration=1.455087085 podStartE2EDuration="3.135768328s" totalImagesPullingTime="1.680681243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:03:56.134820076 +0000 UTC m=+1404.369753531" watchObservedRunningTime="2026-09-16 19:03:56.135768328 +0000 UTC m=+1404.370701772"
Sep 16 19:03:56.965495 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:03:56.965463    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8"
Sep 16 19:04:03.965777 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:03.965735    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8"
Sep 16 19:04:03.978056 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:03.978006    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8"
Sep 16 19:04:04.166857 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:04.166831    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8"
Sep 16 19:04:26.116846 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:26.116770    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf"
Sep 16 19:04:27.649749 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:27.649721    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf"]
Sep 16 19:04:27.652924 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:27.652895    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8"]
Sep 16 19:04:27.661821 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:04:27.661772    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:04:27.661821 ip-10-0-142-80 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:04:27.661821 ip-10-0-142-80 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:04:27.661821 ip-10-0-142-80 kubenswrapper[2578]:  > containerID="2b4fdb59fd9477b4f2e74dbbeab1bbfb509668bc03d1ab713da7eec950363e01" cmd=["/bin/sleep","15"]
Sep 16 19:04:27.662110 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:27.661834    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8" message="" err=<
Sep 16 19:04:27.662110 ip-10-0-142-80 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:04:27.662110 ip-10-0-142-80 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:04:27.662110 ip-10-0-142-80 kubenswrapper[2578]:  >
Sep 16 19:04:27.662110 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:04:27.661850    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:04:27.662110 ip-10-0-142-80 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:04:27.662110 ip-10-0-142-80 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:04:27.662110 ip-10-0-142-80 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8" podUID="02813cba-6a55-4181-b392-d504ad2ee4e0" containerName="main" containerID="cri-o://2b4fdb59fd9477b4f2e74dbbeab1bbfb509668bc03d1ab713da7eec950363e01"
Sep 16 19:04:27.662110 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:27.661903    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8" podUID="02813cba-6a55-4181-b392-d504ad2ee4e0" containerName="main" containerID="cri-o://2b4fdb59fd9477b4f2e74dbbeab1bbfb509668bc03d1ab713da7eec950363e01" gracePeriod=60
Sep 16 19:04:27.998429 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:27.998407    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8"
Sep 16 19:04:28.129004 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:28.128978    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02813cba-6a55-4181-b392-d504ad2ee4e0-model-cache\") pod \"02813cba-6a55-4181-b392-d504ad2ee4e0\" (UID: \"02813cba-6a55-4181-b392-d504ad2ee4e0\") "
Sep 16 19:04:28.129165 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:28.129024    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgcv6\" (UniqueName: \"kubernetes.io/projected/02813cba-6a55-4181-b392-d504ad2ee4e0-kube-api-access-kgcv6\") pod \"02813cba-6a55-4181-b392-d504ad2ee4e0\" (UID: \"02813cba-6a55-4181-b392-d504ad2ee4e0\") "
Sep 16 19:04:28.129165 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:28.129062    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02813cba-6a55-4181-b392-d504ad2ee4e0-tls-certs\") pod \"02813cba-6a55-4181-b392-d504ad2ee4e0\" (UID: \"02813cba-6a55-4181-b392-d504ad2ee4e0\") "
Sep 16 19:04:28.129165 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:28.129099    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02813cba-6a55-4181-b392-d504ad2ee4e0-dshm\") pod \"02813cba-6a55-4181-b392-d504ad2ee4e0\" (UID: \"02813cba-6a55-4181-b392-d504ad2ee4e0\") "
Sep 16 19:04:28.129165 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:28.129125    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02813cba-6a55-4181-b392-d504ad2ee4e0-home\") pod \"02813cba-6a55-4181-b392-d504ad2ee4e0\" (UID: \"02813cba-6a55-4181-b392-d504ad2ee4e0\") "
Sep 16 19:04:28.129165 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:28.129149    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02813cba-6a55-4181-b392-d504ad2ee4e0-tmp-dir\") pod \"02813cba-6a55-4181-b392-d504ad2ee4e0\" (UID: \"02813cba-6a55-4181-b392-d504ad2ee4e0\") "
Sep 16 19:04:28.129422 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:28.129271    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02813cba-6a55-4181-b392-d504ad2ee4e0-model-cache" (OuterVolumeSpecName: "model-cache") pod "02813cba-6a55-4181-b392-d504ad2ee4e0" (UID: "02813cba-6a55-4181-b392-d504ad2ee4e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:28.129422 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:28.129385    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02813cba-6a55-4181-b392-d504ad2ee4e0-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:04:28.129514 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:28.129439    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02813cba-6a55-4181-b392-d504ad2ee4e0-home" (OuterVolumeSpecName: "home") pod "02813cba-6a55-4181-b392-d504ad2ee4e0" (UID: "02813cba-6a55-4181-b392-d504ad2ee4e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:28.129514 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:28.129456    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02813cba-6a55-4181-b392-d504ad2ee4e0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "02813cba-6a55-4181-b392-d504ad2ee4e0" (UID: "02813cba-6a55-4181-b392-d504ad2ee4e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:28.131333 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:28.131309    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02813cba-6a55-4181-b392-d504ad2ee4e0-kube-api-access-kgcv6" (OuterVolumeSpecName: "kube-api-access-kgcv6") pod "02813cba-6a55-4181-b392-d504ad2ee4e0" (UID: "02813cba-6a55-4181-b392-d504ad2ee4e0"). InnerVolumeSpecName "kube-api-access-kgcv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:04:28.131446 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:28.131372    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02813cba-6a55-4181-b392-d504ad2ee4e0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "02813cba-6a55-4181-b392-d504ad2ee4e0" (UID: "02813cba-6a55-4181-b392-d504ad2ee4e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:04:28.131446 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:28.131376    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02813cba-6a55-4181-b392-d504ad2ee4e0-dshm" (OuterVolumeSpecName: "dshm") pod "02813cba-6a55-4181-b392-d504ad2ee4e0" (UID: "02813cba-6a55-4181-b392-d504ad2ee4e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:28.229872 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:28.229851    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02813cba-6a55-4181-b392-d504ad2ee4e0-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:04:28.229872 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:28.229872    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02813cba-6a55-4181-b392-d504ad2ee4e0-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:04:28.229996 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:28.229882    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgcv6\" (UniqueName: \"kubernetes.io/projected/02813cba-6a55-4181-b392-d504ad2ee4e0-kube-api-access-kgcv6\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:04:28.229996 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:28.229891    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02813cba-6a55-4181-b392-d504ad2ee4e0-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:04:28.229996 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:28.229899    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02813cba-6a55-4181-b392-d504ad2ee4e0-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:04:28.282171 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:28.282139    2578 generic.go:356] "Generic (PLEG): container finished" podID="02813cba-6a55-4181-b392-d504ad2ee4e0" containerID="2b4fdb59fd9477b4f2e74dbbeab1bbfb509668bc03d1ab713da7eec950363e01" exitCode=0
Sep 16 19:04:28.282309 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:28.282187    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8"
Sep 16 19:04:28.282309 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:28.282197    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8" event={"ID":"02813cba-6a55-4181-b392-d504ad2ee4e0","Type":"ContainerDied","Data":"2b4fdb59fd9477b4f2e74dbbeab1bbfb509668bc03d1ab713da7eec950363e01"}
Sep 16 19:04:28.282309 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:28.282245    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8" event={"ID":"02813cba-6a55-4181-b392-d504ad2ee4e0","Type":"ContainerDied","Data":"cf862954bd5871fcafeaf7ce521ba630bd0df5e7dbac3f6bf1fbdf19982a41e3"}
Sep 16 19:04:28.282309 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:28.282275    2578 scope.go:117] "RemoveContainer" containerID="2b4fdb59fd9477b4f2e74dbbeab1bbfb509668bc03d1ab713da7eec950363e01"
Sep 16 19:04:28.292683 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:28.292666    2578 scope.go:117] "RemoveContainer" containerID="2b4fdb59fd9477b4f2e74dbbeab1bbfb509668bc03d1ab713da7eec950363e01"
Sep 16 19:04:28.292955 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:04:28.292932    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b4fdb59fd9477b4f2e74dbbeab1bbfb509668bc03d1ab713da7eec950363e01\": container with ID starting with 2b4fdb59fd9477b4f2e74dbbeab1bbfb509668bc03d1ab713da7eec950363e01 not found: ID does not exist" containerID="2b4fdb59fd9477b4f2e74dbbeab1bbfb509668bc03d1ab713da7eec950363e01"
Sep 16 19:04:28.293021 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:28.292967    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b4fdb59fd9477b4f2e74dbbeab1bbfb509668bc03d1ab713da7eec950363e01"} err="failed to get container status \"2b4fdb59fd9477b4f2e74dbbeab1bbfb509668bc03d1ab713da7eec950363e01\": rpc error: code = NotFound desc = could not find container \"2b4fdb59fd9477b4f2e74dbbeab1bbfb509668bc03d1ab713da7eec950363e01\": container with ID starting with 2b4fdb59fd9477b4f2e74dbbeab1bbfb509668bc03d1ab713da7eec950363e01 not found: ID does not exist"
Sep 16 19:04:28.305797 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:28.305774    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8"]
Sep 16 19:04:28.312402 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:28.312375    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fbdq8"]
Sep 16 19:04:30.217179 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:30.217149    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02813cba-6a55-4181-b392-d504ad2ee4e0" path="/var/lib/kubelet/pods/02813cba-6a55-4181-b392-d504ad2ee4e0/volumes"
Sep 16 19:04:36.116719 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:36.116684    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf" podUID="7a8b15c3-7ee9-45d1-bc6c-54981e420e88" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:04:42.670660 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:42.670623    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf" podUID="7a8b15c3-7ee9-45d1-bc6c-54981e420e88" containerName="main" containerID="cri-o://9bb416d5b1948d4acab35b8c8a7ddf6047b3f61c852a621b99799397a3401c71" gracePeriod=45
Sep 16 19:04:46.117120 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:46.117080    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf" podUID="7a8b15c3-7ee9-45d1-bc6c-54981e420e88" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:04:56.116420 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:56.116382    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf" podUID="7a8b15c3-7ee9-45d1-bc6c-54981e420e88" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:04:56.116785 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:04:56.116532    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf"
Sep 16 19:05:06.117372 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:06.117326    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf" podUID="7a8b15c3-7ee9-45d1-bc6c-54981e420e88" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:05:12.908123 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:12.908098    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf"
Sep 16 19:05:12.950048 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:12.949973    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a8b15c3-7ee9-45d1-bc6c-54981e420e88-tls-certs\") pod \"7a8b15c3-7ee9-45d1-bc6c-54981e420e88\" (UID: \"7a8b15c3-7ee9-45d1-bc6c-54981e420e88\") "
Sep 16 19:05:12.950048 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:12.950020    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a8b15c3-7ee9-45d1-bc6c-54981e420e88-tokenizer-cache\") pod \"7a8b15c3-7ee9-45d1-bc6c-54981e420e88\" (UID: \"7a8b15c3-7ee9-45d1-bc6c-54981e420e88\") "
Sep 16 19:05:12.950224 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:12.950099    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62ghj\" (UniqueName: \"kubernetes.io/projected/7a8b15c3-7ee9-45d1-bc6c-54981e420e88-kube-api-access-62ghj\") pod \"7a8b15c3-7ee9-45d1-bc6c-54981e420e88\" (UID: \"7a8b15c3-7ee9-45d1-bc6c-54981e420e88\") "
Sep 16 19:05:12.950224 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:12.950145    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a8b15c3-7ee9-45d1-bc6c-54981e420e88-cachi2-cache\") pod \"7a8b15c3-7ee9-45d1-bc6c-54981e420e88\" (UID: \"7a8b15c3-7ee9-45d1-bc6c-54981e420e88\") "
Sep 16 19:05:12.950323 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:12.950279    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8b15c3-7ee9-45d1-bc6c-54981e420e88-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "7a8b15c3-7ee9-45d1-bc6c-54981e420e88" (UID: "7a8b15c3-7ee9-45d1-bc6c-54981e420e88"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:12.950418 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:12.950397    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8b15c3-7ee9-45d1-bc6c-54981e420e88-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "7a8b15c3-7ee9-45d1-bc6c-54981e420e88" (UID: "7a8b15c3-7ee9-45d1-bc6c-54981e420e88"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:12.950489 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:12.950458    2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a8b15c3-7ee9-45d1-bc6c-54981e420e88-tokenizer-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:05:12.951932 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:12.951912    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a8b15c3-7ee9-45d1-bc6c-54981e420e88-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7a8b15c3-7ee9-45d1-bc6c-54981e420e88" (UID: "7a8b15c3-7ee9-45d1-bc6c-54981e420e88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:05:12.952057 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:12.952028    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a8b15c3-7ee9-45d1-bc6c-54981e420e88-kube-api-access-62ghj" (OuterVolumeSpecName: "kube-api-access-62ghj") pod "7a8b15c3-7ee9-45d1-bc6c-54981e420e88" (UID: "7a8b15c3-7ee9-45d1-bc6c-54981e420e88"). InnerVolumeSpecName "kube-api-access-62ghj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:13.051804 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:13.051778    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62ghj\" (UniqueName: \"kubernetes.io/projected/7a8b15c3-7ee9-45d1-bc6c-54981e420e88-kube-api-access-62ghj\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:05:13.051804 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:13.051802    2578 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a8b15c3-7ee9-45d1-bc6c-54981e420e88-cachi2-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:05:13.051927 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:13.051812    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a8b15c3-7ee9-45d1-bc6c-54981e420e88-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:05:13.513225 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:13.513194    2578 generic.go:356] "Generic (PLEG): container finished" podID="7a8b15c3-7ee9-45d1-bc6c-54981e420e88" containerID="9bb416d5b1948d4acab35b8c8a7ddf6047b3f61c852a621b99799397a3401c71" exitCode=0
Sep 16 19:05:13.513375 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:13.513239    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf"
Sep 16 19:05:13.513375 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:13.513276    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf" event={"ID":"7a8b15c3-7ee9-45d1-bc6c-54981e420e88","Type":"ContainerDied","Data":"9bb416d5b1948d4acab35b8c8a7ddf6047b3f61c852a621b99799397a3401c71"}
Sep 16 19:05:13.513375 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:13.513325    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf" event={"ID":"7a8b15c3-7ee9-45d1-bc6c-54981e420e88","Type":"ContainerDied","Data":"09814bc9211d10fc1d98ec5a2c88875d8b33c481347b183f39019ff0e696893f"}
Sep 16 19:05:13.513375 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:13.513352    2578 scope.go:117] "RemoveContainer" containerID="9bb416d5b1948d4acab35b8c8a7ddf6047b3f61c852a621b99799397a3401c71"
Sep 16 19:05:13.522380 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:13.522368    2578 scope.go:117] "RemoveContainer" containerID="9bb416d5b1948d4acab35b8c8a7ddf6047b3f61c852a621b99799397a3401c71"
Sep 16 19:05:13.522650 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:05:13.522633    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bb416d5b1948d4acab35b8c8a7ddf6047b3f61c852a621b99799397a3401c71\": container with ID starting with 9bb416d5b1948d4acab35b8c8a7ddf6047b3f61c852a621b99799397a3401c71 not found: ID does not exist" containerID="9bb416d5b1948d4acab35b8c8a7ddf6047b3f61c852a621b99799397a3401c71"
Sep 16 19:05:13.522703 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:13.522659    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bb416d5b1948d4acab35b8c8a7ddf6047b3f61c852a621b99799397a3401c71"} err="failed to get container status \"9bb416d5b1948d4acab35b8c8a7ddf6047b3f61c852a621b99799397a3401c71\": rpc error: code = NotFound desc = could not find container \"9bb416d5b1948d4acab35b8c8a7ddf6047b3f61c852a621b99799397a3401c71\": container with ID starting with 9bb416d5b1948d4acab35b8c8a7ddf6047b3f61c852a621b99799397a3401c71 not found: ID does not exist"
Sep 16 19:05:13.539058 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:13.539021    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf"]
Sep 16 19:05:13.546284 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:13.546262    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcntmwf"]
Sep 16 19:05:14.217387 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:14.217353    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a8b15c3-7ee9-45d1-bc6c-54981e420e88" path="/var/lib/kubelet/pods/7a8b15c3-7ee9-45d1-bc6c-54981e420e88/volumes"
Sep 16 19:05:28.422813 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.422773    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"]
Sep 16 19:05:28.423353 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.423328    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a8b15c3-7ee9-45d1-bc6c-54981e420e88" containerName="main"
Sep 16 19:05:28.423353 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.423355    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a8b15c3-7ee9-45d1-bc6c-54981e420e88" containerName="main"
Sep 16 19:05:28.423489 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.423383    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02813cba-6a55-4181-b392-d504ad2ee4e0" containerName="main"
Sep 16 19:05:28.423489 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.423392    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="02813cba-6a55-4181-b392-d504ad2ee4e0" containerName="main"
Sep 16 19:05:28.423489 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.423474    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="02813cba-6a55-4181-b392-d504ad2ee4e0" containerName="main"
Sep 16 19:05:28.423489 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.423488    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a8b15c3-7ee9-45d1-bc6c-54981e420e88" containerName="main"
Sep 16 19:05:28.428126 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.428101    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"
Sep 16 19:05:28.431226 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.431198    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 16 19:05:28.432117 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.432094    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-6jw2t\""
Sep 16 19:05:28.432320 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.432302    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 16 19:05:28.432406 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.432311    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 16 19:05:28.432406 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.432315    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-cvks6\""
Sep 16 19:05:28.443440 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.443412    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"]
Sep 16 19:05:28.565673 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.565646    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhrb7\" (UniqueName: \"kubernetes.io/projected/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-kube-api-access-qhrb7\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl\" (UID: \"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"
Sep 16 19:05:28.565832 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.565678    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl\" (UID: \"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"
Sep 16 19:05:28.565832 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.565701    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl\" (UID: \"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"
Sep 16 19:05:28.565832 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.565717    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl\" (UID: \"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"
Sep 16 19:05:28.565832 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.565807    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl\" (UID: \"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"
Sep 16 19:05:28.566029 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.565875    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl\" (UID: \"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"
Sep 16 19:05:28.666916 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.666887    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl\" (UID: \"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"
Sep 16 19:05:28.667086 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.666929    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhrb7\" (UniqueName: \"kubernetes.io/projected/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-kube-api-access-qhrb7\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl\" (UID: \"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"
Sep 16 19:05:28.667086 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.666949    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl\" (UID: \"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"
Sep 16 19:05:28.667086 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.666972    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl\" (UID: \"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"
Sep 16 19:05:28.667086 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.666992    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl\" (UID: \"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"
Sep 16 19:05:28.667086 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.667058    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl\" (UID: \"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"
Sep 16 19:05:28.667373 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.667340    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl\" (UID: \"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"
Sep 16 19:05:28.667478 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.667434    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl\" (UID: \"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"
Sep 16 19:05:28.667478 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.667435    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl\" (UID: \"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"
Sep 16 19:05:28.669256 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.669231    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl\" (UID: \"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"
Sep 16 19:05:28.669429 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.669409    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl\" (UID: \"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"
Sep 16 19:05:28.676974 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.676954    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhrb7\" (UniqueName: \"kubernetes.io/projected/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-kube-api-access-qhrb7\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl\" (UID: \"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"
Sep 16 19:05:28.738214 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.738187    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74"]
Sep 16 19:05:28.738332 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.738317    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"
Sep 16 19:05:28.741399 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.741353    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74"
Sep 16 19:05:28.744107 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.744087    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-7klw7\""
Sep 16 19:05:28.752944 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.752925    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74"]
Sep 16 19:05:28.868709 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.868679    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmcht\" (UniqueName: \"kubernetes.io/projected/d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445-kube-api-access-nmcht\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74\" (UID: \"d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74"
Sep 16 19:05:28.868862 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.868717    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74\" (UID: \"d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74"
Sep 16 19:05:28.969398 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.969323    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmcht\" (UniqueName: \"kubernetes.io/projected/d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445-kube-api-access-nmcht\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74\" (UID: \"d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74"
Sep 16 19:05:28.969398 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.969364    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74\" (UID: \"d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74"
Sep 16 19:05:28.971697 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.971669    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74\" (UID: \"d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74"
Sep 16 19:05:28.979334 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:28.979313    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmcht\" (UniqueName: \"kubernetes.io/projected/d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445-kube-api-access-nmcht\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74\" (UID: \"d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74"
Sep 16 19:05:29.019318 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:29.019294    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"]
Sep 16 19:05:29.020751 ip-10-0-142-80 kubenswrapper[2578]: W0916 19:05:29.020730    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5edb7c_053b_46bf_8e63_bac6c16d6ac3.slice/crio-cbbadcc7fcb06c4400d85d1a9f71cd19dbab8d9734ac17d7dc1114dc90e3818c WatchSource:0}: Error finding container cbbadcc7fcb06c4400d85d1a9f71cd19dbab8d9734ac17d7dc1114dc90e3818c: Status 404 returned error can't find the container with id cbbadcc7fcb06c4400d85d1a9f71cd19dbab8d9734ac17d7dc1114dc90e3818c
Sep 16 19:05:29.068144 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:29.068121    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74"
Sep 16 19:05:29.339558 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:29.339526    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74"]
Sep 16 19:05:29.342136 ip-10-0-142-80 kubenswrapper[2578]: W0916 19:05:29.342111    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6dc0cc5_e43a_46e0_8a61_c3cf84b1c445.slice/crio-3c1d2e0a6c7803cf83878ce311725b40e541c6c49dcf82fc9d75e124bfcaaaca WatchSource:0}: Error finding container 3c1d2e0a6c7803cf83878ce311725b40e541c6c49dcf82fc9d75e124bfcaaaca: Status 404 returned error can't find the container with id 3c1d2e0a6c7803cf83878ce311725b40e541c6c49dcf82fc9d75e124bfcaaaca
Sep 16 19:05:29.594433 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:29.594341    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl" event={"ID":"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3","Type":"ContainerStarted","Data":"cbbadcc7fcb06c4400d85d1a9f71cd19dbab8d9734ac17d7dc1114dc90e3818c"}
Sep 16 19:05:29.596086 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:29.596033    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74" event={"ID":"d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445","Type":"ContainerStarted","Data":"a1998287c98ad276a4a2efc9a2564692ee6b8ad43d2ff82fd1a4a89ca79f9528"}
Sep 16 19:05:29.596217 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:29.596103    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74" event={"ID":"d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445","Type":"ContainerStarted","Data":"3c1d2e0a6c7803cf83878ce311725b40e541c6c49dcf82fc9d75e124bfcaaaca"}
Sep 16 19:05:29.596263 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:29.596209    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74"
Sep 16 19:05:29.617614 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:29.617564    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74" podStartSLOduration=1.6175454999999999 podStartE2EDuration="1.6175455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:29.617244116 +0000 UTC m=+1497.852177559" watchObservedRunningTime="2026-09-16 19:05:29.6175455 +0000 UTC m=+1497.852478944"
Sep 16 19:05:31.606743 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:31.606711    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl" event={"ID":"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3","Type":"ContainerStarted","Data":"33b18f134541392bab87031f5e6f32920fe683b3fc9ef9cf2e57d5c09bc57765"}
Sep 16 19:05:31.607174 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:31.606839    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"
Sep 16 19:05:32.613133 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:32.613033    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl" event={"ID":"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3","Type":"ContainerStarted","Data":"498806b9b2322dfe21bc616eabc3e0b75ae1238d0cc57e7c4550690ab706358d"}
Sep 16 19:05:32.634572 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:32.634528    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl" podStartSLOduration=3.06821922 podStartE2EDuration="4.634514895s" totalImagesPullingTime="1.566295675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:32.632631476 +0000 UTC m=+1500.867564917" watchObservedRunningTime="2026-09-16 19:05:32.634514895 +0000 UTC m=+1500.869448338"
Sep 16 19:05:32.739255 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:32.739228    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"
Sep 16 19:05:38.738873 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:38.738841    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"
Sep 16 19:05:38.750419 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:38.750395    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"
Sep 16 19:05:39.659344 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:39.659310    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"
Sep 16 19:05:40.665174 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:05:40.665133    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"
Sep 16 19:06:00.602556 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:00.602480    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74"
Sep 16 19:06:52.620397 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:52.620361    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"]
Sep 16 19:06:52.623168 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:52.623140    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74"]
Sep 16 19:06:52.631113 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:06:52.631079    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:06:52.631113 ip-10-0-142-80 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:06:52.631113 ip-10-0-142-80 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:06:52.631113 ip-10-0-142-80 kubenswrapper[2578]:  > containerID="498806b9b2322dfe21bc616eabc3e0b75ae1238d0cc57e7c4550690ab706358d" cmd=["/bin/sleep","15"]
Sep 16 19:06:52.631329 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:52.631135    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl" message="" err=<
Sep 16 19:06:52.631329 ip-10-0-142-80 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:06:52.631329 ip-10-0-142-80 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:06:52.631329 ip-10-0-142-80 kubenswrapper[2578]:  >
Sep 16 19:06:52.631329 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:06:52.631151    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:06:52.631329 ip-10-0-142-80 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:06:52.631329 ip-10-0-142-80 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:06:52.631329 ip-10-0-142-80 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl" podUID="7d5edb7c-053b-46bf-8e63-bac6c16d6ac3" containerName="main" containerID="cri-o://498806b9b2322dfe21bc616eabc3e0b75ae1238d0cc57e7c4550690ab706358d"
Sep 16 19:06:52.631329 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:52.631194    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl" podUID="7d5edb7c-053b-46bf-8e63-bac6c16d6ac3" containerName="main" containerID="cri-o://498806b9b2322dfe21bc616eabc3e0b75ae1238d0cc57e7c4550690ab706358d" gracePeriod=60
Sep 16 19:06:52.648346 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:52.648295    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl" podUID="7d5edb7c-053b-46bf-8e63-bac6c16d6ac3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8001/health\": dial tcp 10.133.0.49:8001: connect: connection refused"
Sep 16 19:06:52.848450 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:52.848418    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl" podUID="7d5edb7c-053b-46bf-8e63-bac6c16d6ac3" containerName="llm-d-routing-sidecar" containerID="cri-o://33b18f134541392bab87031f5e6f32920fe683b3fc9ef9cf2e57d5c09bc57765" gracePeriod=60
Sep 16 19:06:53.019944 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:53.019910    2578 generic.go:356] "Generic (PLEG): container finished" podID="7d5edb7c-053b-46bf-8e63-bac6c16d6ac3" containerID="498806b9b2322dfe21bc616eabc3e0b75ae1238d0cc57e7c4550690ab706358d" exitCode=0
Sep 16 19:06:53.019944 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:53.019940    2578 generic.go:356] "Generic (PLEG): container finished" podID="7d5edb7c-053b-46bf-8e63-bac6c16d6ac3" containerID="33b18f134541392bab87031f5e6f32920fe683b3fc9ef9cf2e57d5c09bc57765" exitCode=0
Sep 16 19:06:53.020181 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:53.019992    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl" event={"ID":"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3","Type":"ContainerDied","Data":"498806b9b2322dfe21bc616eabc3e0b75ae1238d0cc57e7c4550690ab706358d"}
Sep 16 19:06:53.020181 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:53.020081    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl" event={"ID":"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3","Type":"ContainerDied","Data":"33b18f134541392bab87031f5e6f32920fe683b3fc9ef9cf2e57d5c09bc57765"}
Sep 16 19:06:53.089902 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:53.089879    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"
Sep 16 19:06:53.241103 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:53.240996    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-tmp-dir\") pod \"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3\" (UID: \"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3\") "
Sep 16 19:06:53.241103 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:53.241060    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-dshm\") pod \"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3\" (UID: \"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3\") "
Sep 16 19:06:53.241103 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:53.241085    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-home\") pod \"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3\" (UID: \"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3\") "
Sep 16 19:06:53.241395 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:53.241130    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-tls-certs\") pod \"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3\" (UID: \"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3\") "
Sep 16 19:06:53.241395 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:53.241213    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhrb7\" (UniqueName: \"kubernetes.io/projected/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-kube-api-access-qhrb7\") pod \"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3\" (UID: \"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3\") "
Sep 16 19:06:53.241395 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:53.241240    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-model-cache\") pod \"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3\" (UID: \"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3\") "
Sep 16 19:06:53.241395 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:53.241296    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7d5edb7c-053b-46bf-8e63-bac6c16d6ac3" (UID: "7d5edb7c-053b-46bf-8e63-bac6c16d6ac3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:06:53.241395 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:53.241351    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-home" (OuterVolumeSpecName: "home") pod "7d5edb7c-053b-46bf-8e63-bac6c16d6ac3" (UID: "7d5edb7c-053b-46bf-8e63-bac6c16d6ac3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:06:53.241656 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:53.241532    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:06:53.241656 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:53.241534    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-model-cache" (OuterVolumeSpecName: "model-cache") pod "7d5edb7c-053b-46bf-8e63-bac6c16d6ac3" (UID: "7d5edb7c-053b-46bf-8e63-bac6c16d6ac3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:06:53.241656 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:53.241554    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:06:53.243390 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:53.243367    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-dshm" (OuterVolumeSpecName: "dshm") pod "7d5edb7c-053b-46bf-8e63-bac6c16d6ac3" (UID: "7d5edb7c-053b-46bf-8e63-bac6c16d6ac3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:06:53.243496 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:53.243400    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d5edb7c-053b-46bf-8e63-bac6c16d6ac3" (UID: "7d5edb7c-053b-46bf-8e63-bac6c16d6ac3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:06:53.243496 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:53.243464    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-kube-api-access-qhrb7" (OuterVolumeSpecName: "kube-api-access-qhrb7") pod "7d5edb7c-053b-46bf-8e63-bac6c16d6ac3" (UID: "7d5edb7c-053b-46bf-8e63-bac6c16d6ac3"). InnerVolumeSpecName "kube-api-access-qhrb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:06:53.343220 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:53.342913    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:06:53.343220 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:53.342953    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:06:53.343220 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:53.342969    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhrb7\" (UniqueName: \"kubernetes.io/projected/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-kube-api-access-qhrb7\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:06:53.343220 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:53.342982    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:06:54.029494 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:54.029456    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl" event={"ID":"7d5edb7c-053b-46bf-8e63-bac6c16d6ac3","Type":"ContainerDied","Data":"cbbadcc7fcb06c4400d85d1a9f71cd19dbab8d9734ac17d7dc1114dc90e3818c"}
Sep 16 19:06:54.029942 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:54.029512    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"
Sep 16 19:06:54.029942 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:54.029539    2578 scope.go:117] "RemoveContainer" containerID="498806b9b2322dfe21bc616eabc3e0b75ae1238d0cc57e7c4550690ab706358d"
Sep 16 19:06:54.039410 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:54.039391    2578 scope.go:117] "RemoveContainer" containerID="33b18f134541392bab87031f5e6f32920fe683b3fc9ef9cf2e57d5c09bc57765"
Sep 16 19:06:54.054555 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:54.054527    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"]
Sep 16 19:06:54.062277 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:54.062254    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fdvhl"]
Sep 16 19:06:54.223244 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:06:54.223206    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d5edb7c-053b-46bf-8e63-bac6c16d6ac3" path="/var/lib/kubelet/pods/7d5edb7c-053b-46bf-8e63-bac6c16d6ac3/volumes"
Sep 16 19:07:00.602909 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:00.602868    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74" podUID="d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:07:07.644301 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:07.644233    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74" podUID="d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445" containerName="main" containerID="cri-o://a1998287c98ad276a4a2efc9a2564692ee6b8ad43d2ff82fd1a4a89ca79f9528" gracePeriod=45
Sep 16 19:07:10.602759 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:10.602718    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74" podUID="d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:07:20.602317 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:20.602278    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74" podUID="d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:07:20.604521 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:20.602379    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74"
Sep 16 19:07:30.603109 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:30.603002    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74" podUID="d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:07:37.889421 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:37.889395    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74"
Sep 16 19:07:38.021329 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:38.021294    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445-tls-certs\") pod \"d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445\" (UID: \"d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445\") "
Sep 16 19:07:38.021521 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:38.021354    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmcht\" (UniqueName: \"kubernetes.io/projected/d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445-kube-api-access-nmcht\") pod \"d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445\" (UID: \"d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445\") "
Sep 16 19:07:38.023392 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:38.023348    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445" (UID: "d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:07:38.023511 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:38.023434    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445-kube-api-access-nmcht" (OuterVolumeSpecName: "kube-api-access-nmcht") pod "d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445" (UID: "d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445"). InnerVolumeSpecName "kube-api-access-nmcht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:07:38.121918 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:38.121878    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmcht\" (UniqueName: \"kubernetes.io/projected/d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445-kube-api-access-nmcht\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:07:38.121918 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:38.121913    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:07:38.253906 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:38.253877    2578 generic.go:356] "Generic (PLEG): container finished" podID="d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445" containerID="a1998287c98ad276a4a2efc9a2564692ee6b8ad43d2ff82fd1a4a89ca79f9528" exitCode=0
Sep 16 19:07:38.254091 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:38.253913    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74"
Sep 16 19:07:38.254091 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:38.253952    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74" event={"ID":"d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445","Type":"ContainerDied","Data":"a1998287c98ad276a4a2efc9a2564692ee6b8ad43d2ff82fd1a4a89ca79f9528"}
Sep 16 19:07:38.254091 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:38.254006    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74" event={"ID":"d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445","Type":"ContainerDied","Data":"3c1d2e0a6c7803cf83878ce311725b40e541c6c49dcf82fc9d75e124bfcaaaca"}
Sep 16 19:07:38.254091 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:38.254030    2578 scope.go:117] "RemoveContainer" containerID="a1998287c98ad276a4a2efc9a2564692ee6b8ad43d2ff82fd1a4a89ca79f9528"
Sep 16 19:07:38.262407 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:38.262388    2578 scope.go:117] "RemoveContainer" containerID="a1998287c98ad276a4a2efc9a2564692ee6b8ad43d2ff82fd1a4a89ca79f9528"
Sep 16 19:07:38.262667 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:07:38.262646    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1998287c98ad276a4a2efc9a2564692ee6b8ad43d2ff82fd1a4a89ca79f9528\": container with ID starting with a1998287c98ad276a4a2efc9a2564692ee6b8ad43d2ff82fd1a4a89ca79f9528 not found: ID does not exist" containerID="a1998287c98ad276a4a2efc9a2564692ee6b8ad43d2ff82fd1a4a89ca79f9528"
Sep 16 19:07:38.262727 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:38.262676    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1998287c98ad276a4a2efc9a2564692ee6b8ad43d2ff82fd1a4a89ca79f9528"} err="failed to get container status \"a1998287c98ad276a4a2efc9a2564692ee6b8ad43d2ff82fd1a4a89ca79f9528\": rpc error: code = NotFound desc = could not find container \"a1998287c98ad276a4a2efc9a2564692ee6b8ad43d2ff82fd1a4a89ca79f9528\": container with ID starting with a1998287c98ad276a4a2efc9a2564692ee6b8ad43d2ff82fd1a4a89ca79f9528 not found: ID does not exist"
Sep 16 19:07:38.277737 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:38.277680    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74"]
Sep 16 19:07:38.286982 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:38.286962    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75qkl74"]
Sep 16 19:07:40.217997 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:40.217966    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445" path="/var/lib/kubelet/pods/d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445/volumes"
Sep 16 19:07:45.332566 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.332530    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"]
Sep 16 19:07:45.333215 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.333178    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d5edb7c-053b-46bf-8e63-bac6c16d6ac3" containerName="main"
Sep 16 19:07:45.333215 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.333208    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d5edb7c-053b-46bf-8e63-bac6c16d6ac3" containerName="main"
Sep 16 19:07:45.333419 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.333223    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d5edb7c-053b-46bf-8e63-bac6c16d6ac3" containerName="llm-d-routing-sidecar"
Sep 16 19:07:45.333419 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.333233    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d5edb7c-053b-46bf-8e63-bac6c16d6ac3" containerName="llm-d-routing-sidecar"
Sep 16 19:07:45.333419 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.333256    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445" containerName="main"
Sep 16 19:07:45.333419 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.333265    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445" containerName="main"
Sep 16 19:07:45.333419 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.333353    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d5edb7c-053b-46bf-8e63-bac6c16d6ac3" containerName="main"
Sep 16 19:07:45.333419 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.333365    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6dc0cc5-e43a-46e0-8a61-c3cf84b1c445" containerName="main"
Sep 16 19:07:45.333419 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.333373    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d5edb7c-053b-46bf-8e63-bac6c16d6ac3" containerName="llm-d-routing-sidecar"
Sep 16 19:07:45.337840 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.337818    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"
Sep 16 19:07:45.341621 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.341599    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 16 19:07:45.341784 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.341760    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 16 19:07:45.341873 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.341622    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-67jn8\""
Sep 16 19:07:45.341873 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.341679    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 16 19:07:45.341873 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.341682    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-6vf94\""
Sep 16 19:07:45.351576 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.351555    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"]
Sep 16 19:07:45.486289 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.486255    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76a39cb3-c729-4007-9781-a024666cf459-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49\" (UID: \"76a39cb3-c729-4007-9781-a024666cf459\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"
Sep 16 19:07:45.486478 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.486296    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76a39cb3-c729-4007-9781-a024666cf459-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49\" (UID: \"76a39cb3-c729-4007-9781-a024666cf459\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"
Sep 16 19:07:45.486478 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.486358    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76a39cb3-c729-4007-9781-a024666cf459-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49\" (UID: \"76a39cb3-c729-4007-9781-a024666cf459\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"
Sep 16 19:07:45.486478 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.486397    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76a39cb3-c729-4007-9781-a024666cf459-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49\" (UID: \"76a39cb3-c729-4007-9781-a024666cf459\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"
Sep 16 19:07:45.486478 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.486441    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp56k\" (UniqueName: \"kubernetes.io/projected/76a39cb3-c729-4007-9781-a024666cf459-kube-api-access-tp56k\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49\" (UID: \"76a39cb3-c729-4007-9781-a024666cf459\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"
Sep 16 19:07:45.486650 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.486494    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76a39cb3-c729-4007-9781-a024666cf459-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49\" (UID: \"76a39cb3-c729-4007-9781-a024666cf459\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"
Sep 16 19:07:45.587733 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.587653    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76a39cb3-c729-4007-9781-a024666cf459-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49\" (UID: \"76a39cb3-c729-4007-9781-a024666cf459\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"
Sep 16 19:07:45.587733 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.587691    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp56k\" (UniqueName: \"kubernetes.io/projected/76a39cb3-c729-4007-9781-a024666cf459-kube-api-access-tp56k\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49\" (UID: \"76a39cb3-c729-4007-9781-a024666cf459\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"
Sep 16 19:07:45.587964 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.587742    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76a39cb3-c729-4007-9781-a024666cf459-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49\" (UID: \"76a39cb3-c729-4007-9781-a024666cf459\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"
Sep 16 19:07:45.587964 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.587790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76a39cb3-c729-4007-9781-a024666cf459-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49\" (UID: \"76a39cb3-c729-4007-9781-a024666cf459\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"
Sep 16 19:07:45.587964 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.587816    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76a39cb3-c729-4007-9781-a024666cf459-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49\" (UID: \"76a39cb3-c729-4007-9781-a024666cf459\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"
Sep 16 19:07:45.587964 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.587863    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76a39cb3-c729-4007-9781-a024666cf459-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49\" (UID: \"76a39cb3-c729-4007-9781-a024666cf459\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"
Sep 16 19:07:45.588193 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.588103    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76a39cb3-c729-4007-9781-a024666cf459-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49\" (UID: \"76a39cb3-c729-4007-9781-a024666cf459\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"
Sep 16 19:07:45.588193 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.588133    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76a39cb3-c729-4007-9781-a024666cf459-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49\" (UID: \"76a39cb3-c729-4007-9781-a024666cf459\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"
Sep 16 19:07:45.588279 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.588254    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76a39cb3-c729-4007-9781-a024666cf459-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49\" (UID: \"76a39cb3-c729-4007-9781-a024666cf459\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"
Sep 16 19:07:45.589934 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.589914    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76a39cb3-c729-4007-9781-a024666cf459-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49\" (UID: \"76a39cb3-c729-4007-9781-a024666cf459\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"
Sep 16 19:07:45.590216 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.590200    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76a39cb3-c729-4007-9781-a024666cf459-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49\" (UID: \"76a39cb3-c729-4007-9781-a024666cf459\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"
Sep 16 19:07:45.605653 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.605621    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp56k\" (UniqueName: \"kubernetes.io/projected/76a39cb3-c729-4007-9781-a024666cf459-kube-api-access-tp56k\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49\" (UID: \"76a39cb3-c729-4007-9781-a024666cf459\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"
Sep 16 19:07:45.648259 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.648227    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"
Sep 16 19:07:45.923147 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:45.923118    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"]
Sep 16 19:07:45.925206 ip-10-0-142-80 kubenswrapper[2578]: W0916 19:07:45.925167    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76a39cb3_c729_4007_9781_a024666cf459.slice/crio-5dd240550151badf55f3ec13d96923d1ca61e12be14e0ec5c629c38b6b922183 WatchSource:0}: Error finding container 5dd240550151badf55f3ec13d96923d1ca61e12be14e0ec5c629c38b6b922183: Status 404 returned error can't find the container with id 5dd240550151badf55f3ec13d96923d1ca61e12be14e0ec5c629c38b6b922183
Sep 16 19:07:46.295765 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:46.295725    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49" event={"ID":"76a39cb3-c729-4007-9781-a024666cf459","Type":"ContainerStarted","Data":"bd09910649af71d4c5e383c37ad982cb1d3ff352291f6e976fd53b8e523d53db"}
Sep 16 19:07:46.295964 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:46.295788    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49" event={"ID":"76a39cb3-c729-4007-9781-a024666cf459","Type":"ContainerStarted","Data":"5dd240550151badf55f3ec13d96923d1ca61e12be14e0ec5c629c38b6b922183"}
Sep 16 19:07:46.295964 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:46.295877    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"
Sep 16 19:07:47.302985 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:47.302951    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49" event={"ID":"76a39cb3-c729-4007-9781-a024666cf459","Type":"ContainerStarted","Data":"2ad9f1e8f90cc6d46e5d1e0920722b996543c05ae071d4687d8253dfaae70fb1"}
Sep 16 19:07:47.346792 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:47.346744    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49" podStartSLOduration=2.346727978 podStartE2EDuration="2.346727978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:47.34521041 +0000 UTC m=+1635.580143853" watchObservedRunningTime="2026-09-16 19:07:47.346727978 +0000 UTC m=+1635.581661422"
Sep 16 19:07:47.648434 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:47.648347    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"
Sep 16 19:07:55.648818 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:55.648780    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"
Sep 16 19:07:55.661721 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:55.661692    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"
Sep 16 19:07:55.667625 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:55.667598    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"
Sep 16 19:07:56.357692 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:07:56.357659    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"
Sep 16 19:08:24.065807 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:24.065773    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"]
Sep 16 19:08:24.074291 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:08:24.074249    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:08:24.074291 ip-10-0-142-80 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:08:24.074291 ip-10-0-142-80 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:08:24.074291 ip-10-0-142-80 kubenswrapper[2578]:  > containerID="2ad9f1e8f90cc6d46e5d1e0920722b996543c05ae071d4687d8253dfaae70fb1" cmd=["/bin/sleep","15"]
Sep 16 19:08:24.074549 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:24.074304    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49" message="" err=<
Sep 16 19:08:24.074549 ip-10-0-142-80 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:08:24.074549 ip-10-0-142-80 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:08:24.074549 ip-10-0-142-80 kubenswrapper[2578]:  >
Sep 16 19:08:24.074549 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:08:24.074315    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:08:24.074549 ip-10-0-142-80 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:08:24.074549 ip-10-0-142-80 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:08:24.074549 ip-10-0-142-80 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49" podUID="76a39cb3-c729-4007-9781-a024666cf459" containerName="main" containerID="cri-o://2ad9f1e8f90cc6d46e5d1e0920722b996543c05ae071d4687d8253dfaae70fb1"
Sep 16 19:08:24.074549 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:24.074346    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49" podUID="76a39cb3-c729-4007-9781-a024666cf459" containerName="main" containerID="cri-o://2ad9f1e8f90cc6d46e5d1e0920722b996543c05ae071d4687d8253dfaae70fb1" gracePeriod=60
Sep 16 19:08:24.290297 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:24.290257    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49" podUID="76a39cb3-c729-4007-9781-a024666cf459" containerName="llm-d-routing-sidecar" containerID="cri-o://bd09910649af71d4c5e383c37ad982cb1d3ff352291f6e976fd53b8e523d53db" gracePeriod=60
Sep 16 19:08:24.347343 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:24.347264    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49" podUID="76a39cb3-c729-4007-9781-a024666cf459" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8001/health\": dial tcp 10.133.0.51:8001: connect: connection refused"
Sep 16 19:08:24.488394 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:24.488364    2578 generic.go:356] "Generic (PLEG): container finished" podID="76a39cb3-c729-4007-9781-a024666cf459" containerID="2ad9f1e8f90cc6d46e5d1e0920722b996543c05ae071d4687d8253dfaae70fb1" exitCode=0
Sep 16 19:08:24.488394 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:24.488389    2578 generic.go:356] "Generic (PLEG): container finished" podID="76a39cb3-c729-4007-9781-a024666cf459" containerID="bd09910649af71d4c5e383c37ad982cb1d3ff352291f6e976fd53b8e523d53db" exitCode=0
Sep 16 19:08:24.488598 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:24.488441    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49" event={"ID":"76a39cb3-c729-4007-9781-a024666cf459","Type":"ContainerDied","Data":"2ad9f1e8f90cc6d46e5d1e0920722b996543c05ae071d4687d8253dfaae70fb1"}
Sep 16 19:08:24.488598 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:24.488535    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49" event={"ID":"76a39cb3-c729-4007-9781-a024666cf459","Type":"ContainerDied","Data":"bd09910649af71d4c5e383c37ad982cb1d3ff352291f6e976fd53b8e523d53db"}
Sep 16 19:08:24.538485 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:24.538459    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"
Sep 16 19:08:24.628366 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:24.628283    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tp56k\" (UniqueName: \"kubernetes.io/projected/76a39cb3-c729-4007-9781-a024666cf459-kube-api-access-tp56k\") pod \"76a39cb3-c729-4007-9781-a024666cf459\" (UID: \"76a39cb3-c729-4007-9781-a024666cf459\") "
Sep 16 19:08:24.628366 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:24.628319    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76a39cb3-c729-4007-9781-a024666cf459-tls-certs\") pod \"76a39cb3-c729-4007-9781-a024666cf459\" (UID: \"76a39cb3-c729-4007-9781-a024666cf459\") "
Sep 16 19:08:24.628595 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:24.628372    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76a39cb3-c729-4007-9781-a024666cf459-tmp-dir\") pod \"76a39cb3-c729-4007-9781-a024666cf459\" (UID: \"76a39cb3-c729-4007-9781-a024666cf459\") "
Sep 16 19:08:24.628595 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:24.628389    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76a39cb3-c729-4007-9781-a024666cf459-model-cache\") pod \"76a39cb3-c729-4007-9781-a024666cf459\" (UID: \"76a39cb3-c729-4007-9781-a024666cf459\") "
Sep 16 19:08:24.628595 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:24.628411    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76a39cb3-c729-4007-9781-a024666cf459-home\") pod \"76a39cb3-c729-4007-9781-a024666cf459\" (UID: \"76a39cb3-c729-4007-9781-a024666cf459\") "
Sep 16 19:08:24.628595 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:24.628444    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76a39cb3-c729-4007-9781-a024666cf459-dshm\") pod \"76a39cb3-c729-4007-9781-a024666cf459\" (UID: \"76a39cb3-c729-4007-9781-a024666cf459\") "
Sep 16 19:08:24.628778 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:24.628671    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a39cb3-c729-4007-9781-a024666cf459-model-cache" (OuterVolumeSpecName: "model-cache") pod "76a39cb3-c729-4007-9781-a024666cf459" (UID: "76a39cb3-c729-4007-9781-a024666cf459"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:24.628778 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:24.628694    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a39cb3-c729-4007-9781-a024666cf459-home" (OuterVolumeSpecName: "home") pod "76a39cb3-c729-4007-9781-a024666cf459" (UID: "76a39cb3-c729-4007-9781-a024666cf459"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:24.628778 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:24.628681    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a39cb3-c729-4007-9781-a024666cf459-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "76a39cb3-c729-4007-9781-a024666cf459" (UID: "76a39cb3-c729-4007-9781-a024666cf459"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:24.634083 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:24.631592    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76a39cb3-c729-4007-9781-a024666cf459-kube-api-access-tp56k" (OuterVolumeSpecName: "kube-api-access-tp56k") pod "76a39cb3-c729-4007-9781-a024666cf459" (UID: "76a39cb3-c729-4007-9781-a024666cf459"). InnerVolumeSpecName "kube-api-access-tp56k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:08:24.634083 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:24.632288    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76a39cb3-c729-4007-9781-a024666cf459-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76a39cb3-c729-4007-9781-a024666cf459" (UID: "76a39cb3-c729-4007-9781-a024666cf459"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:24.634083 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:24.633876    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a39cb3-c729-4007-9781-a024666cf459-dshm" (OuterVolumeSpecName: "dshm") pod "76a39cb3-c729-4007-9781-a024666cf459" (UID: "76a39cb3-c729-4007-9781-a024666cf459"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:24.729443 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:24.729407    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp56k\" (UniqueName: \"kubernetes.io/projected/76a39cb3-c729-4007-9781-a024666cf459-kube-api-access-tp56k\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:08:24.729443 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:24.729437    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76a39cb3-c729-4007-9781-a024666cf459-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:08:24.729443 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:24.729448    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76a39cb3-c729-4007-9781-a024666cf459-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:08:24.729713 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:24.729457    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76a39cb3-c729-4007-9781-a024666cf459-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:08:24.729713 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:24.729464    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76a39cb3-c729-4007-9781-a024666cf459-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:08:24.729713 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:24.729472    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76a39cb3-c729-4007-9781-a024666cf459-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:08:25.495447 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:25.495415    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"
Sep 16 19:08:25.495447 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:25.495437    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49" event={"ID":"76a39cb3-c729-4007-9781-a024666cf459","Type":"ContainerDied","Data":"5dd240550151badf55f3ec13d96923d1ca61e12be14e0ec5c629c38b6b922183"}
Sep 16 19:08:25.496003 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:25.495506    2578 scope.go:117] "RemoveContainer" containerID="2ad9f1e8f90cc6d46e5d1e0920722b996543c05ae071d4687d8253dfaae70fb1"
Sep 16 19:08:25.505139 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:25.505125    2578 scope.go:117] "RemoveContainer" containerID="bd09910649af71d4c5e383c37ad982cb1d3ff352291f6e976fd53b8e523d53db"
Sep 16 19:08:25.525673 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:25.525646    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"]
Sep 16 19:08:25.530835 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:25.530813    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-gkj49"]
Sep 16 19:08:26.216748 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:26.216712    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76a39cb3-c729-4007-9781-a024666cf459" path="/var/lib/kubelet/pods/76a39cb3-c729-4007-9781-a024666cf459/volumes"
Sep 16 19:08:56.051886 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.051807    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4"]
Sep 16 19:08:56.052554 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.052298    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76a39cb3-c729-4007-9781-a024666cf459" containerName="llm-d-routing-sidecar"
Sep 16 19:08:56.052554 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.052324    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="76a39cb3-c729-4007-9781-a024666cf459" containerName="llm-d-routing-sidecar"
Sep 16 19:08:56.052554 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.052352    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76a39cb3-c729-4007-9781-a024666cf459" containerName="main"
Sep 16 19:08:56.052554 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.052362    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="76a39cb3-c729-4007-9781-a024666cf459" containerName="main"
Sep 16 19:08:56.052554 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.052477    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="76a39cb3-c729-4007-9781-a024666cf459" containerName="main"
Sep 16 19:08:56.052554 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.052494    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="76a39cb3-c729-4007-9781-a024666cf459" containerName="llm-d-routing-sidecar"
Sep 16 19:08:56.055672 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.055654    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4"
Sep 16 19:08:56.059776 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.059747    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"kube-root-ca.crt\""
Sep 16 19:08:56.059896 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.059817    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"default-dockercfg-zvx5r\""
Sep 16 19:08:56.059896 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.059817    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v1-kserve-self-signed-certs\""
Sep 16 19:08:56.059896 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.059821    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"openshift-service-ca.crt\""
Sep 16 19:08:56.071609 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.071576    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4"]
Sep 16 19:08:56.090399 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.090376    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac186c57-b86f-40c9-9e7b-3aefce63817e-home\") pod \"test-div-v1-kserve-fb566d667-bb9g4\" (UID: \"ac186c57-b86f-40c9-9e7b-3aefce63817e\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4"
Sep 16 19:08:56.090526 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.090404    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac186c57-b86f-40c9-9e7b-3aefce63817e-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-bb9g4\" (UID: \"ac186c57-b86f-40c9-9e7b-3aefce63817e\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4"
Sep 16 19:08:56.090526 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.090428    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac186c57-b86f-40c9-9e7b-3aefce63817e-model-cache\") pod \"test-div-v1-kserve-fb566d667-bb9g4\" (UID: \"ac186c57-b86f-40c9-9e7b-3aefce63817e\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4"
Sep 16 19:08:56.090526 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.090446    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac186c57-b86f-40c9-9e7b-3aefce63817e-tls-certs\") pod \"test-div-v1-kserve-fb566d667-bb9g4\" (UID: \"ac186c57-b86f-40c9-9e7b-3aefce63817e\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4"
Sep 16 19:08:56.090526 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.090464    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz99t\" (UniqueName: \"kubernetes.io/projected/ac186c57-b86f-40c9-9e7b-3aefce63817e-kube-api-access-hz99t\") pod \"test-div-v1-kserve-fb566d667-bb9g4\" (UID: \"ac186c57-b86f-40c9-9e7b-3aefce63817e\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4"
Sep 16 19:08:56.090692 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.090579    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac186c57-b86f-40c9-9e7b-3aefce63817e-dshm\") pod \"test-div-v1-kserve-fb566d667-bb9g4\" (UID: \"ac186c57-b86f-40c9-9e7b-3aefce63817e\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4"
Sep 16 19:08:56.191278 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.191237    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac186c57-b86f-40c9-9e7b-3aefce63817e-dshm\") pod \"test-div-v1-kserve-fb566d667-bb9g4\" (UID: \"ac186c57-b86f-40c9-9e7b-3aefce63817e\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4"
Sep 16 19:08:56.191497 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.191293    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac186c57-b86f-40c9-9e7b-3aefce63817e-home\") pod \"test-div-v1-kserve-fb566d667-bb9g4\" (UID: \"ac186c57-b86f-40c9-9e7b-3aefce63817e\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4"
Sep 16 19:08:56.191497 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.191316    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac186c57-b86f-40c9-9e7b-3aefce63817e-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-bb9g4\" (UID: \"ac186c57-b86f-40c9-9e7b-3aefce63817e\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4"
Sep 16 19:08:56.191497 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.191351    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac186c57-b86f-40c9-9e7b-3aefce63817e-model-cache\") pod \"test-div-v1-kserve-fb566d667-bb9g4\" (UID: \"ac186c57-b86f-40c9-9e7b-3aefce63817e\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4"
Sep 16 19:08:56.191497 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.191389    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac186c57-b86f-40c9-9e7b-3aefce63817e-tls-certs\") pod \"test-div-v1-kserve-fb566d667-bb9g4\" (UID: \"ac186c57-b86f-40c9-9e7b-3aefce63817e\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4"
Sep 16 19:08:56.191497 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.191422    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz99t\" (UniqueName: \"kubernetes.io/projected/ac186c57-b86f-40c9-9e7b-3aefce63817e-kube-api-access-hz99t\") pod \"test-div-v1-kserve-fb566d667-bb9g4\" (UID: \"ac186c57-b86f-40c9-9e7b-3aefce63817e\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4"
Sep 16 19:08:56.191786 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.191758    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac186c57-b86f-40c9-9e7b-3aefce63817e-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-bb9g4\" (UID: \"ac186c57-b86f-40c9-9e7b-3aefce63817e\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4"
Sep 16 19:08:56.191854 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.191803    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac186c57-b86f-40c9-9e7b-3aefce63817e-home\") pod \"test-div-v1-kserve-fb566d667-bb9g4\" (UID: \"ac186c57-b86f-40c9-9e7b-3aefce63817e\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4"
Sep 16 19:08:56.191897 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.191880    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac186c57-b86f-40c9-9e7b-3aefce63817e-model-cache\") pod \"test-div-v1-kserve-fb566d667-bb9g4\" (UID: \"ac186c57-b86f-40c9-9e7b-3aefce63817e\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4"
Sep 16 19:08:56.193656 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.193620    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac186c57-b86f-40c9-9e7b-3aefce63817e-dshm\") pod \"test-div-v1-kserve-fb566d667-bb9g4\" (UID: \"ac186c57-b86f-40c9-9e7b-3aefce63817e\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4"
Sep 16 19:08:56.193840 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.193805    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac186c57-b86f-40c9-9e7b-3aefce63817e-tls-certs\") pod \"test-div-v1-kserve-fb566d667-bb9g4\" (UID: \"ac186c57-b86f-40c9-9e7b-3aefce63817e\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4"
Sep 16 19:08:56.201817 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.201792    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz99t\" (UniqueName: \"kubernetes.io/projected/ac186c57-b86f-40c9-9e7b-3aefce63817e-kube-api-access-hz99t\") pod \"test-div-v1-kserve-fb566d667-bb9g4\" (UID: \"ac186c57-b86f-40c9-9e7b-3aefce63817e\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4"
Sep 16 19:08:56.366702 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.366614    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4"
Sep 16 19:08:56.646854 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.646818    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4"]
Sep 16 19:08:56.648093 ip-10-0-142-80 kubenswrapper[2578]: W0916 19:08:56.648057    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac186c57_b86f_40c9_9e7b_3aefce63817e.slice/crio-eb74260b9c51c2897cdb12c5dce7f8abe88d4082f95399313ccda00d44094f90 WatchSource:0}: Error finding container eb74260b9c51c2897cdb12c5dce7f8abe88d4082f95399313ccda00d44094f90: Status 404 returned error can't find the container with id eb74260b9c51c2897cdb12c5dce7f8abe88d4082f95399313ccda00d44094f90
Sep 16 19:08:56.650551 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:56.650531    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:08:57.664317 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:57.664286    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4" event={"ID":"ac186c57-b86f-40c9-9e7b-3aefce63817e","Type":"ContainerStarted","Data":"eb74260b9c51c2897cdb12c5dce7f8abe88d4082f95399313ccda00d44094f90"}
Sep 16 19:08:58.670799 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:58.670759    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4" event={"ID":"ac186c57-b86f-40c9-9e7b-3aefce63817e","Type":"ContainerStarted","Data":"81f0a815308bbee3990623f976e1e4ef9617a9527d2882224ed51503c9323ec6"}
Sep 16 19:08:58.710528 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:58.710476    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4" podStartSLOduration=1.7243316800000001 podStartE2EDuration="2.710461311s" totalImagesPullingTime="986.129631ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:08:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:58.70855131 +0000 UTC m=+1706.943484752" watchObservedRunningTime="2026-09-16 19:08:58.710461311 +0000 UTC m=+1706.945394754"
Sep 16 19:08:59.367749 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:08:59.367713    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4"
Sep 16 19:09:06.367357 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:06.367317    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4"
Sep 16 19:09:06.379445 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:06.379420    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4"
Sep 16 19:09:06.719711 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:06.719677    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4"
Sep 16 19:09:38.096218 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.096169    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"]
Sep 16 19:09:38.099535 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.099516    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"
Sep 16 19:09:38.103296 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.103272    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 16 19:09:38.103296 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.103286    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 16 19:09:38.103490 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.103324    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 16 19:09:38.103490 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.103338    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 16 19:09:38.103490 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.103281    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:09:38.103490 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.103280    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-zjzwv\""
Sep 16 19:09:38.116242 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.116219    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"]
Sep 16 19:09:38.181614 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.181579    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/615ce087-755a-4ece-a039-d4a5872a5521-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf\" (UID: \"615ce087-755a-4ece-a039-d4a5872a5521\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"
Sep 16 19:09:38.181796 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.181631    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/615ce087-755a-4ece-a039-d4a5872a5521-home\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf\" (UID: \"615ce087-755a-4ece-a039-d4a5872a5521\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"
Sep 16 19:09:38.181796 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.181684    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/615ce087-755a-4ece-a039-d4a5872a5521-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf\" (UID: \"615ce087-755a-4ece-a039-d4a5872a5521\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"
Sep 16 19:09:38.181796 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.181701    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/615ce087-755a-4ece-a039-d4a5872a5521-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf\" (UID: \"615ce087-755a-4ece-a039-d4a5872a5521\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"
Sep 16 19:09:38.181796 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.181769    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/615ce087-755a-4ece-a039-d4a5872a5521-dshm\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf\" (UID: \"615ce087-755a-4ece-a039-d4a5872a5521\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"
Sep 16 19:09:38.181975 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.181839    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/615ce087-755a-4ece-a039-d4a5872a5521-model-cache\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf\" (UID: \"615ce087-755a-4ece-a039-d4a5872a5521\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"
Sep 16 19:09:38.181975 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.181876    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/615ce087-755a-4ece-a039-d4a5872a5521-tls-certs\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf\" (UID: \"615ce087-755a-4ece-a039-d4a5872a5521\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"
Sep 16 19:09:38.181975 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.181906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6wcf\" (UniqueName: \"kubernetes.io/projected/615ce087-755a-4ece-a039-d4a5872a5521-kube-api-access-q6wcf\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf\" (UID: \"615ce087-755a-4ece-a039-d4a5872a5521\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"
Sep 16 19:09:38.283051 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.283013    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/615ce087-755a-4ece-a039-d4a5872a5521-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf\" (UID: \"615ce087-755a-4ece-a039-d4a5872a5521\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"
Sep 16 19:09:38.283275 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.283081    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/615ce087-755a-4ece-a039-d4a5872a5521-home\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf\" (UID: \"615ce087-755a-4ece-a039-d4a5872a5521\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"
Sep 16 19:09:38.283275 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.283115    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/615ce087-755a-4ece-a039-d4a5872a5521-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf\" (UID: \"615ce087-755a-4ece-a039-d4a5872a5521\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"
Sep 16 19:09:38.283275 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.283131    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/615ce087-755a-4ece-a039-d4a5872a5521-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf\" (UID: \"615ce087-755a-4ece-a039-d4a5872a5521\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"
Sep 16 19:09:38.283275 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.283173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/615ce087-755a-4ece-a039-d4a5872a5521-dshm\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf\" (UID: \"615ce087-755a-4ece-a039-d4a5872a5521\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"
Sep 16 19:09:38.283275 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.283213    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/615ce087-755a-4ece-a039-d4a5872a5521-model-cache\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf\" (UID: \"615ce087-755a-4ece-a039-d4a5872a5521\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"
Sep 16 19:09:38.283275 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.283239    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/615ce087-755a-4ece-a039-d4a5872a5521-tls-certs\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf\" (UID: \"615ce087-755a-4ece-a039-d4a5872a5521\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"
Sep 16 19:09:38.283275 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.283269    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6wcf\" (UniqueName: \"kubernetes.io/projected/615ce087-755a-4ece-a039-d4a5872a5521-kube-api-access-q6wcf\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf\" (UID: \"615ce087-755a-4ece-a039-d4a5872a5521\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"
Sep 16 19:09:38.283685 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.283581    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/615ce087-755a-4ece-a039-d4a5872a5521-home\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf\" (UID: \"615ce087-755a-4ece-a039-d4a5872a5521\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"
Sep 16 19:09:38.283685 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.283582    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/615ce087-755a-4ece-a039-d4a5872a5521-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf\" (UID: \"615ce087-755a-4ece-a039-d4a5872a5521\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"
Sep 16 19:09:38.284054 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.284009    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/615ce087-755a-4ece-a039-d4a5872a5521-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf\" (UID: \"615ce087-755a-4ece-a039-d4a5872a5521\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"
Sep 16 19:09:38.284180 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.284070    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/615ce087-755a-4ece-a039-d4a5872a5521-model-cache\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf\" (UID: \"615ce087-755a-4ece-a039-d4a5872a5521\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"
Sep 16 19:09:38.284180 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.284109    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/615ce087-755a-4ece-a039-d4a5872a5521-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf\" (UID: \"615ce087-755a-4ece-a039-d4a5872a5521\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"
Sep 16 19:09:38.285744 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.285724    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/615ce087-755a-4ece-a039-d4a5872a5521-dshm\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf\" (UID: \"615ce087-755a-4ece-a039-d4a5872a5521\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"
Sep 16 19:09:38.286491 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.286464    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/615ce087-755a-4ece-a039-d4a5872a5521-tls-certs\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf\" (UID: \"615ce087-755a-4ece-a039-d4a5872a5521\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"
Sep 16 19:09:38.293423 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.293402    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6wcf\" (UniqueName: \"kubernetes.io/projected/615ce087-755a-4ece-a039-d4a5872a5521-kube-api-access-q6wcf\") pod \"tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf\" (UID: \"615ce087-755a-4ece-a039-d4a5872a5521\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"
Sep 16 19:09:38.409292 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.409205    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"
Sep 16 19:09:38.693866 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.693839    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"]
Sep 16 19:09:38.696462 ip-10-0-142-80 kubenswrapper[2578]: W0916 19:09:38.696429    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod615ce087_755a_4ece_a039_d4a5872a5521.slice/crio-c2faae5324e3dbe5e9827287e493a2882076ab65d499f5c662ed53215d5dfebe WatchSource:0}: Error finding container c2faae5324e3dbe5e9827287e493a2882076ab65d499f5c662ed53215d5dfebe: Status 404 returned error can't find the container with id c2faae5324e3dbe5e9827287e493a2882076ab65d499f5c662ed53215d5dfebe
Sep 16 19:09:38.871524 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.871489    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf" event={"ID":"615ce087-755a-4ece-a039-d4a5872a5521","Type":"ContainerStarted","Data":"4c47cf340e5b05abded4a7be97f125c8baa605cbe766a0aca80a1287daea588b"}
Sep 16 19:09:38.871711 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:38.871545    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf" event={"ID":"615ce087-755a-4ece-a039-d4a5872a5521","Type":"ContainerStarted","Data":"c2faae5324e3dbe5e9827287e493a2882076ab65d499f5c662ed53215d5dfebe"}
Sep 16 19:09:41.890866 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:41.890826    2578 generic.go:356] "Generic (PLEG): container finished" podID="615ce087-755a-4ece-a039-d4a5872a5521" containerID="4c47cf340e5b05abded4a7be97f125c8baa605cbe766a0aca80a1287daea588b" exitCode=0
Sep 16 19:09:41.891294 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:41.890903    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf" event={"ID":"615ce087-755a-4ece-a039-d4a5872a5521","Type":"ContainerDied","Data":"4c47cf340e5b05abded4a7be97f125c8baa605cbe766a0aca80a1287daea588b"}
Sep 16 19:09:42.898459 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:42.898426    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf" event={"ID":"615ce087-755a-4ece-a039-d4a5872a5521","Type":"ContainerStarted","Data":"85d75e7af1bab84fefecdca9b3ffbaf5f05d57a389a557cac0a1fb2148db4dac"}
Sep 16 19:09:42.921531 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:42.921481    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf" podStartSLOduration=2.638819986 podStartE2EDuration="4.921465544s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.282645558s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:09:42.919924004 +0000 UTC m=+1751.154857446" watchObservedRunningTime="2026-09-16 19:09:42.921465544 +0000 UTC m=+1751.156399041"
Sep 16 19:09:43.409796 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:43.409762    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"
Sep 16 19:09:47.356428 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:47.356387    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4"]
Sep 16 19:09:48.410296 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:48.410244    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"
Sep 16 19:09:48.422534 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:48.422509    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"
Sep 16 19:09:48.944077 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:09:48.944032    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"
Sep 16 19:10:02.375232 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:02.375170    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4" podUID="ac186c57-b86f-40c9-9e7b-3aefce63817e" containerName="main" containerID="cri-o://81f0a815308bbee3990623f976e1e4ef9617a9527d2882224ed51503c9323ec6" gracePeriod=45
Sep 16 19:10:02.612361 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:02.612335    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4"
Sep 16 19:10:02.695207 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:02.695177    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac186c57-b86f-40c9-9e7b-3aefce63817e-tmp-dir\") pod \"ac186c57-b86f-40c9-9e7b-3aefce63817e\" (UID: \"ac186c57-b86f-40c9-9e7b-3aefce63817e\") "
Sep 16 19:10:02.695207 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:02.695208    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac186c57-b86f-40c9-9e7b-3aefce63817e-model-cache\") pod \"ac186c57-b86f-40c9-9e7b-3aefce63817e\" (UID: \"ac186c57-b86f-40c9-9e7b-3aefce63817e\") "
Sep 16 19:10:02.695452 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:02.695229    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac186c57-b86f-40c9-9e7b-3aefce63817e-tls-certs\") pod \"ac186c57-b86f-40c9-9e7b-3aefce63817e\" (UID: \"ac186c57-b86f-40c9-9e7b-3aefce63817e\") "
Sep 16 19:10:02.695452 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:02.695284    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac186c57-b86f-40c9-9e7b-3aefce63817e-home\") pod \"ac186c57-b86f-40c9-9e7b-3aefce63817e\" (UID: \"ac186c57-b86f-40c9-9e7b-3aefce63817e\") "
Sep 16 19:10:02.695452 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:02.695321    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac186c57-b86f-40c9-9e7b-3aefce63817e-dshm\") pod \"ac186c57-b86f-40c9-9e7b-3aefce63817e\" (UID: \"ac186c57-b86f-40c9-9e7b-3aefce63817e\") "
Sep 16 19:10:02.695452 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:02.695343    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hz99t\" (UniqueName: \"kubernetes.io/projected/ac186c57-b86f-40c9-9e7b-3aefce63817e-kube-api-access-hz99t\") pod \"ac186c57-b86f-40c9-9e7b-3aefce63817e\" (UID: \"ac186c57-b86f-40c9-9e7b-3aefce63817e\") "
Sep 16 19:10:02.695658 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:02.695519    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac186c57-b86f-40c9-9e7b-3aefce63817e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ac186c57-b86f-40c9-9e7b-3aefce63817e" (UID: "ac186c57-b86f-40c9-9e7b-3aefce63817e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:02.695658 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:02.695525    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac186c57-b86f-40c9-9e7b-3aefce63817e-model-cache" (OuterVolumeSpecName: "model-cache") pod "ac186c57-b86f-40c9-9e7b-3aefce63817e" (UID: "ac186c57-b86f-40c9-9e7b-3aefce63817e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:02.695658 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:02.695614    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac186c57-b86f-40c9-9e7b-3aefce63817e-home" (OuterVolumeSpecName: "home") pod "ac186c57-b86f-40c9-9e7b-3aefce63817e" (UID: "ac186c57-b86f-40c9-9e7b-3aefce63817e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:02.695786 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:02.695675    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac186c57-b86f-40c9-9e7b-3aefce63817e-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:10:02.695786 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:02.695693    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac186c57-b86f-40c9-9e7b-3aefce63817e-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:10:02.695786 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:02.695701    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac186c57-b86f-40c9-9e7b-3aefce63817e-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:10:02.697554 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:02.697525    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac186c57-b86f-40c9-9e7b-3aefce63817e-dshm" (OuterVolumeSpecName: "dshm") pod "ac186c57-b86f-40c9-9e7b-3aefce63817e" (UID: "ac186c57-b86f-40c9-9e7b-3aefce63817e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:02.697663 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:02.697561    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac186c57-b86f-40c9-9e7b-3aefce63817e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ac186c57-b86f-40c9-9e7b-3aefce63817e" (UID: "ac186c57-b86f-40c9-9e7b-3aefce63817e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:10:02.697663 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:02.697583    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac186c57-b86f-40c9-9e7b-3aefce63817e-kube-api-access-hz99t" (OuterVolumeSpecName: "kube-api-access-hz99t") pod "ac186c57-b86f-40c9-9e7b-3aefce63817e" (UID: "ac186c57-b86f-40c9-9e7b-3aefce63817e"). InnerVolumeSpecName "kube-api-access-hz99t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:10:02.796839 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:02.796808    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac186c57-b86f-40c9-9e7b-3aefce63817e-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:10:02.796839 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:02.796833    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz99t\" (UniqueName: \"kubernetes.io/projected/ac186c57-b86f-40c9-9e7b-3aefce63817e-kube-api-access-hz99t\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:10:02.796839 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:02.796843    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac186c57-b86f-40c9-9e7b-3aefce63817e-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:10:03.010104 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:03.009990    2578 generic.go:356] "Generic (PLEG): container finished" podID="ac186c57-b86f-40c9-9e7b-3aefce63817e" containerID="81f0a815308bbee3990623f976e1e4ef9617a9527d2882224ed51503c9323ec6" exitCode=0
Sep 16 19:10:03.010104 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:03.010071    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4"
Sep 16 19:10:03.010104 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:03.010082    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4" event={"ID":"ac186c57-b86f-40c9-9e7b-3aefce63817e","Type":"ContainerDied","Data":"81f0a815308bbee3990623f976e1e4ef9617a9527d2882224ed51503c9323ec6"}
Sep 16 19:10:03.010365 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:03.010171    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4" event={"ID":"ac186c57-b86f-40c9-9e7b-3aefce63817e","Type":"ContainerDied","Data":"eb74260b9c51c2897cdb12c5dce7f8abe88d4082f95399313ccda00d44094f90"}
Sep 16 19:10:03.010365 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:03.010195    2578 scope.go:117] "RemoveContainer" containerID="81f0a815308bbee3990623f976e1e4ef9617a9527d2882224ed51503c9323ec6"
Sep 16 19:10:03.019635 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:03.019614    2578 scope.go:117] "RemoveContainer" containerID="81f0a815308bbee3990623f976e1e4ef9617a9527d2882224ed51503c9323ec6"
Sep 16 19:10:03.019892 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:10:03.019874    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81f0a815308bbee3990623f976e1e4ef9617a9527d2882224ed51503c9323ec6\": container with ID starting with 81f0a815308bbee3990623f976e1e4ef9617a9527d2882224ed51503c9323ec6 not found: ID does not exist" containerID="81f0a815308bbee3990623f976e1e4ef9617a9527d2882224ed51503c9323ec6"
Sep 16 19:10:03.019951 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:03.019904    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81f0a815308bbee3990623f976e1e4ef9617a9527d2882224ed51503c9323ec6"} err="failed to get container status \"81f0a815308bbee3990623f976e1e4ef9617a9527d2882224ed51503c9323ec6\": rpc error: code = NotFound desc = could not find container \"81f0a815308bbee3990623f976e1e4ef9617a9527d2882224ed51503c9323ec6\": container with ID starting with 81f0a815308bbee3990623f976e1e4ef9617a9527d2882224ed51503c9323ec6 not found: ID does not exist"
Sep 16 19:10:03.035647 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:03.035620    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4"]
Sep 16 19:10:03.042461 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:03.042440    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-bb9g4"]
Sep 16 19:10:04.217515 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:04.217478    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac186c57-b86f-40c9-9e7b-3aefce63817e" path="/var/lib/kubelet/pods/ac186c57-b86f-40c9-9e7b-3aefce63817e/volumes"
Sep 16 19:10:27.730871 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:27.730783    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx"]
Sep 16 19:10:27.731473 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:27.731450    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac186c57-b86f-40c9-9e7b-3aefce63817e" containerName="main"
Sep 16 19:10:27.731473 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:27.731475    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac186c57-b86f-40c9-9e7b-3aefce63817e" containerName="main"
Sep 16 19:10:27.731618 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:27.731566    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac186c57-b86f-40c9-9e7b-3aefce63817e" containerName="main"
Sep 16 19:10:27.734552 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:27.734529    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx"
Sep 16 19:10:27.737360 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:27.737333    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"kube-root-ca.crt\""
Sep 16 19:10:27.738545 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:27.738524    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v1-kserve-self-signed-certs\""
Sep 16 19:10:27.738672 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:27.738651    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"default-dockercfg-z9kpq\""
Sep 16 19:10:27.738723 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:27.738684    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"openshift-service-ca.crt\""
Sep 16 19:10:27.747103 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:27.747081    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx"]
Sep 16 19:10:27.819692 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:27.819651    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-tls-certs\") pod \"test-div-v1-kserve-fb566d667-tzskx\" (UID: \"54b7b3dc-47f3-44c6-b7a7-33593d3eb441\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx"
Sep 16 19:10:27.819869 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:27.819704    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-tzskx\" (UID: \"54b7b3dc-47f3-44c6-b7a7-33593d3eb441\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx"
Sep 16 19:10:27.819869 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:27.819734    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-model-cache\") pod \"test-div-v1-kserve-fb566d667-tzskx\" (UID: \"54b7b3dc-47f3-44c6-b7a7-33593d3eb441\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx"
Sep 16 19:10:27.819869 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:27.819778    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-dshm\") pod \"test-div-v1-kserve-fb566d667-tzskx\" (UID: \"54b7b3dc-47f3-44c6-b7a7-33593d3eb441\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx"
Sep 16 19:10:27.819869 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:27.819794    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-home\") pod \"test-div-v1-kserve-fb566d667-tzskx\" (UID: \"54b7b3dc-47f3-44c6-b7a7-33593d3eb441\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx"
Sep 16 19:10:27.819869 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:27.819810    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7m25\" (UniqueName: \"kubernetes.io/projected/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-kube-api-access-q7m25\") pod \"test-div-v1-kserve-fb566d667-tzskx\" (UID: \"54b7b3dc-47f3-44c6-b7a7-33593d3eb441\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx"
Sep 16 19:10:27.920937 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:27.920898    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-tls-certs\") pod \"test-div-v1-kserve-fb566d667-tzskx\" (UID: \"54b7b3dc-47f3-44c6-b7a7-33593d3eb441\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx"
Sep 16 19:10:27.921165 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:27.920951    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-tzskx\" (UID: \"54b7b3dc-47f3-44c6-b7a7-33593d3eb441\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx"
Sep 16 19:10:27.921165 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:27.920984    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-model-cache\") pod \"test-div-v1-kserve-fb566d667-tzskx\" (UID: \"54b7b3dc-47f3-44c6-b7a7-33593d3eb441\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx"
Sep 16 19:10:27.921165 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:27.921001    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-dshm\") pod \"test-div-v1-kserve-fb566d667-tzskx\" (UID: \"54b7b3dc-47f3-44c6-b7a7-33593d3eb441\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx"
Sep 16 19:10:27.921165 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:27.921017    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-home\") pod \"test-div-v1-kserve-fb566d667-tzskx\" (UID: \"54b7b3dc-47f3-44c6-b7a7-33593d3eb441\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx"
Sep 16 19:10:27.921165 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:27.921067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7m25\" (UniqueName: \"kubernetes.io/projected/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-kube-api-access-q7m25\") pod \"test-div-v1-kserve-fb566d667-tzskx\" (UID: \"54b7b3dc-47f3-44c6-b7a7-33593d3eb441\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx"
Sep 16 19:10:27.921486 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:27.921462    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-model-cache\") pod \"test-div-v1-kserve-fb566d667-tzskx\" (UID: \"54b7b3dc-47f3-44c6-b7a7-33593d3eb441\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx"
Sep 16 19:10:27.921547 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:27.921529    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-home\") pod \"test-div-v1-kserve-fb566d667-tzskx\" (UID: \"54b7b3dc-47f3-44c6-b7a7-33593d3eb441\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx"
Sep 16 19:10:27.921583 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:27.921549    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-tzskx\" (UID: \"54b7b3dc-47f3-44c6-b7a7-33593d3eb441\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx"
Sep 16 19:10:27.923292 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:27.923267    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-dshm\") pod \"test-div-v1-kserve-fb566d667-tzskx\" (UID: \"54b7b3dc-47f3-44c6-b7a7-33593d3eb441\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx"
Sep 16 19:10:27.923411 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:27.923390    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-tls-certs\") pod \"test-div-v1-kserve-fb566d667-tzskx\" (UID: \"54b7b3dc-47f3-44c6-b7a7-33593d3eb441\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx"
Sep 16 19:10:27.933717 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:27.933689    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7m25\" (UniqueName: \"kubernetes.io/projected/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-kube-api-access-q7m25\") pod \"test-div-v1-kserve-fb566d667-tzskx\" (UID: \"54b7b3dc-47f3-44c6-b7a7-33593d3eb441\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx"
Sep 16 19:10:28.044936 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:28.044833    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx"
Sep 16 19:10:28.326245 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:28.325530    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx"]
Sep 16 19:10:28.329859 ip-10-0-142-80 kubenswrapper[2578]: W0916 19:10:28.329827    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b7b3dc_47f3_44c6_b7a7_33593d3eb441.slice/crio-271aafff05163829984d3a8975e519ea3ecf2fa09d0ca55778698002041ed995 WatchSource:0}: Error finding container 271aafff05163829984d3a8975e519ea3ecf2fa09d0ca55778698002041ed995: Status 404 returned error can't find the container with id 271aafff05163829984d3a8975e519ea3ecf2fa09d0ca55778698002041ed995
Sep 16 19:10:29.143034 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:29.142992    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx" event={"ID":"54b7b3dc-47f3-44c6-b7a7-33593d3eb441","Type":"ContainerStarted","Data":"1d5e316360762cf87182a178e11fd3e57301a8588c45a33e4cb52d359e4a7b0d"}
Sep 16 19:10:29.143466 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:29.143079    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx" event={"ID":"54b7b3dc-47f3-44c6-b7a7-33593d3eb441","Type":"ContainerStarted","Data":"271aafff05163829984d3a8975e519ea3ecf2fa09d0ca55778698002041ed995"}
Sep 16 19:10:29.162927 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:29.162873    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx" podStartSLOduration=2.162857811 podStartE2EDuration="2.162857811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:29.162418741 +0000 UTC m=+1797.397352184" watchObservedRunningTime="2026-09-16 19:10:29.162857811 +0000 UTC m=+1797.397791253"
Sep 16 19:10:30.045742 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:30.045705    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx"
Sep 16 19:10:38.045970 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:38.045933    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx"
Sep 16 19:10:38.058072 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:38.058028    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx"
Sep 16 19:10:38.196782 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:38.196753    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx"
Sep 16 19:10:43.355458 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:43.355426    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk"]
Sep 16 19:10:43.358526 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:43.358504    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk"
Sep 16 19:10:43.361242 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:43.361220    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v2-kserve-self-signed-certs\""
Sep 16 19:10:43.374439 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:43.374415    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk"]
Sep 16 19:10:43.454900 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:43.454865    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/808f3dd5-6179-4e46-91dc-23c8fde13111-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-gfjrk\" (UID: \"808f3dd5-6179-4e46-91dc-23c8fde13111\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk"
Sep 16 19:10:43.454900 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:43.454905    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/808f3dd5-6179-4e46-91dc-23c8fde13111-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-gfjrk\" (UID: \"808f3dd5-6179-4e46-91dc-23c8fde13111\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk"
Sep 16 19:10:43.455148 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:43.454975    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/808f3dd5-6179-4e46-91dc-23c8fde13111-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-gfjrk\" (UID: \"808f3dd5-6179-4e46-91dc-23c8fde13111\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk"
Sep 16 19:10:43.455148 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:43.455058    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk42z\" (UniqueName: \"kubernetes.io/projected/808f3dd5-6179-4e46-91dc-23c8fde13111-kube-api-access-rk42z\") pod \"test-div-v2-kserve-5f6fdb6b77-gfjrk\" (UID: \"808f3dd5-6179-4e46-91dc-23c8fde13111\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk"
Sep 16 19:10:43.455148 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:43.455107    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/808f3dd5-6179-4e46-91dc-23c8fde13111-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-gfjrk\" (UID: \"808f3dd5-6179-4e46-91dc-23c8fde13111\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk"
Sep 16 19:10:43.455148 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:43.455132    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/808f3dd5-6179-4e46-91dc-23c8fde13111-home\") pod \"test-div-v2-kserve-5f6fdb6b77-gfjrk\" (UID: \"808f3dd5-6179-4e46-91dc-23c8fde13111\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk"
Sep 16 19:10:43.555594 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:43.555559    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/808f3dd5-6179-4e46-91dc-23c8fde13111-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-gfjrk\" (UID: \"808f3dd5-6179-4e46-91dc-23c8fde13111\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk"
Sep 16 19:10:43.555594 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:43.555600    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/808f3dd5-6179-4e46-91dc-23c8fde13111-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-gfjrk\" (UID: \"808f3dd5-6179-4e46-91dc-23c8fde13111\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk"
Sep 16 19:10:43.555838 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:43.555621    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/808f3dd5-6179-4e46-91dc-23c8fde13111-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-gfjrk\" (UID: \"808f3dd5-6179-4e46-91dc-23c8fde13111\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk"
Sep 16 19:10:43.555838 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:43.555651    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk42z\" (UniqueName: \"kubernetes.io/projected/808f3dd5-6179-4e46-91dc-23c8fde13111-kube-api-access-rk42z\") pod \"test-div-v2-kserve-5f6fdb6b77-gfjrk\" (UID: \"808f3dd5-6179-4e46-91dc-23c8fde13111\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk"
Sep 16 19:10:43.555838 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:43.555696    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/808f3dd5-6179-4e46-91dc-23c8fde13111-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-gfjrk\" (UID: \"808f3dd5-6179-4e46-91dc-23c8fde13111\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk"
Sep 16 19:10:43.555838 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:43.555726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/808f3dd5-6179-4e46-91dc-23c8fde13111-home\") pod \"test-div-v2-kserve-5f6fdb6b77-gfjrk\" (UID: \"808f3dd5-6179-4e46-91dc-23c8fde13111\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk"
Sep 16 19:10:43.556068 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:43.555990    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/808f3dd5-6179-4e46-91dc-23c8fde13111-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-gfjrk\" (UID: \"808f3dd5-6179-4e46-91dc-23c8fde13111\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk"
Sep 16 19:10:43.556122 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:43.556034    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/808f3dd5-6179-4e46-91dc-23c8fde13111-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-gfjrk\" (UID: \"808f3dd5-6179-4e46-91dc-23c8fde13111\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk"
Sep 16 19:10:43.556176 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:43.556159    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/808f3dd5-6179-4e46-91dc-23c8fde13111-home\") pod \"test-div-v2-kserve-5f6fdb6b77-gfjrk\" (UID: \"808f3dd5-6179-4e46-91dc-23c8fde13111\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk"
Sep 16 19:10:43.558030 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:43.558007    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/808f3dd5-6179-4e46-91dc-23c8fde13111-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-gfjrk\" (UID: \"808f3dd5-6179-4e46-91dc-23c8fde13111\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk"
Sep 16 19:10:43.558164 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:43.558145    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/808f3dd5-6179-4e46-91dc-23c8fde13111-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-gfjrk\" (UID: \"808f3dd5-6179-4e46-91dc-23c8fde13111\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk"
Sep 16 19:10:43.566125 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:43.566101    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk42z\" (UniqueName: \"kubernetes.io/projected/808f3dd5-6179-4e46-91dc-23c8fde13111-kube-api-access-rk42z\") pod \"test-div-v2-kserve-5f6fdb6b77-gfjrk\" (UID: \"808f3dd5-6179-4e46-91dc-23c8fde13111\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk"
Sep 16 19:10:43.668289 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:43.668196    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk"
Sep 16 19:10:43.943134 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:43.943099    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk"]
Sep 16 19:10:43.945665 ip-10-0-142-80 kubenswrapper[2578]: W0916 19:10:43.945603    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod808f3dd5_6179_4e46_91dc_23c8fde13111.slice/crio-a69068c181a0480c86eff31d517705f5acf2215406316e6b95bf6456090472ee WatchSource:0}: Error finding container a69068c181a0480c86eff31d517705f5acf2215406316e6b95bf6456090472ee: Status 404 returned error can't find the container with id a69068c181a0480c86eff31d517705f5acf2215406316e6b95bf6456090472ee
Sep 16 19:10:44.219221 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:44.219120    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk" event={"ID":"808f3dd5-6179-4e46-91dc-23c8fde13111","Type":"ContainerStarted","Data":"ed55b4ddcb75ffc1b25fe8deb5e9bef22a33efcedf6432a0a8c18bad22da27e5"}
Sep 16 19:10:44.219221 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:44.219197    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk" event={"ID":"808f3dd5-6179-4e46-91dc-23c8fde13111","Type":"ContainerStarted","Data":"a69068c181a0480c86eff31d517705f5acf2215406316e6b95bf6456090472ee"}
Sep 16 19:10:44.669180 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:44.669089    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk"
Sep 16 19:10:53.669086 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:53.669033    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk"
Sep 16 19:10:53.681276 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:53.681248    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk"
Sep 16 19:10:53.711354 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:53.711299    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk" podStartSLOduration=10.7112831 podStartE2EDuration="10.7112831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:44.245991294 +0000 UTC m=+1812.480924739" watchObservedRunningTime="2026-09-16 19:10:53.7112831 +0000 UTC m=+1821.946216543"
Sep 16 19:10:54.287904 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:10:54.287875    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk"
Sep 16 19:11:10.439645 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:10.439087    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"]
Sep 16 19:11:10.452415 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:11:10.452376    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:11:10.452415 ip-10-0-142-80 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:11:10.452415 ip-10-0-142-80 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:11:10.452415 ip-10-0-142-80 kubenswrapper[2578]:  > containerID="85d75e7af1bab84fefecdca9b3ffbaf5f05d57a389a557cac0a1fb2148db4dac" cmd=["/bin/sleep","15"]
Sep 16 19:11:10.452683 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:10.452444    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf" message="" err=<
Sep 16 19:11:10.452683 ip-10-0-142-80 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:11:10.452683 ip-10-0-142-80 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:11:10.452683 ip-10-0-142-80 kubenswrapper[2578]:  >
Sep 16 19:11:10.452683 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:11:10.452465    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:11:10.452683 ip-10-0-142-80 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:11:10.452683 ip-10-0-142-80 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:11:10.452683 ip-10-0-142-80 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf" podUID="615ce087-755a-4ece-a039-d4a5872a5521" containerName="main" containerID="cri-o://85d75e7af1bab84fefecdca9b3ffbaf5f05d57a389a557cac0a1fb2148db4dac"
Sep 16 19:11:10.452683 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:10.452516    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf" podUID="615ce087-755a-4ece-a039-d4a5872a5521" containerName="main" containerID="cri-o://85d75e7af1bab84fefecdca9b3ffbaf5f05d57a389a557cac0a1fb2148db4dac" gracePeriod=60
Sep 16 19:11:10.806702 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:10.806679    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"
Sep 16 19:11:10.881285 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:10.881255    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/615ce087-755a-4ece-a039-d4a5872a5521-home\") pod \"615ce087-755a-4ece-a039-d4a5872a5521\" (UID: \"615ce087-755a-4ece-a039-d4a5872a5521\") "
Sep 16 19:11:10.881430 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:10.881295    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/615ce087-755a-4ece-a039-d4a5872a5521-cabundle-cert\") pod \"615ce087-755a-4ece-a039-d4a5872a5521\" (UID: \"615ce087-755a-4ece-a039-d4a5872a5521\") "
Sep 16 19:11:10.881430 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:10.881315    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/615ce087-755a-4ece-a039-d4a5872a5521-kserve-provision-location\") pod \"615ce087-755a-4ece-a039-d4a5872a5521\" (UID: \"615ce087-755a-4ece-a039-d4a5872a5521\") "
Sep 16 19:11:10.881430 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:10.881338    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/615ce087-755a-4ece-a039-d4a5872a5521-dshm\") pod \"615ce087-755a-4ece-a039-d4a5872a5521\" (UID: \"615ce087-755a-4ece-a039-d4a5872a5521\") "
Sep 16 19:11:10.881430 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:10.881364    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/615ce087-755a-4ece-a039-d4a5872a5521-tls-certs\") pod \"615ce087-755a-4ece-a039-d4a5872a5521\" (UID: \"615ce087-755a-4ece-a039-d4a5872a5521\") "
Sep 16 19:11:10.881430 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:10.881385    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/615ce087-755a-4ece-a039-d4a5872a5521-tmp-dir\") pod \"615ce087-755a-4ece-a039-d4a5872a5521\" (UID: \"615ce087-755a-4ece-a039-d4a5872a5521\") "
Sep 16 19:11:10.881430 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:10.881399    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/615ce087-755a-4ece-a039-d4a5872a5521-model-cache\") pod \"615ce087-755a-4ece-a039-d4a5872a5521\" (UID: \"615ce087-755a-4ece-a039-d4a5872a5521\") "
Sep 16 19:11:10.881664 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:10.881529    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615ce087-755a-4ece-a039-d4a5872a5521-home" (OuterVolumeSpecName: "home") pod "615ce087-755a-4ece-a039-d4a5872a5521" (UID: "615ce087-755a-4ece-a039-d4a5872a5521"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:10.881710 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:10.881667    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615ce087-755a-4ece-a039-d4a5872a5521-model-cache" (OuterVolumeSpecName: "model-cache") pod "615ce087-755a-4ece-a039-d4a5872a5521" (UID: "615ce087-755a-4ece-a039-d4a5872a5521"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:10.881710 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:10.881674    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/615ce087-755a-4ece-a039-d4a5872a5521-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "615ce087-755a-4ece-a039-d4a5872a5521" (UID: "615ce087-755a-4ece-a039-d4a5872a5521"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:10.881710 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:10.881679    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615ce087-755a-4ece-a039-d4a5872a5521-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "615ce087-755a-4ece-a039-d4a5872a5521" (UID: "615ce087-755a-4ece-a039-d4a5872a5521"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:10.881826 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:10.881770    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6wcf\" (UniqueName: \"kubernetes.io/projected/615ce087-755a-4ece-a039-d4a5872a5521-kube-api-access-q6wcf\") pod \"615ce087-755a-4ece-a039-d4a5872a5521\" (UID: \"615ce087-755a-4ece-a039-d4a5872a5521\") "
Sep 16 19:11:10.882063 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:10.882028    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/615ce087-755a-4ece-a039-d4a5872a5521-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:11:10.882063 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:10.882064    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/615ce087-755a-4ece-a039-d4a5872a5521-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:11:10.882063 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:10.882073    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/615ce087-755a-4ece-a039-d4a5872a5521-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:11:10.882285 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:10.882080    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/615ce087-755a-4ece-a039-d4a5872a5521-cabundle-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:11:10.883813 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:10.883777    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615ce087-755a-4ece-a039-d4a5872a5521-dshm" (OuterVolumeSpecName: "dshm") pod "615ce087-755a-4ece-a039-d4a5872a5521" (UID: "615ce087-755a-4ece-a039-d4a5872a5521"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:10.883933 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:10.883847    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/615ce087-755a-4ece-a039-d4a5872a5521-kube-api-access-q6wcf" (OuterVolumeSpecName: "kube-api-access-q6wcf") pod "615ce087-755a-4ece-a039-d4a5872a5521" (UID: "615ce087-755a-4ece-a039-d4a5872a5521"). InnerVolumeSpecName "kube-api-access-q6wcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:10.883992 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:10.883940    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/615ce087-755a-4ece-a039-d4a5872a5521-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "615ce087-755a-4ece-a039-d4a5872a5521" (UID: "615ce087-755a-4ece-a039-d4a5872a5521"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:10.901151 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:10.901127    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615ce087-755a-4ece-a039-d4a5872a5521-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "615ce087-755a-4ece-a039-d4a5872a5521" (UID: "615ce087-755a-4ece-a039-d4a5872a5521"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:10.983400 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:10.983340    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6wcf\" (UniqueName: \"kubernetes.io/projected/615ce087-755a-4ece-a039-d4a5872a5521-kube-api-access-q6wcf\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:11:10.983400 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:10.983364    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/615ce087-755a-4ece-a039-d4a5872a5521-kserve-provision-location\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:11:10.983400 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:10.983374    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/615ce087-755a-4ece-a039-d4a5872a5521-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:11:10.983400 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:10.983383    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/615ce087-755a-4ece-a039-d4a5872a5521-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:11:11.367007 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:11.366908    2578 generic.go:356] "Generic (PLEG): container finished" podID="615ce087-755a-4ece-a039-d4a5872a5521" containerID="85d75e7af1bab84fefecdca9b3ffbaf5f05d57a389a557cac0a1fb2148db4dac" exitCode=0
Sep 16 19:11:11.367007 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:11.366966    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"
Sep 16 19:11:11.367268 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:11.366996    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf" event={"ID":"615ce087-755a-4ece-a039-d4a5872a5521","Type":"ContainerDied","Data":"85d75e7af1bab84fefecdca9b3ffbaf5f05d57a389a557cac0a1fb2148db4dac"}
Sep 16 19:11:11.367268 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:11.367082    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf" event={"ID":"615ce087-755a-4ece-a039-d4a5872a5521","Type":"ContainerDied","Data":"c2faae5324e3dbe5e9827287e493a2882076ab65d499f5c662ed53215d5dfebe"}
Sep 16 19:11:11.367268 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:11.367118    2578 scope.go:117] "RemoveContainer" containerID="85d75e7af1bab84fefecdca9b3ffbaf5f05d57a389a557cac0a1fb2148db4dac"
Sep 16 19:11:11.376928 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:11.376913    2578 scope.go:117] "RemoveContainer" containerID="4c47cf340e5b05abded4a7be97f125c8baa605cbe766a0aca80a1287daea588b"
Sep 16 19:11:11.384409 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:11.384392    2578 scope.go:117] "RemoveContainer" containerID="85d75e7af1bab84fefecdca9b3ffbaf5f05d57a389a557cac0a1fb2148db4dac"
Sep 16 19:11:11.384664 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:11:11.384647    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85d75e7af1bab84fefecdca9b3ffbaf5f05d57a389a557cac0a1fb2148db4dac\": container with ID starting with 85d75e7af1bab84fefecdca9b3ffbaf5f05d57a389a557cac0a1fb2148db4dac not found: ID does not exist" containerID="85d75e7af1bab84fefecdca9b3ffbaf5f05d57a389a557cac0a1fb2148db4dac"
Sep 16 19:11:11.384711 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:11.384673    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85d75e7af1bab84fefecdca9b3ffbaf5f05d57a389a557cac0a1fb2148db4dac"} err="failed to get container status \"85d75e7af1bab84fefecdca9b3ffbaf5f05d57a389a557cac0a1fb2148db4dac\": rpc error: code = NotFound desc = could not find container \"85d75e7af1bab84fefecdca9b3ffbaf5f05d57a389a557cac0a1fb2148db4dac\": container with ID starting with 85d75e7af1bab84fefecdca9b3ffbaf5f05d57a389a557cac0a1fb2148db4dac not found: ID does not exist"
Sep 16 19:11:11.384711 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:11.384688    2578 scope.go:117] "RemoveContainer" containerID="4c47cf340e5b05abded4a7be97f125c8baa605cbe766a0aca80a1287daea588b"
Sep 16 19:11:11.384942 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:11:11.384924    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c47cf340e5b05abded4a7be97f125c8baa605cbe766a0aca80a1287daea588b\": container with ID starting with 4c47cf340e5b05abded4a7be97f125c8baa605cbe766a0aca80a1287daea588b not found: ID does not exist" containerID="4c47cf340e5b05abded4a7be97f125c8baa605cbe766a0aca80a1287daea588b"
Sep 16 19:11:11.384978 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:11.384947    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c47cf340e5b05abded4a7be97f125c8baa605cbe766a0aca80a1287daea588b"} err="failed to get container status \"4c47cf340e5b05abded4a7be97f125c8baa605cbe766a0aca80a1287daea588b\": rpc error: code = NotFound desc = could not find container \"4c47cf340e5b05abded4a7be97f125c8baa605cbe766a0aca80a1287daea588b\": container with ID starting with 4c47cf340e5b05abded4a7be97f125c8baa605cbe766a0aca80a1287daea588b not found: ID does not exist"
Sep 16 19:11:11.400682 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:11.400661    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"]
Sep 16 19:11:11.408066 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:11.408028    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-65d4ccb669-mbjtf"]
Sep 16 19:11:12.217604 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:12.217569    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="615ce087-755a-4ece-a039-d4a5872a5521" path="/var/lib/kubelet/pods/615ce087-755a-4ece-a039-d4a5872a5521/volumes"
Sep 16 19:11:17.688339 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:17.688301    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk"]
Sep 16 19:11:18.166676 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:18.166599    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx"]
Sep 16 19:11:32.706591 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:32.706523    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk" podUID="808f3dd5-6179-4e46-91dc-23c8fde13111" containerName="main" containerID="cri-o://ed55b4ddcb75ffc1b25fe8deb5e9bef22a33efcedf6432a0a8c18bad22da27e5" gracePeriod=45
Sep 16 19:11:32.947514 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:32.947491    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk"
Sep 16 19:11:33.066916 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.066843    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/808f3dd5-6179-4e46-91dc-23c8fde13111-tls-certs\") pod \"808f3dd5-6179-4e46-91dc-23c8fde13111\" (UID: \"808f3dd5-6179-4e46-91dc-23c8fde13111\") "
Sep 16 19:11:33.066916 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.066895    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/808f3dd5-6179-4e46-91dc-23c8fde13111-tmp-dir\") pod \"808f3dd5-6179-4e46-91dc-23c8fde13111\" (UID: \"808f3dd5-6179-4e46-91dc-23c8fde13111\") "
Sep 16 19:11:33.067139 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.066926    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/808f3dd5-6179-4e46-91dc-23c8fde13111-dshm\") pod \"808f3dd5-6179-4e46-91dc-23c8fde13111\" (UID: \"808f3dd5-6179-4e46-91dc-23c8fde13111\") "
Sep 16 19:11:33.067139 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.066980    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/808f3dd5-6179-4e46-91dc-23c8fde13111-model-cache\") pod \"808f3dd5-6179-4e46-91dc-23c8fde13111\" (UID: \"808f3dd5-6179-4e46-91dc-23c8fde13111\") "
Sep 16 19:11:33.067139 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.066994    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/808f3dd5-6179-4e46-91dc-23c8fde13111-home\") pod \"808f3dd5-6179-4e46-91dc-23c8fde13111\" (UID: \"808f3dd5-6179-4e46-91dc-23c8fde13111\") "
Sep 16 19:11:33.067139 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.067031    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rk42z\" (UniqueName: \"kubernetes.io/projected/808f3dd5-6179-4e46-91dc-23c8fde13111-kube-api-access-rk42z\") pod \"808f3dd5-6179-4e46-91dc-23c8fde13111\" (UID: \"808f3dd5-6179-4e46-91dc-23c8fde13111\") "
Sep 16 19:11:33.067347 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.067300    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/808f3dd5-6179-4e46-91dc-23c8fde13111-home" (OuterVolumeSpecName: "home") pod "808f3dd5-6179-4e46-91dc-23c8fde13111" (UID: "808f3dd5-6179-4e46-91dc-23c8fde13111"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:33.067347 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.067322    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/808f3dd5-6179-4e46-91dc-23c8fde13111-model-cache" (OuterVolumeSpecName: "model-cache") pod "808f3dd5-6179-4e46-91dc-23c8fde13111" (UID: "808f3dd5-6179-4e46-91dc-23c8fde13111"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:33.067449 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.067337    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/808f3dd5-6179-4e46-91dc-23c8fde13111-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "808f3dd5-6179-4e46-91dc-23c8fde13111" (UID: "808f3dd5-6179-4e46-91dc-23c8fde13111"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:33.069324 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.069302    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/808f3dd5-6179-4e46-91dc-23c8fde13111-dshm" (OuterVolumeSpecName: "dshm") pod "808f3dd5-6179-4e46-91dc-23c8fde13111" (UID: "808f3dd5-6179-4e46-91dc-23c8fde13111"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:33.069435 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.069358    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/808f3dd5-6179-4e46-91dc-23c8fde13111-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "808f3dd5-6179-4e46-91dc-23c8fde13111" (UID: "808f3dd5-6179-4e46-91dc-23c8fde13111"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:33.069647 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.069595    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/808f3dd5-6179-4e46-91dc-23c8fde13111-kube-api-access-rk42z" (OuterVolumeSpecName: "kube-api-access-rk42z") pod "808f3dd5-6179-4e46-91dc-23c8fde13111" (UID: "808f3dd5-6179-4e46-91dc-23c8fde13111"). InnerVolumeSpecName "kube-api-access-rk42z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:33.168313 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.168280    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/808f3dd5-6179-4e46-91dc-23c8fde13111-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:11:33.168313 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.168308    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/808f3dd5-6179-4e46-91dc-23c8fde13111-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:11:33.168313 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.168317    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/808f3dd5-6179-4e46-91dc-23c8fde13111-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:11:33.168517 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.168326    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/808f3dd5-6179-4e46-91dc-23c8fde13111-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:11:33.168517 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.168334    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk42z\" (UniqueName: \"kubernetes.io/projected/808f3dd5-6179-4e46-91dc-23c8fde13111-kube-api-access-rk42z\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:11:33.168517 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.168342    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/808f3dd5-6179-4e46-91dc-23c8fde13111-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:11:33.185600 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.185548    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx" podUID="54b7b3dc-47f3-44c6-b7a7-33593d3eb441" containerName="main" containerID="cri-o://1d5e316360762cf87182a178e11fd3e57301a8588c45a33e4cb52d359e4a7b0d" gracePeriod=45
Sep 16 19:11:33.189840 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.189805    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx" podUID="54b7b3dc-47f3-44c6-b7a7-33593d3eb441" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": EOF"
Sep 16 19:11:33.424772 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.424748    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx"
Sep 16 19:11:33.485808 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.485780    2578 generic.go:356] "Generic (PLEG): container finished" podID="808f3dd5-6179-4e46-91dc-23c8fde13111" containerID="ed55b4ddcb75ffc1b25fe8deb5e9bef22a33efcedf6432a0a8c18bad22da27e5" exitCode=0
Sep 16 19:11:33.485964 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.485820    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk"
Sep 16 19:11:33.485964 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.485874    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk" event={"ID":"808f3dd5-6179-4e46-91dc-23c8fde13111","Type":"ContainerDied","Data":"ed55b4ddcb75ffc1b25fe8deb5e9bef22a33efcedf6432a0a8c18bad22da27e5"}
Sep 16 19:11:33.485964 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.485928    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk" event={"ID":"808f3dd5-6179-4e46-91dc-23c8fde13111","Type":"ContainerDied","Data":"a69068c181a0480c86eff31d517705f5acf2215406316e6b95bf6456090472ee"}
Sep 16 19:11:33.485964 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.485960    2578 scope.go:117] "RemoveContainer" containerID="ed55b4ddcb75ffc1b25fe8deb5e9bef22a33efcedf6432a0a8c18bad22da27e5"
Sep 16 19:11:33.487309 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.487282    2578 generic.go:356] "Generic (PLEG): container finished" podID="54b7b3dc-47f3-44c6-b7a7-33593d3eb441" containerID="1d5e316360762cf87182a178e11fd3e57301a8588c45a33e4cb52d359e4a7b0d" exitCode=0
Sep 16 19:11:33.487448 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.487315    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx"
Sep 16 19:11:33.487448 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.487338    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx" event={"ID":"54b7b3dc-47f3-44c6-b7a7-33593d3eb441","Type":"ContainerDied","Data":"1d5e316360762cf87182a178e11fd3e57301a8588c45a33e4cb52d359e4a7b0d"}
Sep 16 19:11:33.487448 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.487379    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx" event={"ID":"54b7b3dc-47f3-44c6-b7a7-33593d3eb441","Type":"ContainerDied","Data":"271aafff05163829984d3a8975e519ea3ecf2fa09d0ca55778698002041ed995"}
Sep 16 19:11:33.494788 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.494772    2578 scope.go:117] "RemoveContainer" containerID="ed55b4ddcb75ffc1b25fe8deb5e9bef22a33efcedf6432a0a8c18bad22da27e5"
Sep 16 19:11:33.495079 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:11:33.495055    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed55b4ddcb75ffc1b25fe8deb5e9bef22a33efcedf6432a0a8c18bad22da27e5\": container with ID starting with ed55b4ddcb75ffc1b25fe8deb5e9bef22a33efcedf6432a0a8c18bad22da27e5 not found: ID does not exist" containerID="ed55b4ddcb75ffc1b25fe8deb5e9bef22a33efcedf6432a0a8c18bad22da27e5"
Sep 16 19:11:33.495151 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.495089    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed55b4ddcb75ffc1b25fe8deb5e9bef22a33efcedf6432a0a8c18bad22da27e5"} err="failed to get container status \"ed55b4ddcb75ffc1b25fe8deb5e9bef22a33efcedf6432a0a8c18bad22da27e5\": rpc error: code = NotFound desc = could not find container \"ed55b4ddcb75ffc1b25fe8deb5e9bef22a33efcedf6432a0a8c18bad22da27e5\": container with ID starting with ed55b4ddcb75ffc1b25fe8deb5e9bef22a33efcedf6432a0a8c18bad22da27e5 not found: ID does not exist"
Sep 16 19:11:33.495151 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.495110    2578 scope.go:117] "RemoveContainer" containerID="1d5e316360762cf87182a178e11fd3e57301a8588c45a33e4cb52d359e4a7b0d"
Sep 16 19:11:33.502572 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.502553    2578 scope.go:117] "RemoveContainer" containerID="1d5e316360762cf87182a178e11fd3e57301a8588c45a33e4cb52d359e4a7b0d"
Sep 16 19:11:33.502850 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:11:33.502834    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d5e316360762cf87182a178e11fd3e57301a8588c45a33e4cb52d359e4a7b0d\": container with ID starting with 1d5e316360762cf87182a178e11fd3e57301a8588c45a33e4cb52d359e4a7b0d not found: ID does not exist" containerID="1d5e316360762cf87182a178e11fd3e57301a8588c45a33e4cb52d359e4a7b0d"
Sep 16 19:11:33.502896 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.502858    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d5e316360762cf87182a178e11fd3e57301a8588c45a33e4cb52d359e4a7b0d"} err="failed to get container status \"1d5e316360762cf87182a178e11fd3e57301a8588c45a33e4cb52d359e4a7b0d\": rpc error: code = NotFound desc = could not find container \"1d5e316360762cf87182a178e11fd3e57301a8588c45a33e4cb52d359e4a7b0d\": container with ID starting with 1d5e316360762cf87182a178e11fd3e57301a8588c45a33e4cb52d359e4a7b0d not found: ID does not exist"
Sep 16 19:11:33.519118 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.517419    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk"]
Sep 16 19:11:33.523476 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.523453    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-gfjrk"]
Sep 16 19:11:33.570328 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.570302    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-tmp-dir\") pod \"54b7b3dc-47f3-44c6-b7a7-33593d3eb441\" (UID: \"54b7b3dc-47f3-44c6-b7a7-33593d3eb441\") "
Sep 16 19:11:33.570434 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.570389    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-home\") pod \"54b7b3dc-47f3-44c6-b7a7-33593d3eb441\" (UID: \"54b7b3dc-47f3-44c6-b7a7-33593d3eb441\") "
Sep 16 19:11:33.570434 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.570425    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-tls-certs\") pod \"54b7b3dc-47f3-44c6-b7a7-33593d3eb441\" (UID: \"54b7b3dc-47f3-44c6-b7a7-33593d3eb441\") "
Sep 16 19:11:33.570555 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.570453    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-model-cache\") pod \"54b7b3dc-47f3-44c6-b7a7-33593d3eb441\" (UID: \"54b7b3dc-47f3-44c6-b7a7-33593d3eb441\") "
Sep 16 19:11:33.570555 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.570485    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7m25\" (UniqueName: \"kubernetes.io/projected/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-kube-api-access-q7m25\") pod \"54b7b3dc-47f3-44c6-b7a7-33593d3eb441\" (UID: \"54b7b3dc-47f3-44c6-b7a7-33593d3eb441\") "
Sep 16 19:11:33.570555 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.570508    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-dshm\") pod \"54b7b3dc-47f3-44c6-b7a7-33593d3eb441\" (UID: \"54b7b3dc-47f3-44c6-b7a7-33593d3eb441\") "
Sep 16 19:11:33.570696 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.570584    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "54b7b3dc-47f3-44c6-b7a7-33593d3eb441" (UID: "54b7b3dc-47f3-44c6-b7a7-33593d3eb441"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:33.570759 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.570700    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-home" (OuterVolumeSpecName: "home") pod "54b7b3dc-47f3-44c6-b7a7-33593d3eb441" (UID: "54b7b3dc-47f3-44c6-b7a7-33593d3eb441"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:33.570759 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.570721    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-model-cache" (OuterVolumeSpecName: "model-cache") pod "54b7b3dc-47f3-44c6-b7a7-33593d3eb441" (UID: "54b7b3dc-47f3-44c6-b7a7-33593d3eb441"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:33.570865 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.570796    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:11:33.570865 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.570815    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:11:33.570865 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.570830    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:11:33.572439 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.572417    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-kube-api-access-q7m25" (OuterVolumeSpecName: "kube-api-access-q7m25") pod "54b7b3dc-47f3-44c6-b7a7-33593d3eb441" (UID: "54b7b3dc-47f3-44c6-b7a7-33593d3eb441"). InnerVolumeSpecName "kube-api-access-q7m25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:33.572502 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.572460    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-dshm" (OuterVolumeSpecName: "dshm") pod "54b7b3dc-47f3-44c6-b7a7-33593d3eb441" (UID: "54b7b3dc-47f3-44c6-b7a7-33593d3eb441"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:33.572502 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.572460    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "54b7b3dc-47f3-44c6-b7a7-33593d3eb441" (UID: "54b7b3dc-47f3-44c6-b7a7-33593d3eb441"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:33.671878 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.671812    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:11:33.671878 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.671836    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7m25\" (UniqueName: \"kubernetes.io/projected/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-kube-api-access-q7m25\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:11:33.671878 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.671845    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54b7b3dc-47f3-44c6-b7a7-33593d3eb441-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:11:33.813524 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.813494    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx"]
Sep 16 19:11:33.820598 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:33.820577    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-tzskx"]
Sep 16 19:11:34.216828 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:34.216797    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54b7b3dc-47f3-44c6-b7a7-33593d3eb441" path="/var/lib/kubelet/pods/54b7b3dc-47f3-44c6-b7a7-33593d3eb441/volumes"
Sep 16 19:11:34.217205 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:11:34.217189    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="808f3dd5-6179-4e46-91dc-23c8fde13111" path="/var/lib/kubelet/pods/808f3dd5-6179-4e46-91dc-23c8fde13111/volumes"
Sep 16 19:12:45.643344 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.643312    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"]
Sep 16 19:12:45.643770 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.643615    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54b7b3dc-47f3-44c6-b7a7-33593d3eb441" containerName="main"
Sep 16 19:12:45.643770 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.643627    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="54b7b3dc-47f3-44c6-b7a7-33593d3eb441" containerName="main"
Sep 16 19:12:45.643770 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.643640    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="615ce087-755a-4ece-a039-d4a5872a5521" containerName="storage-initializer"
Sep 16 19:12:45.643770 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.643646    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="615ce087-755a-4ece-a039-d4a5872a5521" containerName="storage-initializer"
Sep 16 19:12:45.643770 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.643652    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="615ce087-755a-4ece-a039-d4a5872a5521" containerName="main"
Sep 16 19:12:45.643770 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.643657    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="615ce087-755a-4ece-a039-d4a5872a5521" containerName="main"
Sep 16 19:12:45.643770 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.643670    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="808f3dd5-6179-4e46-91dc-23c8fde13111" containerName="main"
Sep 16 19:12:45.643770 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.643675    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="808f3dd5-6179-4e46-91dc-23c8fde13111" containerName="main"
Sep 16 19:12:45.643770 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.643720    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="615ce087-755a-4ece-a039-d4a5872a5521" containerName="main"
Sep 16 19:12:45.643770 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.643727    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="808f3dd5-6179-4e46-91dc-23c8fde13111" containerName="main"
Sep 16 19:12:45.643770 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.643735    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="54b7b3dc-47f3-44c6-b7a7-33593d3eb441" containerName="main"
Sep 16 19:12:45.648887 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.648855    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"
Sep 16 19:12:45.651683 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.651656    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 16 19:12:45.652168 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.652140    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:12:45.652294 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.652144    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 16 19:12:45.652294 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.652243    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 16 19:12:45.652829 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.652798    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 16 19:12:45.652829 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.652810    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-tl57x\""
Sep 16 19:12:45.658502 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.658477    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"]
Sep 16 19:12:45.732289 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.732262    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2afd79ae-ce4f-4c66-86da-2d8935ebea40-tls-certs\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b\" (UID: \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"
Sep 16 19:12:45.732413 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.732300    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2afd79ae-ce4f-4c66-86da-2d8935ebea40-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b\" (UID: \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"
Sep 16 19:12:45.732413 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.732319    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2afd79ae-ce4f-4c66-86da-2d8935ebea40-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b\" (UID: \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"
Sep 16 19:12:45.732413 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.732390    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2afd79ae-ce4f-4c66-86da-2d8935ebea40-dshm\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b\" (UID: \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"
Sep 16 19:12:45.732517 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.732431    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2afd79ae-ce4f-4c66-86da-2d8935ebea40-home\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b\" (UID: \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"
Sep 16 19:12:45.732517 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.732456    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb4d2\" (UniqueName: \"kubernetes.io/projected/2afd79ae-ce4f-4c66-86da-2d8935ebea40-kube-api-access-kb4d2\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b\" (UID: \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"
Sep 16 19:12:45.732517 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.732492    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2afd79ae-ce4f-4c66-86da-2d8935ebea40-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b\" (UID: \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"
Sep 16 19:12:45.732615 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.732538    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2afd79ae-ce4f-4c66-86da-2d8935ebea40-model-cache\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b\" (UID: \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"
Sep 16 19:12:45.833800 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.833769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2afd79ae-ce4f-4c66-86da-2d8935ebea40-dshm\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b\" (UID: \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"
Sep 16 19:12:45.833800 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.833808    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2afd79ae-ce4f-4c66-86da-2d8935ebea40-home\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b\" (UID: \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"
Sep 16 19:12:45.834020 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.833829    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb4d2\" (UniqueName: \"kubernetes.io/projected/2afd79ae-ce4f-4c66-86da-2d8935ebea40-kube-api-access-kb4d2\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b\" (UID: \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"
Sep 16 19:12:45.834020 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.833859    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2afd79ae-ce4f-4c66-86da-2d8935ebea40-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b\" (UID: \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"
Sep 16 19:12:45.834020 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.833887    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2afd79ae-ce4f-4c66-86da-2d8935ebea40-model-cache\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b\" (UID: \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"
Sep 16 19:12:45.834020 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.833930    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2afd79ae-ce4f-4c66-86da-2d8935ebea40-tls-certs\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b\" (UID: \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"
Sep 16 19:12:45.834020 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.833958    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2afd79ae-ce4f-4c66-86da-2d8935ebea40-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b\" (UID: \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"
Sep 16 19:12:45.834020 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.833984    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2afd79ae-ce4f-4c66-86da-2d8935ebea40-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b\" (UID: \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"
Sep 16 19:12:45.834398 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.834326    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2afd79ae-ce4f-4c66-86da-2d8935ebea40-home\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b\" (UID: \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"
Sep 16 19:12:45.834398 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.834356    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2afd79ae-ce4f-4c66-86da-2d8935ebea40-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b\" (UID: \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"
Sep 16 19:12:45.834398 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.834375    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2afd79ae-ce4f-4c66-86da-2d8935ebea40-model-cache\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b\" (UID: \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"
Sep 16 19:12:45.834528 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.834428    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2afd79ae-ce4f-4c66-86da-2d8935ebea40-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b\" (UID: \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"
Sep 16 19:12:45.834715 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.834699    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2afd79ae-ce4f-4c66-86da-2d8935ebea40-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b\" (UID: \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"
Sep 16 19:12:45.836089 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.836068    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2afd79ae-ce4f-4c66-86da-2d8935ebea40-dshm\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b\" (UID: \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"
Sep 16 19:12:45.836386 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.836367    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2afd79ae-ce4f-4c66-86da-2d8935ebea40-tls-certs\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b\" (UID: \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"
Sep 16 19:12:45.842579 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.842556    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb4d2\" (UniqueName: \"kubernetes.io/projected/2afd79ae-ce4f-4c66-86da-2d8935ebea40-kube-api-access-kb4d2\") pod \"tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b\" (UID: \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"
Sep 16 19:12:45.961302 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:45.961272    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"
Sep 16 19:12:46.239664 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:46.239637    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"]
Sep 16 19:12:46.241611 ip-10-0-142-80 kubenswrapper[2578]: W0916 19:12:46.241579    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2afd79ae_ce4f_4c66_86da_2d8935ebea40.slice/crio-36ef9132a7e7600905b226834b1fc9cc14681993eda07ad092250c0cc70c2254 WatchSource:0}: Error finding container 36ef9132a7e7600905b226834b1fc9cc14681993eda07ad092250c0cc70c2254: Status 404 returned error can't find the container with id 36ef9132a7e7600905b226834b1fc9cc14681993eda07ad092250c0cc70c2254
Sep 16 19:12:46.859251 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:46.859198    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b" event={"ID":"2afd79ae-ce4f-4c66-86da-2d8935ebea40","Type":"ContainerStarted","Data":"23e1e87e2d6ff5ec9aa8f948d96ac6b2dd7fa02a97f9ce8234bb6c68ae689892"}
Sep 16 19:12:46.859761 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:46.859265    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b" event={"ID":"2afd79ae-ce4f-4c66-86da-2d8935ebea40","Type":"ContainerStarted","Data":"36ef9132a7e7600905b226834b1fc9cc14681993eda07ad092250c0cc70c2254"}
Sep 16 19:12:48.873738 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:48.873656    2578 generic.go:356] "Generic (PLEG): container finished" podID="2afd79ae-ce4f-4c66-86da-2d8935ebea40" containerID="23e1e87e2d6ff5ec9aa8f948d96ac6b2dd7fa02a97f9ce8234bb6c68ae689892" exitCode=0
Sep 16 19:12:48.873738 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:48.873713    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b" event={"ID":"2afd79ae-ce4f-4c66-86da-2d8935ebea40","Type":"ContainerDied","Data":"23e1e87e2d6ff5ec9aa8f948d96ac6b2dd7fa02a97f9ce8234bb6c68ae689892"}
Sep 16 19:12:49.879875 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:49.879843    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b" event={"ID":"2afd79ae-ce4f-4c66-86da-2d8935ebea40","Type":"ContainerStarted","Data":"ec2b6a1ebf11c2f885c8267059539c821bf83b402e60f52a0bbe80ab90677939"}
Sep 16 19:12:49.904714 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:49.904670    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b" podStartSLOduration=2.692948591 podStartE2EDuration="4.904655937s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.211707346s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:49.903507513 +0000 UTC m=+1938.138440956" watchObservedRunningTime="2026-09-16 19:12:49.904655937 +0000 UTC m=+1938.139589423"
Sep 16 19:12:49.961436 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:49.961406    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"
Sep 16 19:12:55.961983 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:55.961951    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"
Sep 16 19:12:55.973535 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:55.973510    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"
Sep 16 19:12:56.925331 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:12:56.925301    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"
Sep 16 19:13:36.542659 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:36.542572    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"]
Sep 16 19:13:36.552103 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:13:36.552063    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:13:36.552103 ip-10-0-142-80 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:13:36.552103 ip-10-0-142-80 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:13:36.552103 ip-10-0-142-80 kubenswrapper[2578]:  > containerID="ec2b6a1ebf11c2f885c8267059539c821bf83b402e60f52a0bbe80ab90677939" cmd=["/bin/sleep","15"]
Sep 16 19:13:36.552731 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:36.552114    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b" message="" err=<
Sep 16 19:13:36.552731 ip-10-0-142-80 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:13:36.552731 ip-10-0-142-80 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:13:36.552731 ip-10-0-142-80 kubenswrapper[2578]:  >
Sep 16 19:13:36.552731 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:13:36.552128    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:13:36.552731 ip-10-0-142-80 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:13:36.552731 ip-10-0-142-80 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:13:36.552731 ip-10-0-142-80 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b" podUID="2afd79ae-ce4f-4c66-86da-2d8935ebea40" containerName="main" containerID="cri-o://ec2b6a1ebf11c2f885c8267059539c821bf83b402e60f52a0bbe80ab90677939"
Sep 16 19:13:36.552731 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:36.552162    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b" podUID="2afd79ae-ce4f-4c66-86da-2d8935ebea40" containerName="main" containerID="cri-o://ec2b6a1ebf11c2f885c8267059539c821bf83b402e60f52a0bbe80ab90677939" gracePeriod=60
Sep 16 19:13:36.906109 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:36.906083    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"
Sep 16 19:13:37.046582 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.046541    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2afd79ae-ce4f-4c66-86da-2d8935ebea40-dshm\") pod \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\" (UID: \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\") "
Sep 16 19:13:37.046785 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.046602    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2afd79ae-ce4f-4c66-86da-2d8935ebea40-tls-certs\") pod \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\" (UID: \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\") "
Sep 16 19:13:37.046785 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.046628    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2afd79ae-ce4f-4c66-86da-2d8935ebea40-kserve-provision-location\") pod \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\" (UID: \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\") "
Sep 16 19:13:37.046785 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.046654    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2afd79ae-ce4f-4c66-86da-2d8935ebea40-cabundle-cert\") pod \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\" (UID: \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\") "
Sep 16 19:13:37.046994 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.046809    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2afd79ae-ce4f-4c66-86da-2d8935ebea40-model-cache\") pod \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\" (UID: \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\") "
Sep 16 19:13:37.046994 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.046984    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2afd79ae-ce4f-4c66-86da-2d8935ebea40-tmp-dir\") pod \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\" (UID: \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\") "
Sep 16 19:13:37.047389 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.047026    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kb4d2\" (UniqueName: \"kubernetes.io/projected/2afd79ae-ce4f-4c66-86da-2d8935ebea40-kube-api-access-kb4d2\") pod \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\" (UID: \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\") "
Sep 16 19:13:37.047389 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.047060    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2afd79ae-ce4f-4c66-86da-2d8935ebea40-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2afd79ae-ce4f-4c66-86da-2d8935ebea40" (UID: "2afd79ae-ce4f-4c66-86da-2d8935ebea40"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:13:37.047389 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.047093    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2afd79ae-ce4f-4c66-86da-2d8935ebea40-home\") pod \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\" (UID: \"2afd79ae-ce4f-4c66-86da-2d8935ebea40\") "
Sep 16 19:13:37.047389 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.047200    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afd79ae-ce4f-4c66-86da-2d8935ebea40-model-cache" (OuterVolumeSpecName: "model-cache") pod "2afd79ae-ce4f-4c66-86da-2d8935ebea40" (UID: "2afd79ae-ce4f-4c66-86da-2d8935ebea40"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:37.047389 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.047216    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afd79ae-ce4f-4c66-86da-2d8935ebea40-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2afd79ae-ce4f-4c66-86da-2d8935ebea40" (UID: "2afd79ae-ce4f-4c66-86da-2d8935ebea40"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:37.047607 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.047421    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afd79ae-ce4f-4c66-86da-2d8935ebea40-home" (OuterVolumeSpecName: "home") pod "2afd79ae-ce4f-4c66-86da-2d8935ebea40" (UID: "2afd79ae-ce4f-4c66-86da-2d8935ebea40"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:37.047607 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.047437    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2afd79ae-ce4f-4c66-86da-2d8935ebea40-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.047607 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.047458    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2afd79ae-ce4f-4c66-86da-2d8935ebea40-cabundle-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.047607 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.047475    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2afd79ae-ce4f-4c66-86da-2d8935ebea40-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.048830 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.048805    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afd79ae-ce4f-4c66-86da-2d8935ebea40-dshm" (OuterVolumeSpecName: "dshm") pod "2afd79ae-ce4f-4c66-86da-2d8935ebea40" (UID: "2afd79ae-ce4f-4c66-86da-2d8935ebea40"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:37.049233 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.049198    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2afd79ae-ce4f-4c66-86da-2d8935ebea40-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2afd79ae-ce4f-4c66-86da-2d8935ebea40" (UID: "2afd79ae-ce4f-4c66-86da-2d8935ebea40"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:13:37.049429 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.049410    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2afd79ae-ce4f-4c66-86da-2d8935ebea40-kube-api-access-kb4d2" (OuterVolumeSpecName: "kube-api-access-kb4d2") pod "2afd79ae-ce4f-4c66-86da-2d8935ebea40" (UID: "2afd79ae-ce4f-4c66-86da-2d8935ebea40"). InnerVolumeSpecName "kube-api-access-kb4d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:13:37.068229 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.068163    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afd79ae-ce4f-4c66-86da-2d8935ebea40-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2afd79ae-ce4f-4c66-86da-2d8935ebea40" (UID: "2afd79ae-ce4f-4c66-86da-2d8935ebea40"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:37.115823 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.115786    2578 generic.go:356] "Generic (PLEG): container finished" podID="2afd79ae-ce4f-4c66-86da-2d8935ebea40" containerID="ec2b6a1ebf11c2f885c8267059539c821bf83b402e60f52a0bbe80ab90677939" exitCode=0
Sep 16 19:13:37.115986 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.115844    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"
Sep 16 19:13:37.115986 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.115873    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b" event={"ID":"2afd79ae-ce4f-4c66-86da-2d8935ebea40","Type":"ContainerDied","Data":"ec2b6a1ebf11c2f885c8267059539c821bf83b402e60f52a0bbe80ab90677939"}
Sep 16 19:13:37.115986 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.115934    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b" event={"ID":"2afd79ae-ce4f-4c66-86da-2d8935ebea40","Type":"ContainerDied","Data":"36ef9132a7e7600905b226834b1fc9cc14681993eda07ad092250c0cc70c2254"}
Sep 16 19:13:37.115986 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.115958    2578 scope.go:117] "RemoveContainer" containerID="ec2b6a1ebf11c2f885c8267059539c821bf83b402e60f52a0bbe80ab90677939"
Sep 16 19:13:37.125914 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.125891    2578 scope.go:117] "RemoveContainer" containerID="23e1e87e2d6ff5ec9aa8f948d96ac6b2dd7fa02a97f9ce8234bb6c68ae689892"
Sep 16 19:13:37.134121 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.134101    2578 scope.go:117] "RemoveContainer" containerID="ec2b6a1ebf11c2f885c8267059539c821bf83b402e60f52a0bbe80ab90677939"
Sep 16 19:13:37.134388 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:13:37.134367    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec2b6a1ebf11c2f885c8267059539c821bf83b402e60f52a0bbe80ab90677939\": container with ID starting with ec2b6a1ebf11c2f885c8267059539c821bf83b402e60f52a0bbe80ab90677939 not found: ID does not exist" containerID="ec2b6a1ebf11c2f885c8267059539c821bf83b402e60f52a0bbe80ab90677939"
Sep 16 19:13:37.134441 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.134401    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec2b6a1ebf11c2f885c8267059539c821bf83b402e60f52a0bbe80ab90677939"} err="failed to get container status \"ec2b6a1ebf11c2f885c8267059539c821bf83b402e60f52a0bbe80ab90677939\": rpc error: code = NotFound desc = could not find container \"ec2b6a1ebf11c2f885c8267059539c821bf83b402e60f52a0bbe80ab90677939\": container with ID starting with ec2b6a1ebf11c2f885c8267059539c821bf83b402e60f52a0bbe80ab90677939 not found: ID does not exist"
Sep 16 19:13:37.134441 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.134420    2578 scope.go:117] "RemoveContainer" containerID="23e1e87e2d6ff5ec9aa8f948d96ac6b2dd7fa02a97f9ce8234bb6c68ae689892"
Sep 16 19:13:37.134681 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:13:37.134668    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23e1e87e2d6ff5ec9aa8f948d96ac6b2dd7fa02a97f9ce8234bb6c68ae689892\": container with ID starting with 23e1e87e2d6ff5ec9aa8f948d96ac6b2dd7fa02a97f9ce8234bb6c68ae689892 not found: ID does not exist" containerID="23e1e87e2d6ff5ec9aa8f948d96ac6b2dd7fa02a97f9ce8234bb6c68ae689892"
Sep 16 19:13:37.134722 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.134685    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23e1e87e2d6ff5ec9aa8f948d96ac6b2dd7fa02a97f9ce8234bb6c68ae689892"} err="failed to get container status \"23e1e87e2d6ff5ec9aa8f948d96ac6b2dd7fa02a97f9ce8234bb6c68ae689892\": rpc error: code = NotFound desc = could not find container \"23e1e87e2d6ff5ec9aa8f948d96ac6b2dd7fa02a97f9ce8234bb6c68ae689892\": container with ID starting with 23e1e87e2d6ff5ec9aa8f948d96ac6b2dd7fa02a97f9ce8234bb6c68ae689892 not found: ID does not exist"
Sep 16 19:13:37.148423 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.148388    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2afd79ae-ce4f-4c66-86da-2d8935ebea40-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.148423 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.148421    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2afd79ae-ce4f-4c66-86da-2d8935ebea40-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.148615 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.148435    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2afd79ae-ce4f-4c66-86da-2d8935ebea40-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.148615 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.148448    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2afd79ae-ce4f-4c66-86da-2d8935ebea40-kserve-provision-location\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.148615 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.148463    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb4d2\" (UniqueName: \"kubernetes.io/projected/2afd79ae-ce4f-4c66-86da-2d8935ebea40-kube-api-access-kb4d2\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:13:37.149539 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.149515    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"]
Sep 16 19:13:37.159157 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:37.159118    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-799f9bd7bd-b2m7b"]
Sep 16 19:13:38.217988 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:13:38.217954    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2afd79ae-ce4f-4c66-86da-2d8935ebea40" path="/var/lib/kubelet/pods/2afd79ae-ce4f-4c66-86da-2d8935ebea40/volumes"
Sep 16 19:14:36.138395 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.138357    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"]
Sep 16 19:14:36.139064 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.138667    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2afd79ae-ce4f-4c66-86da-2d8935ebea40" containerName="main"
Sep 16 19:14:36.139064 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.138678    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2afd79ae-ce4f-4c66-86da-2d8935ebea40" containerName="main"
Sep 16 19:14:36.139064 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.138699    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2afd79ae-ce4f-4c66-86da-2d8935ebea40" containerName="storage-initializer"
Sep 16 19:14:36.139064 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.138704    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2afd79ae-ce4f-4c66-86da-2d8935ebea40" containerName="storage-initializer"
Sep 16 19:14:36.139064 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.138763    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2afd79ae-ce4f-4c66-86da-2d8935ebea40" containerName="main"
Sep 16 19:14:36.142975 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.142943    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"
Sep 16 19:14:36.149766 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.149729    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 16 19:14:36.150580 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.150553    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-x7z49\""
Sep 16 19:14:36.150580 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.150571    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 16 19:14:36.150790 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.150694    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:14:36.150790 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.150692    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 16 19:14:36.150790 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.150769    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 16 19:14:36.155342 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.155313    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"]
Sep 16 19:14:36.238280 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.238240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-dshm\") pod \"tokenizer-template-args-test-kserve-58487b448-ml7cs\" (UID: \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"
Sep 16 19:14:36.238280 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.238281    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-58487b448-ml7cs\" (UID: \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"
Sep 16 19:14:36.238528 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.238304    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-home\") pod \"tokenizer-template-args-test-kserve-58487b448-ml7cs\" (UID: \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"
Sep 16 19:14:36.238528 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.238381    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-58487b448-ml7cs\" (UID: \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"
Sep 16 19:14:36.238528 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.238417    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-model-cache\") pod \"tokenizer-template-args-test-kserve-58487b448-ml7cs\" (UID: \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"
Sep 16 19:14:36.238528 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.238477    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-tmp-dir\") pod \"tokenizer-template-args-test-kserve-58487b448-ml7cs\" (UID: \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"
Sep 16 19:14:36.238528 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.238494    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-tls-certs\") pod \"tokenizer-template-args-test-kserve-58487b448-ml7cs\" (UID: \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"
Sep 16 19:14:36.238528 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.238524    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxs72\" (UniqueName: \"kubernetes.io/projected/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-kube-api-access-zxs72\") pod \"tokenizer-template-args-test-kserve-58487b448-ml7cs\" (UID: \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"
Sep 16 19:14:36.339835 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.339792    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-tmp-dir\") pod \"tokenizer-template-args-test-kserve-58487b448-ml7cs\" (UID: \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"
Sep 16 19:14:36.340085 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.339853    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-tls-certs\") pod \"tokenizer-template-args-test-kserve-58487b448-ml7cs\" (UID: \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"
Sep 16 19:14:36.340085 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.339896    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxs72\" (UniqueName: \"kubernetes.io/projected/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-kube-api-access-zxs72\") pod \"tokenizer-template-args-test-kserve-58487b448-ml7cs\" (UID: \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"
Sep 16 19:14:36.340085 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.339936    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-dshm\") pod \"tokenizer-template-args-test-kserve-58487b448-ml7cs\" (UID: \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"
Sep 16 19:14:36.340085 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.339959    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-58487b448-ml7cs\" (UID: \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"
Sep 16 19:14:36.340283 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.340088    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-home\") pod \"tokenizer-template-args-test-kserve-58487b448-ml7cs\" (UID: \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"
Sep 16 19:14:36.340283 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.340168    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-58487b448-ml7cs\" (UID: \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"
Sep 16 19:14:36.340283 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.340199    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-model-cache\") pod \"tokenizer-template-args-test-kserve-58487b448-ml7cs\" (UID: \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"
Sep 16 19:14:36.340441 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.340327    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-tmp-dir\") pod \"tokenizer-template-args-test-kserve-58487b448-ml7cs\" (UID: \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"
Sep 16 19:14:36.340441 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.340373    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-58487b448-ml7cs\" (UID: \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"
Sep 16 19:14:36.340551 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.340532    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-model-cache\") pod \"tokenizer-template-args-test-kserve-58487b448-ml7cs\" (UID: \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"
Sep 16 19:14:36.340615 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.340599    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-home\") pod \"tokenizer-template-args-test-kserve-58487b448-ml7cs\" (UID: \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"
Sep 16 19:14:36.340920 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.340894    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-58487b448-ml7cs\" (UID: \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"
Sep 16 19:14:36.342302 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.342280    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-dshm\") pod \"tokenizer-template-args-test-kserve-58487b448-ml7cs\" (UID: \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"
Sep 16 19:14:36.342478 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.342462    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-tls-certs\") pod \"tokenizer-template-args-test-kserve-58487b448-ml7cs\" (UID: \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"
Sep 16 19:14:36.351310 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.351288    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxs72\" (UniqueName: \"kubernetes.io/projected/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-kube-api-access-zxs72\") pod \"tokenizer-template-args-test-kserve-58487b448-ml7cs\" (UID: \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"
Sep 16 19:14:36.460841 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.460792    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"
Sep 16 19:14:36.741714 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:36.741677    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"]
Sep 16 19:14:36.743643 ip-10-0-142-80 kubenswrapper[2578]: W0916 19:14:36.743603    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod513ff471_0bf9_40bf_9fb2_42e1ae1c65d3.slice/crio-1d2715a792e20639a215844e361742750a3fd9036369a1da6cdcd89335979abd WatchSource:0}: Error finding container 1d2715a792e20639a215844e361742750a3fd9036369a1da6cdcd89335979abd: Status 404 returned error can't find the container with id 1d2715a792e20639a215844e361742750a3fd9036369a1da6cdcd89335979abd
Sep 16 19:14:37.424560 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:37.424512    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs" event={"ID":"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3","Type":"ContainerStarted","Data":"0cedece5991dc986b548fecf943463e080c7a2a145b234095ca34fe074c1a368"}
Sep 16 19:14:37.425112 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:37.424574    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs" event={"ID":"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3","Type":"ContainerStarted","Data":"1d2715a792e20639a215844e361742750a3fd9036369a1da6cdcd89335979abd"}
Sep 16 19:14:39.439795 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:39.439759    2578 generic.go:356] "Generic (PLEG): container finished" podID="513ff471-0bf9-40bf-9fb2-42e1ae1c65d3" containerID="0cedece5991dc986b548fecf943463e080c7a2a145b234095ca34fe074c1a368" exitCode=0
Sep 16 19:14:39.440396 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:39.439835    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs" event={"ID":"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3","Type":"ContainerDied","Data":"0cedece5991dc986b548fecf943463e080c7a2a145b234095ca34fe074c1a368"}
Sep 16 19:14:40.447054 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:40.447010    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs" event={"ID":"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3","Type":"ContainerStarted","Data":"af76d91a9911db31407e7adcfb2554881316b5c71e04800bfbc9bf594f6b0579"}
Sep 16 19:14:40.461881 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:40.461843    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"
Sep 16 19:14:40.477532 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:40.477476    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs" podStartSLOduration=2.008235667 podStartE2EDuration="4.477456146s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.469220479s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:40.476644026 +0000 UTC m=+2048.711577452" watchObservedRunningTime="2026-09-16 19:14:40.477456146 +0000 UTC m=+2048.712389592"
Sep 16 19:14:46.461754 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:46.461718    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"
Sep 16 19:14:46.474602 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:46.474571    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"
Sep 16 19:14:46.490181 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:14:46.490148    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"
Sep 16 19:15:50.115472 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.115434    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"]
Sep 16 19:15:50.126122 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:15:50.126083    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:15:50.126122 ip-10-0-142-80 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:15:50.126122 ip-10-0-142-80 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:15:50.126122 ip-10-0-142-80 kubenswrapper[2578]:  > containerID="af76d91a9911db31407e7adcfb2554881316b5c71e04800bfbc9bf594f6b0579" cmd=["/bin/sleep","15"]
Sep 16 19:15:50.126433 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.126141    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs" message="" err=<
Sep 16 19:15:50.126433 ip-10-0-142-80 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:15:50.126433 ip-10-0-142-80 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:15:50.126433 ip-10-0-142-80 kubenswrapper[2578]:  >
Sep 16 19:15:50.126433 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:15:50.126153    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:15:50.126433 ip-10-0-142-80 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:15:50.126433 ip-10-0-142-80 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 16 19:15:50.126433 ip-10-0-142-80 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs" podUID="513ff471-0bf9-40bf-9fb2-42e1ae1c65d3" containerName="main" containerID="cri-o://af76d91a9911db31407e7adcfb2554881316b5c71e04800bfbc9bf594f6b0579"
Sep 16 19:15:50.126433 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.126191    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs" podUID="513ff471-0bf9-40bf-9fb2-42e1ae1c65d3" containerName="main" containerID="cri-o://af76d91a9911db31407e7adcfb2554881316b5c71e04800bfbc9bf594f6b0579" gracePeriod=60
Sep 16 19:15:50.476918 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.476893    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"
Sep 16 19:15:50.565909 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.565879    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-tls-certs\") pod \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\" (UID: \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\") "
Sep 16 19:15:50.566137 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.565937    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-home\") pod \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\" (UID: \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\") "
Sep 16 19:15:50.566137 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.566000    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-tmp-dir\") pod \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\" (UID: \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\") "
Sep 16 19:15:50.566137 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.566019    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxs72\" (UniqueName: \"kubernetes.io/projected/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-kube-api-access-zxs72\") pod \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\" (UID: \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\") "
Sep 16 19:15:50.566137 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.566069    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-kserve-provision-location\") pod \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\" (UID: \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\") "
Sep 16 19:15:50.566137 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.566098    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-model-cache\") pod \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\" (UID: \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\") "
Sep 16 19:15:50.566137 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.566127    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-dshm\") pod \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\" (UID: \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\") "
Sep 16 19:15:50.566531 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.566170    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-cabundle-cert\") pod \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\" (UID: \"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3\") "
Sep 16 19:15:50.566531 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.566302    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-home" (OuterVolumeSpecName: "home") pod "513ff471-0bf9-40bf-9fb2-42e1ae1c65d3" (UID: "513ff471-0bf9-40bf-9fb2-42e1ae1c65d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:50.566531 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.566338    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "513ff471-0bf9-40bf-9fb2-42e1ae1c65d3" (UID: "513ff471-0bf9-40bf-9fb2-42e1ae1c65d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:50.566531 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.566422    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:15:50.566531 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.566438    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:15:50.566766 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.566558    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-model-cache" (OuterVolumeSpecName: "model-cache") pod "513ff471-0bf9-40bf-9fb2-42e1ae1c65d3" (UID: "513ff471-0bf9-40bf-9fb2-42e1ae1c65d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:50.566766 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.566648    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "513ff471-0bf9-40bf-9fb2-42e1ae1c65d3" (UID: "513ff471-0bf9-40bf-9fb2-42e1ae1c65d3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:15:50.568423 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.568399    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "513ff471-0bf9-40bf-9fb2-42e1ae1c65d3" (UID: "513ff471-0bf9-40bf-9fb2-42e1ae1c65d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:15:50.568524 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.568478    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-dshm" (OuterVolumeSpecName: "dshm") pod "513ff471-0bf9-40bf-9fb2-42e1ae1c65d3" (UID: "513ff471-0bf9-40bf-9fb2-42e1ae1c65d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:50.568524 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.568490    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-kube-api-access-zxs72" (OuterVolumeSpecName: "kube-api-access-zxs72") pod "513ff471-0bf9-40bf-9fb2-42e1ae1c65d3" (UID: "513ff471-0bf9-40bf-9fb2-42e1ae1c65d3"). InnerVolumeSpecName "kube-api-access-zxs72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:15:50.586456 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.586427    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "513ff471-0bf9-40bf-9fb2-42e1ae1c65d3" (UID: "513ff471-0bf9-40bf-9fb2-42e1ae1c65d3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:50.667247 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.667164    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxs72\" (UniqueName: \"kubernetes.io/projected/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-kube-api-access-zxs72\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:15:50.667247 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.667196    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-kserve-provision-location\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:15:50.667247 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.667206    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:15:50.667247 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.667215    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:15:50.667247 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.667223    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-cabundle-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:15:50.667247 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.667230    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:15:50.817384 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.817349    2578 generic.go:356] "Generic (PLEG): container finished" podID="513ff471-0bf9-40bf-9fb2-42e1ae1c65d3" containerID="af76d91a9911db31407e7adcfb2554881316b5c71e04800bfbc9bf594f6b0579" exitCode=0
Sep 16 19:15:50.817598 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.817412    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"
Sep 16 19:15:50.817598 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.817426    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs" event={"ID":"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3","Type":"ContainerDied","Data":"af76d91a9911db31407e7adcfb2554881316b5c71e04800bfbc9bf594f6b0579"}
Sep 16 19:15:50.817598 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.817490    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs" event={"ID":"513ff471-0bf9-40bf-9fb2-42e1ae1c65d3","Type":"ContainerDied","Data":"1d2715a792e20639a215844e361742750a3fd9036369a1da6cdcd89335979abd"}
Sep 16 19:15:50.817598 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.817513    2578 scope.go:117] "RemoveContainer" containerID="af76d91a9911db31407e7adcfb2554881316b5c71e04800bfbc9bf594f6b0579"
Sep 16 19:15:50.828453 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.828435    2578 scope.go:117] "RemoveContainer" containerID="0cedece5991dc986b548fecf943463e080c7a2a145b234095ca34fe074c1a368"
Sep 16 19:15:50.836737 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.836720    2578 scope.go:117] "RemoveContainer" containerID="af76d91a9911db31407e7adcfb2554881316b5c71e04800bfbc9bf594f6b0579"
Sep 16 19:15:50.837081 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:15:50.837034    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af76d91a9911db31407e7adcfb2554881316b5c71e04800bfbc9bf594f6b0579\": container with ID starting with af76d91a9911db31407e7adcfb2554881316b5c71e04800bfbc9bf594f6b0579 not found: ID does not exist" containerID="af76d91a9911db31407e7adcfb2554881316b5c71e04800bfbc9bf594f6b0579"
Sep 16 19:15:50.837150 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.837093    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af76d91a9911db31407e7adcfb2554881316b5c71e04800bfbc9bf594f6b0579"} err="failed to get container status \"af76d91a9911db31407e7adcfb2554881316b5c71e04800bfbc9bf594f6b0579\": rpc error: code = NotFound desc = could not find container \"af76d91a9911db31407e7adcfb2554881316b5c71e04800bfbc9bf594f6b0579\": container with ID starting with af76d91a9911db31407e7adcfb2554881316b5c71e04800bfbc9bf594f6b0579 not found: ID does not exist"
Sep 16 19:15:50.837150 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.837112    2578 scope.go:117] "RemoveContainer" containerID="0cedece5991dc986b548fecf943463e080c7a2a145b234095ca34fe074c1a368"
Sep 16 19:15:50.837366 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:15:50.837352    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cedece5991dc986b548fecf943463e080c7a2a145b234095ca34fe074c1a368\": container with ID starting with 0cedece5991dc986b548fecf943463e080c7a2a145b234095ca34fe074c1a368 not found: ID does not exist" containerID="0cedece5991dc986b548fecf943463e080c7a2a145b234095ca34fe074c1a368"
Sep 16 19:15:50.837416 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.837368    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cedece5991dc986b548fecf943463e080c7a2a145b234095ca34fe074c1a368"} err="failed to get container status \"0cedece5991dc986b548fecf943463e080c7a2a145b234095ca34fe074c1a368\": rpc error: code = NotFound desc = could not find container \"0cedece5991dc986b548fecf943463e080c7a2a145b234095ca34fe074c1a368\": container with ID starting with 0cedece5991dc986b548fecf943463e080c7a2a145b234095ca34fe074c1a368 not found: ID does not exist"
Sep 16 19:15:50.848666 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.848639    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"]
Sep 16 19:15:50.862066 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:50.862025    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-58487b448-ml7cs"]
Sep 16 19:15:52.217678 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:15:52.217645    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="513ff471-0bf9-40bf-9fb2-42e1ae1c65d3" path="/var/lib/kubelet/pods/513ff471-0bf9-40bf-9fb2-42e1ae1c65d3/volumes"
Sep 16 19:17:08.015093 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:17:08.015033    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j"]
Sep 16 19:17:08.015642 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:17:08.015448    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="513ff471-0bf9-40bf-9fb2-42e1ae1c65d3" containerName="main"
Sep 16 19:17:08.015642 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:17:08.015463    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="513ff471-0bf9-40bf-9fb2-42e1ae1c65d3" containerName="main"
Sep 16 19:17:08.015642 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:17:08.015482    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="513ff471-0bf9-40bf-9fb2-42e1ae1c65d3" containerName="storage-initializer"
Sep 16 19:17:08.015642 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:17:08.015489    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="513ff471-0bf9-40bf-9fb2-42e1ae1c65d3" containerName="storage-initializer"
Sep 16 19:17:08.015642 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:17:08.015550    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="513ff471-0bf9-40bf-9fb2-42e1ae1c65d3" containerName="main"
Sep 16 19:17:08.018561 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:17:08.018538    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j"
Sep 16 19:17:08.022673 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:17:08.022646    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-wk2ld\""
Sep 16 19:17:08.022822 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:17:08.022724    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 16 19:17:08.022822 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:17:08.022776    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 16 19:17:08.023011 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:17:08.022991    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 16 19:17:08.023107 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:17:08.023058    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-gzcj4\""
Sep 16 19:17:08.029031 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:17:08.029009    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j"]
Sep 16 19:17:08.122213 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:17:08.122175    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59d9ab8f-aab9-4881-a125-355f46d53524-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j\" (UID: \"59d9ab8f-aab9-4881-a125-355f46d53524\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j"
Sep 16 19:17:08.122399 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:17:08.122249    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t495\" (UniqueName: \"kubernetes.io/projected/59d9ab8f-aab9-4881-a125-355f46d53524-kube-api-access-5t495\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j\" (UID: \"59d9ab8f-aab9-4881-a125-355f46d53524\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j"
Sep 16 19:17:08.222848 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:17:08.222808    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t495\" (UniqueName: \"kubernetes.io/projected/59d9ab8f-aab9-4881-a125-355f46d53524-kube-api-access-5t495\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j\" (UID: \"59d9ab8f-aab9-4881-a125-355f46d53524\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j"
Sep 16 19:17:08.223021 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:17:08.222920    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59d9ab8f-aab9-4881-a125-355f46d53524-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j\" (UID: \"59d9ab8f-aab9-4881-a125-355f46d53524\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j"
Sep 16 19:17:08.225485 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:17:08.225458    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59d9ab8f-aab9-4881-a125-355f46d53524-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j\" (UID: \"59d9ab8f-aab9-4881-a125-355f46d53524\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j"
Sep 16 19:17:08.232749 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:17:08.232727    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t495\" (UniqueName: \"kubernetes.io/projected/59d9ab8f-aab9-4881-a125-355f46d53524-kube-api-access-5t495\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j\" (UID: \"59d9ab8f-aab9-4881-a125-355f46d53524\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j"
Sep 16 19:17:08.331234 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:17:08.331124    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j"
Sep 16 19:17:08.608832 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:17:08.608655    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j"]
Sep 16 19:17:08.611830 ip-10-0-142-80 kubenswrapper[2578]: W0916 19:17:08.611804    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59d9ab8f_aab9_4881_a125_355f46d53524.slice/crio-3856ccc111c0237b60fca7e0c1f720ab328c26b6f9c6c75f048d7e140483b772 WatchSource:0}: Error finding container 3856ccc111c0237b60fca7e0c1f720ab328c26b6f9c6c75f048d7e140483b772: Status 404 returned error can't find the container with id 3856ccc111c0237b60fca7e0c1f720ab328c26b6f9c6c75f048d7e140483b772
Sep 16 19:17:09.208748 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:17:09.208701    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j" event={"ID":"59d9ab8f-aab9-4881-a125-355f46d53524","Type":"ContainerStarted","Data":"c8bd8ebac4cf364a9bea2e0eaa7f8be5f0148c01f822eb0ead9f3270502abbab"}
Sep 16 19:17:09.209268 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:17:09.208816    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j" event={"ID":"59d9ab8f-aab9-4881-a125-355f46d53524","Type":"ContainerStarted","Data":"3856ccc111c0237b60fca7e0c1f720ab328c26b6f9c6c75f048d7e140483b772"}
Sep 16 19:17:09.209268 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:17:09.208981    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j"
Sep 16 19:17:40.217848 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:17:40.217818    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j"
Sep 16 19:17:40.237568 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:17:40.237521    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j" podStartSLOduration=33.237505534 podStartE2EDuration="33.237505534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:17:09.22980359 +0000 UTC m=+2197.464737035" watchObservedRunningTime="2026-09-16 19:17:40.237505534 +0000 UTC m=+2228.472439005"
Sep 16 19:17:47.235322 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:17:47.235286    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j"]
Sep 16 19:17:50.214843 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:17:50.214803    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j" podUID="59d9ab8f-aab9-4881-a125-355f46d53524" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:18:00.215663 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:00.215580    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j" podUID="59d9ab8f-aab9-4881-a125-355f46d53524" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:18:02.254543 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:02.254489    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j" podUID="59d9ab8f-aab9-4881-a125-355f46d53524" containerName="main" containerID="cri-o://c8bd8ebac4cf364a9bea2e0eaa7f8be5f0148c01f822eb0ead9f3270502abbab" gracePeriod=45
Sep 16 19:18:10.215011 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:10.214966    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j" podUID="59d9ab8f-aab9-4881-a125-355f46d53524" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:18:10.217632 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:10.217609    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j"
Sep 16 19:18:20.215001 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:20.214959    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j" podUID="59d9ab8f-aab9-4881-a125-355f46d53524" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:18:30.214882 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:30.214839    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j" podUID="59d9ab8f-aab9-4881-a125-355f46d53524" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:18:32.509443 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:32.509415    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j"
Sep 16 19:18:32.633401 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:32.633301    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5t495\" (UniqueName: \"kubernetes.io/projected/59d9ab8f-aab9-4881-a125-355f46d53524-kube-api-access-5t495\") pod \"59d9ab8f-aab9-4881-a125-355f46d53524\" (UID: \"59d9ab8f-aab9-4881-a125-355f46d53524\") "
Sep 16 19:18:32.633401 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:32.633383    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59d9ab8f-aab9-4881-a125-355f46d53524-tls-certs\") pod \"59d9ab8f-aab9-4881-a125-355f46d53524\" (UID: \"59d9ab8f-aab9-4881-a125-355f46d53524\") "
Sep 16 19:18:32.635628 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:32.635593    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59d9ab8f-aab9-4881-a125-355f46d53524-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "59d9ab8f-aab9-4881-a125-355f46d53524" (UID: "59d9ab8f-aab9-4881-a125-355f46d53524"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:18:32.635763 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:32.635636    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59d9ab8f-aab9-4881-a125-355f46d53524-kube-api-access-5t495" (OuterVolumeSpecName: "kube-api-access-5t495") pod "59d9ab8f-aab9-4881-a125-355f46d53524" (UID: "59d9ab8f-aab9-4881-a125-355f46d53524"). InnerVolumeSpecName "kube-api-access-5t495". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:18:32.643325 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:32.643301    2578 generic.go:356] "Generic (PLEG): container finished" podID="59d9ab8f-aab9-4881-a125-355f46d53524" containerID="c8bd8ebac4cf364a9bea2e0eaa7f8be5f0148c01f822eb0ead9f3270502abbab" exitCode=0
Sep 16 19:18:32.643448 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:32.643344    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j"
Sep 16 19:18:32.643448 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:32.643379    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j" event={"ID":"59d9ab8f-aab9-4881-a125-355f46d53524","Type":"ContainerDied","Data":"c8bd8ebac4cf364a9bea2e0eaa7f8be5f0148c01f822eb0ead9f3270502abbab"}
Sep 16 19:18:32.643448 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:32.643426    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j" event={"ID":"59d9ab8f-aab9-4881-a125-355f46d53524","Type":"ContainerDied","Data":"3856ccc111c0237b60fca7e0c1f720ab328c26b6f9c6c75f048d7e140483b772"}
Sep 16 19:18:32.643566 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:32.643449    2578 scope.go:117] "RemoveContainer" containerID="c8bd8ebac4cf364a9bea2e0eaa7f8be5f0148c01f822eb0ead9f3270502abbab"
Sep 16 19:18:32.652563 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:32.652541    2578 scope.go:117] "RemoveContainer" containerID="c8bd8ebac4cf364a9bea2e0eaa7f8be5f0148c01f822eb0ead9f3270502abbab"
Sep 16 19:18:32.652825 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:18:32.652806    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8bd8ebac4cf364a9bea2e0eaa7f8be5f0148c01f822eb0ead9f3270502abbab\": container with ID starting with c8bd8ebac4cf364a9bea2e0eaa7f8be5f0148c01f822eb0ead9f3270502abbab not found: ID does not exist" containerID="c8bd8ebac4cf364a9bea2e0eaa7f8be5f0148c01f822eb0ead9f3270502abbab"
Sep 16 19:18:32.652872 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:32.652832    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8bd8ebac4cf364a9bea2e0eaa7f8be5f0148c01f822eb0ead9f3270502abbab"} err="failed to get container status \"c8bd8ebac4cf364a9bea2e0eaa7f8be5f0148c01f822eb0ead9f3270502abbab\": rpc error: code = NotFound desc = could not find container \"c8bd8ebac4cf364a9bea2e0eaa7f8be5f0148c01f822eb0ead9f3270502abbab\": container with ID starting with c8bd8ebac4cf364a9bea2e0eaa7f8be5f0148c01f822eb0ead9f3270502abbab not found: ID does not exist"
Sep 16 19:18:32.668134 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:32.668100    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j"]
Sep 16 19:18:32.672326 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:32.672299    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schen6t9j"]
Sep 16 19:18:32.734358 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:32.734321    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59d9ab8f-aab9-4881-a125-355f46d53524-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:18:32.734358 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:32.734350    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t495\" (UniqueName: \"kubernetes.io/projected/59d9ab8f-aab9-4881-a125-355f46d53524-kube-api-access-5t495\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:18:34.217485 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:34.217442    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59d9ab8f-aab9-4881-a125-355f46d53524" path="/var/lib/kubelet/pods/59d9ab8f-aab9-4881-a125-355f46d53524/volumes"
Sep 16 19:18:53.988056 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:53.988009    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq"]
Sep 16 19:18:53.988689 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:53.988431    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59d9ab8f-aab9-4881-a125-355f46d53524" containerName="main"
Sep 16 19:18:53.988689 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:53.988451    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="59d9ab8f-aab9-4881-a125-355f46d53524" containerName="main"
Sep 16 19:18:53.988689 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:53.988556    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="59d9ab8f-aab9-4881-a125-355f46d53524" containerName="main"
Sep 16 19:18:53.991703 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:53.991683    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq"
Sep 16 19:18:53.994517 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:53.994491    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-bjrrr\""
Sep 16 19:18:53.996034 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:53.996003    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 16 19:18:53.996034 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:53.996019    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 16 19:18:53.996320 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:53.996249    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-hk4zw\""
Sep 16 19:18:53.996394 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:53.996021    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 16 19:18:54.004002 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:54.003977    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq"]
Sep 16 19:18:54.118030 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:54.117994    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c2c550f-341a-42aa-92b3-eb5eb575c1b6-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq\" (UID: \"9c2c550f-341a-42aa-92b3-eb5eb575c1b6\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq"
Sep 16 19:18:54.118243 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:54.118123    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjcxw\" (UniqueName: \"kubernetes.io/projected/9c2c550f-341a-42aa-92b3-eb5eb575c1b6-kube-api-access-sjcxw\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq\" (UID: \"9c2c550f-341a-42aa-92b3-eb5eb575c1b6\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq"
Sep 16 19:18:54.219121 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:54.219089    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjcxw\" (UniqueName: \"kubernetes.io/projected/9c2c550f-341a-42aa-92b3-eb5eb575c1b6-kube-api-access-sjcxw\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq\" (UID: \"9c2c550f-341a-42aa-92b3-eb5eb575c1b6\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq"
Sep 16 19:18:54.219121 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:54.219135    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c2c550f-341a-42aa-92b3-eb5eb575c1b6-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq\" (UID: \"9c2c550f-341a-42aa-92b3-eb5eb575c1b6\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq"
Sep 16 19:18:54.221628 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:54.221604    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c2c550f-341a-42aa-92b3-eb5eb575c1b6-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq\" (UID: \"9c2c550f-341a-42aa-92b3-eb5eb575c1b6\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq"
Sep 16 19:18:54.230369 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:54.230345    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjcxw\" (UniqueName: \"kubernetes.io/projected/9c2c550f-341a-42aa-92b3-eb5eb575c1b6-kube-api-access-sjcxw\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq\" (UID: \"9c2c550f-341a-42aa-92b3-eb5eb575c1b6\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq"
Sep 16 19:18:54.303327 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:54.303230    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq"
Sep 16 19:18:54.578764 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:54.578740    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq"]
Sep 16 19:18:54.584060 ip-10-0-142-80 kubenswrapper[2578]: W0916 19:18:54.583999    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c2c550f_341a_42aa_92b3_eb5eb575c1b6.slice/crio-cd2a2f957e7360d91b0f5fce84907f487430b34964a6291df7f681b61948c18a WatchSource:0}: Error finding container cd2a2f957e7360d91b0f5fce84907f487430b34964a6291df7f681b61948c18a: Status 404 returned error can't find the container with id cd2a2f957e7360d91b0f5fce84907f487430b34964a6291df7f681b61948c18a
Sep 16 19:18:54.754113 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:54.754077    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq" event={"ID":"9c2c550f-341a-42aa-92b3-eb5eb575c1b6","Type":"ContainerStarted","Data":"c94230fe52c84c5b2f5437c435836b8170a98f5454369017810edda4db5ca21b"}
Sep 16 19:18:54.754282 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:54.754140    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq" event={"ID":"9c2c550f-341a-42aa-92b3-eb5eb575c1b6","Type":"ContainerStarted","Data":"cd2a2f957e7360d91b0f5fce84907f487430b34964a6291df7f681b61948c18a"}
Sep 16 19:18:54.754282 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:54.754175    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq"
Sep 16 19:18:54.781443 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:18:54.781393    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq" podStartSLOduration=1.781377891 podStartE2EDuration="1.781377891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:18:54.780271273 +0000 UTC m=+2303.015204717" watchObservedRunningTime="2026-09-16 19:18:54.781377891 +0000 UTC m=+2303.016311335"
Sep 16 19:19:25.764173 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:19:25.764079    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq"
Sep 16 19:21:31.520579 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:21:31.520529    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq"]
Sep 16 19:21:35.764552 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:21:35.764512    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq" podUID="9c2c550f-341a-42aa-92b3-eb5eb575c1b6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:21:45.764032 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:21:45.763986    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq" podUID="9c2c550f-341a-42aa-92b3-eb5eb575c1b6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:21:46.541830 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:21:46.541778    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq" podUID="9c2c550f-341a-42aa-92b3-eb5eb575c1b6" containerName="main" containerID="cri-o://c94230fe52c84c5b2f5437c435836b8170a98f5454369017810edda4db5ca21b" gracePeriod=45
Sep 16 19:21:55.763484 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:21:55.763442    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq" podUID="9c2c550f-341a-42aa-92b3-eb5eb575c1b6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:21:55.763978 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:21:55.763571    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq"
Sep 16 19:22:05.764182 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:05.764141    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq" podUID="9c2c550f-341a-42aa-92b3-eb5eb575c1b6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:22:15.763653 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:15.763609    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq" podUID="9c2c550f-341a-42aa-92b3-eb5eb575c1b6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:22:16.786232 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:16.786202    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq"
Sep 16 19:22:16.799729 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:16.799651    2578 generic.go:356] "Generic (PLEG): container finished" podID="9c2c550f-341a-42aa-92b3-eb5eb575c1b6" containerID="c94230fe52c84c5b2f5437c435836b8170a98f5454369017810edda4db5ca21b" exitCode=0
Sep 16 19:22:16.799729 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:16.799681    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq"
Sep 16 19:22:16.799906 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:16.799734    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq" event={"ID":"9c2c550f-341a-42aa-92b3-eb5eb575c1b6","Type":"ContainerDied","Data":"c94230fe52c84c5b2f5437c435836b8170a98f5454369017810edda4db5ca21b"}
Sep 16 19:22:16.799906 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:16.799788    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq" event={"ID":"9c2c550f-341a-42aa-92b3-eb5eb575c1b6","Type":"ContainerDied","Data":"cd2a2f957e7360d91b0f5fce84907f487430b34964a6291df7f681b61948c18a"}
Sep 16 19:22:16.799906 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:16.799811    2578 scope.go:117] "RemoveContainer" containerID="c94230fe52c84c5b2f5437c435836b8170a98f5454369017810edda4db5ca21b"
Sep 16 19:22:16.808140 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:16.808120    2578 scope.go:117] "RemoveContainer" containerID="c94230fe52c84c5b2f5437c435836b8170a98f5454369017810edda4db5ca21b"
Sep 16 19:22:16.808403 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:22:16.808383    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c94230fe52c84c5b2f5437c435836b8170a98f5454369017810edda4db5ca21b\": container with ID starting with c94230fe52c84c5b2f5437c435836b8170a98f5454369017810edda4db5ca21b not found: ID does not exist" containerID="c94230fe52c84c5b2f5437c435836b8170a98f5454369017810edda4db5ca21b"
Sep 16 19:22:16.808456 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:16.808411    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c94230fe52c84c5b2f5437c435836b8170a98f5454369017810edda4db5ca21b"} err="failed to get container status \"c94230fe52c84c5b2f5437c435836b8170a98f5454369017810edda4db5ca21b\": rpc error: code = NotFound desc = could not find container \"c94230fe52c84c5b2f5437c435836b8170a98f5454369017810edda4db5ca21b\": container with ID starting with c94230fe52c84c5b2f5437c435836b8170a98f5454369017810edda4db5ca21b not found: ID does not exist"
Sep 16 19:22:16.845995 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:16.845968    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c2c550f-341a-42aa-92b3-eb5eb575c1b6-tls-certs\") pod \"9c2c550f-341a-42aa-92b3-eb5eb575c1b6\" (UID: \"9c2c550f-341a-42aa-92b3-eb5eb575c1b6\") "
Sep 16 19:22:16.846142 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:16.846023    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjcxw\" (UniqueName: \"kubernetes.io/projected/9c2c550f-341a-42aa-92b3-eb5eb575c1b6-kube-api-access-sjcxw\") pod \"9c2c550f-341a-42aa-92b3-eb5eb575c1b6\" (UID: \"9c2c550f-341a-42aa-92b3-eb5eb575c1b6\") "
Sep 16 19:22:16.848207 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:16.848179    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c2c550f-341a-42aa-92b3-eb5eb575c1b6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c2c550f-341a-42aa-92b3-eb5eb575c1b6" (UID: "9c2c550f-341a-42aa-92b3-eb5eb575c1b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:22:16.848317 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:16.848206    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c2c550f-341a-42aa-92b3-eb5eb575c1b6-kube-api-access-sjcxw" (OuterVolumeSpecName: "kube-api-access-sjcxw") pod "9c2c550f-341a-42aa-92b3-eb5eb575c1b6" (UID: "9c2c550f-341a-42aa-92b3-eb5eb575c1b6"). InnerVolumeSpecName "kube-api-access-sjcxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:22:16.947623 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:16.947583    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c2c550f-341a-42aa-92b3-eb5eb575c1b6-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:22:16.947623 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:16.947616    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjcxw\" (UniqueName: \"kubernetes.io/projected/9c2c550f-341a-42aa-92b3-eb5eb575c1b6-kube-api-access-sjcxw\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:22:17.131275 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:17.131241    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq"]
Sep 16 19:22:17.140319 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:17.140291    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schewflrq"]
Sep 16 19:22:18.218493 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:18.218456    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c2c550f-341a-42aa-92b3-eb5eb575c1b6" path="/var/lib/kubelet/pods/9c2c550f-341a-42aa-92b3-eb5eb575c1b6/volumes"
Sep 16 19:22:33.462329 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:33.462284    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8"]
Sep 16 19:22:33.462831 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:33.462740    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c2c550f-341a-42aa-92b3-eb5eb575c1b6" containerName="main"
Sep 16 19:22:33.462831 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:33.462761    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c2c550f-341a-42aa-92b3-eb5eb575c1b6" containerName="main"
Sep 16 19:22:33.462946 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:33.462848    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c2c550f-341a-42aa-92b3-eb5eb575c1b6" containerName="main"
Sep 16 19:22:33.467164 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:33.467143    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8"
Sep 16 19:22:33.471300 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:33.471275    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 16 19:22:33.471435 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:33.471275    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-p6c74\""
Sep 16 19:22:33.471435 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:33.471280    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-6t5rj\""
Sep 16 19:22:33.471435 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:33.471338    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 16 19:22:33.471435 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:33.471315    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 16 19:22:33.476784 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:33.476765    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8"]
Sep 16 19:22:33.588337 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:33.588291    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97c90497-4388-4e05-a9a4-df6ddc80808b-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8\" (UID: \"97c90497-4388-4e05-a9a4-df6ddc80808b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8"
Sep 16 19:22:33.588527 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:33.588370    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hcfp\" (UniqueName: \"kubernetes.io/projected/97c90497-4388-4e05-a9a4-df6ddc80808b-kube-api-access-7hcfp\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8\" (UID: \"97c90497-4388-4e05-a9a4-df6ddc80808b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8"
Sep 16 19:22:33.689584 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:33.689542    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97c90497-4388-4e05-a9a4-df6ddc80808b-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8\" (UID: \"97c90497-4388-4e05-a9a4-df6ddc80808b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8"
Sep 16 19:22:33.689771 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:33.689602    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hcfp\" (UniqueName: \"kubernetes.io/projected/97c90497-4388-4e05-a9a4-df6ddc80808b-kube-api-access-7hcfp\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8\" (UID: \"97c90497-4388-4e05-a9a4-df6ddc80808b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8"
Sep 16 19:22:33.692026 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:33.692003    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97c90497-4388-4e05-a9a4-df6ddc80808b-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8\" (UID: \"97c90497-4388-4e05-a9a4-df6ddc80808b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8"
Sep 16 19:22:33.700692 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:33.700666    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hcfp\" (UniqueName: \"kubernetes.io/projected/97c90497-4388-4e05-a9a4-df6ddc80808b-kube-api-access-7hcfp\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8\" (UID: \"97c90497-4388-4e05-a9a4-df6ddc80808b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8"
Sep 16 19:22:33.778033 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:33.777938    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8"
Sep 16 19:22:34.054698 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:34.054671    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8"]
Sep 16 19:22:34.056929 ip-10-0-142-80 kubenswrapper[2578]: W0916 19:22:34.056895    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c90497_4388_4e05_a9a4_df6ddc80808b.slice/crio-412feee858eab460912ee9f67d362ef6134f303eab39d632e34d1b42e2b2c77b WatchSource:0}: Error finding container 412feee858eab460912ee9f67d362ef6134f303eab39d632e34d1b42e2b2c77b: Status 404 returned error can't find the container with id 412feee858eab460912ee9f67d362ef6134f303eab39d632e34d1b42e2b2c77b
Sep 16 19:22:34.891864 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:34.891825    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8" event={"ID":"97c90497-4388-4e05-a9a4-df6ddc80808b","Type":"ContainerStarted","Data":"5aca46b3ea6a3cbf344420c5723e1a370daa3772ecff09fe2d3d8b346fcd7aa5"}
Sep 16 19:22:34.892291 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:34.891877    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8" event={"ID":"97c90497-4388-4e05-a9a4-df6ddc80808b","Type":"ContainerStarted","Data":"412feee858eab460912ee9f67d362ef6134f303eab39d632e34d1b42e2b2c77b"}
Sep 16 19:22:34.892291 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:34.891990    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8"
Sep 16 19:22:34.915596 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:22:34.915544    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8" podStartSLOduration=1.9155280129999999 podStartE2EDuration="1.915528013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:22:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:22:34.914727409 +0000 UTC m=+2523.149660852" watchObservedRunningTime="2026-09-16 19:22:34.915528013 +0000 UTC m=+2523.150461500"
Sep 16 19:23:05.898511 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:23:05.898477    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8"
Sep 16 19:25:49.005966 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:25:49.005930    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8"]
Sep 16 19:25:55.898762 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:25:55.898720    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8" podUID="97c90497-4388-4e05-a9a4-df6ddc80808b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:26:04.030010 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:26:04.029948    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8" podUID="97c90497-4388-4e05-a9a4-df6ddc80808b" containerName="main" containerID="cri-o://5aca46b3ea6a3cbf344420c5723e1a370daa3772ecff09fe2d3d8b346fcd7aa5" gracePeriod=45
Sep 16 19:26:05.899427 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:26:05.899380    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8" podUID="97c90497-4388-4e05-a9a4-df6ddc80808b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:26:15.899302 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:26:15.899260    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8" podUID="97c90497-4388-4e05-a9a4-df6ddc80808b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:26:15.899721 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:26:15.899387    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8"
Sep 16 19:26:25.898747 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:26:25.898710    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8" podUID="97c90497-4388-4e05-a9a4-df6ddc80808b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:26:34.281897 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:26:34.281870    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8"
Sep 16 19:26:34.384897 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:26:34.384805    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97c90497-4388-4e05-a9a4-df6ddc80808b-tls-certs\") pod \"97c90497-4388-4e05-a9a4-df6ddc80808b\" (UID: \"97c90497-4388-4e05-a9a4-df6ddc80808b\") "
Sep 16 19:26:34.384897 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:26:34.384853    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hcfp\" (UniqueName: \"kubernetes.io/projected/97c90497-4388-4e05-a9a4-df6ddc80808b-kube-api-access-7hcfp\") pod \"97c90497-4388-4e05-a9a4-df6ddc80808b\" (UID: \"97c90497-4388-4e05-a9a4-df6ddc80808b\") "
Sep 16 19:26:34.387106 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:26:34.387075    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97c90497-4388-4e05-a9a4-df6ddc80808b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "97c90497-4388-4e05-a9a4-df6ddc80808b" (UID: "97c90497-4388-4e05-a9a4-df6ddc80808b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:26:34.387106 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:26:34.387095    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97c90497-4388-4e05-a9a4-df6ddc80808b-kube-api-access-7hcfp" (OuterVolumeSpecName: "kube-api-access-7hcfp") pod "97c90497-4388-4e05-a9a4-df6ddc80808b" (UID: "97c90497-4388-4e05-a9a4-df6ddc80808b"). InnerVolumeSpecName "kube-api-access-7hcfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:26:34.486213 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:26:34.486171    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97c90497-4388-4e05-a9a4-df6ddc80808b-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:26:34.486213 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:26:34.486205    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hcfp\" (UniqueName: \"kubernetes.io/projected/97c90497-4388-4e05-a9a4-df6ddc80808b-kube-api-access-7hcfp\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:26:35.103607 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:26:35.103571    2578 generic.go:356] "Generic (PLEG): container finished" podID="97c90497-4388-4e05-a9a4-df6ddc80808b" containerID="5aca46b3ea6a3cbf344420c5723e1a370daa3772ecff09fe2d3d8b346fcd7aa5" exitCode=0
Sep 16 19:26:35.103805 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:26:35.103618    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8"
Sep 16 19:26:35.103805 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:26:35.103658    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8" event={"ID":"97c90497-4388-4e05-a9a4-df6ddc80808b","Type":"ContainerDied","Data":"5aca46b3ea6a3cbf344420c5723e1a370daa3772ecff09fe2d3d8b346fcd7aa5"}
Sep 16 19:26:35.103805 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:26:35.103714    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8" event={"ID":"97c90497-4388-4e05-a9a4-df6ddc80808b","Type":"ContainerDied","Data":"412feee858eab460912ee9f67d362ef6134f303eab39d632e34d1b42e2b2c77b"}
Sep 16 19:26:35.103805 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:26:35.103744    2578 scope.go:117] "RemoveContainer" containerID="5aca46b3ea6a3cbf344420c5723e1a370daa3772ecff09fe2d3d8b346fcd7aa5"
Sep 16 19:26:35.113053 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:26:35.113015    2578 scope.go:117] "RemoveContainer" containerID="5aca46b3ea6a3cbf344420c5723e1a370daa3772ecff09fe2d3d8b346fcd7aa5"
Sep 16 19:26:35.113327 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:26:35.113309    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aca46b3ea6a3cbf344420c5723e1a370daa3772ecff09fe2d3d8b346fcd7aa5\": container with ID starting with 5aca46b3ea6a3cbf344420c5723e1a370daa3772ecff09fe2d3d8b346fcd7aa5 not found: ID does not exist" containerID="5aca46b3ea6a3cbf344420c5723e1a370daa3772ecff09fe2d3d8b346fcd7aa5"
Sep 16 19:26:35.113377 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:26:35.113337    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aca46b3ea6a3cbf344420c5723e1a370daa3772ecff09fe2d3d8b346fcd7aa5"} err="failed to get container status \"5aca46b3ea6a3cbf344420c5723e1a370daa3772ecff09fe2d3d8b346fcd7aa5\": rpc error: code = NotFound desc = could not find container \"5aca46b3ea6a3cbf344420c5723e1a370daa3772ecff09fe2d3d8b346fcd7aa5\": container with ID starting with 5aca46b3ea6a3cbf344420c5723e1a370daa3772ecff09fe2d3d8b346fcd7aa5 not found: ID does not exist"
Sep 16 19:26:35.127623 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:26:35.127598    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8"]
Sep 16 19:26:35.134423 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:26:35.134400    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schem7wr8"]
Sep 16 19:26:36.217724 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:26:36.217693    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97c90497-4388-4e05-a9a4-df6ddc80808b" path="/var/lib/kubelet/pods/97c90497-4388-4e05-a9a4-df6ddc80808b/volumes"
Sep 16 19:27:00.777297 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:00.777096    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"]
Sep 16 19:27:00.778826 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:00.777895    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97c90497-4388-4e05-a9a4-df6ddc80808b" containerName="main"
Sep 16 19:27:00.778826 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:00.777923    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="97c90497-4388-4e05-a9a4-df6ddc80808b" containerName="main"
Sep 16 19:27:00.778826 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:00.778111    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="97c90497-4388-4e05-a9a4-df6ddc80808b" containerName="main"
Sep 16 19:27:00.782569 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:00.782540    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"
Sep 16 19:27:00.785699 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:00.785671    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Sep 16 19:27:00.785846 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:00.785673    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 16 19:27:00.785920 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:00.785843    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-9279j\""
Sep 16 19:27:00.785920 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:00.785844    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 16 19:27:00.785920 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:00.785778    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 16 19:27:00.786194 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:00.786176    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:27:00.797167 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:00.797146    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"]
Sep 16 19:27:00.913906 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:00.913869    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5100699a-2aa4-4798-b8d8-37b9e5136c13-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99\" (UID: \"5100699a-2aa4-4798-b8d8-37b9e5136c13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"
Sep 16 19:27:00.913906 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:00.913913    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5100699a-2aa4-4798-b8d8-37b9e5136c13-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99\" (UID: \"5100699a-2aa4-4798-b8d8-37b9e5136c13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"
Sep 16 19:27:00.914156 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:00.913946    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5100699a-2aa4-4798-b8d8-37b9e5136c13-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99\" (UID: \"5100699a-2aa4-4798-b8d8-37b9e5136c13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"
Sep 16 19:27:00.914156 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:00.913974    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppp6l\" (UniqueName: \"kubernetes.io/projected/5100699a-2aa4-4798-b8d8-37b9e5136c13-kube-api-access-ppp6l\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99\" (UID: \"5100699a-2aa4-4798-b8d8-37b9e5136c13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"
Sep 16 19:27:00.914156 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:00.914004    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5100699a-2aa4-4798-b8d8-37b9e5136c13-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99\" (UID: \"5100699a-2aa4-4798-b8d8-37b9e5136c13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"
Sep 16 19:27:00.914156 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:00.914080    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5100699a-2aa4-4798-b8d8-37b9e5136c13-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99\" (UID: \"5100699a-2aa4-4798-b8d8-37b9e5136c13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"
Sep 16 19:27:00.914156 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:00.914111    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5100699a-2aa4-4798-b8d8-37b9e5136c13-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99\" (UID: \"5100699a-2aa4-4798-b8d8-37b9e5136c13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"
Sep 16 19:27:00.914156 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:00.914142    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5100699a-2aa4-4798-b8d8-37b9e5136c13-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99\" (UID: \"5100699a-2aa4-4798-b8d8-37b9e5136c13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"
Sep 16 19:27:01.014862 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:01.014812    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5100699a-2aa4-4798-b8d8-37b9e5136c13-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99\" (UID: \"5100699a-2aa4-4798-b8d8-37b9e5136c13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"
Sep 16 19:27:01.014862 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:01.014879    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5100699a-2aa4-4798-b8d8-37b9e5136c13-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99\" (UID: \"5100699a-2aa4-4798-b8d8-37b9e5136c13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"
Sep 16 19:27:01.015193 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:01.014898    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppp6l\" (UniqueName: \"kubernetes.io/projected/5100699a-2aa4-4798-b8d8-37b9e5136c13-kube-api-access-ppp6l\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99\" (UID: \"5100699a-2aa4-4798-b8d8-37b9e5136c13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"
Sep 16 19:27:01.015193 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:01.014919    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5100699a-2aa4-4798-b8d8-37b9e5136c13-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99\" (UID: \"5100699a-2aa4-4798-b8d8-37b9e5136c13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"
Sep 16 19:27:01.015193 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:01.014951    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5100699a-2aa4-4798-b8d8-37b9e5136c13-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99\" (UID: \"5100699a-2aa4-4798-b8d8-37b9e5136c13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"
Sep 16 19:27:01.015193 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:01.014968    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5100699a-2aa4-4798-b8d8-37b9e5136c13-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99\" (UID: \"5100699a-2aa4-4798-b8d8-37b9e5136c13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"
Sep 16 19:27:01.015193 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:01.014992    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5100699a-2aa4-4798-b8d8-37b9e5136c13-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99\" (UID: \"5100699a-2aa4-4798-b8d8-37b9e5136c13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"
Sep 16 19:27:01.015193 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:01.015022    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5100699a-2aa4-4798-b8d8-37b9e5136c13-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99\" (UID: \"5100699a-2aa4-4798-b8d8-37b9e5136c13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"
Sep 16 19:27:01.015517 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:01.015422    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5100699a-2aa4-4798-b8d8-37b9e5136c13-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99\" (UID: \"5100699a-2aa4-4798-b8d8-37b9e5136c13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"
Sep 16 19:27:01.015517 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:01.015462    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5100699a-2aa4-4798-b8d8-37b9e5136c13-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99\" (UID: \"5100699a-2aa4-4798-b8d8-37b9e5136c13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"
Sep 16 19:27:01.015629 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:01.015540    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5100699a-2aa4-4798-b8d8-37b9e5136c13-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99\" (UID: \"5100699a-2aa4-4798-b8d8-37b9e5136c13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"
Sep 16 19:27:01.015719 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:01.015695    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5100699a-2aa4-4798-b8d8-37b9e5136c13-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99\" (UID: \"5100699a-2aa4-4798-b8d8-37b9e5136c13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"
Sep 16 19:27:01.015812 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:01.015796    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5100699a-2aa4-4798-b8d8-37b9e5136c13-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99\" (UID: \"5100699a-2aa4-4798-b8d8-37b9e5136c13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"
Sep 16 19:27:01.017405 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:01.017372    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5100699a-2aa4-4798-b8d8-37b9e5136c13-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99\" (UID: \"5100699a-2aa4-4798-b8d8-37b9e5136c13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"
Sep 16 19:27:01.017615 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:01.017596    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5100699a-2aa4-4798-b8d8-37b9e5136c13-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99\" (UID: \"5100699a-2aa4-4798-b8d8-37b9e5136c13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"
Sep 16 19:27:01.025144 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:01.025123    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppp6l\" (UniqueName: \"kubernetes.io/projected/5100699a-2aa4-4798-b8d8-37b9e5136c13-kube-api-access-ppp6l\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99\" (UID: \"5100699a-2aa4-4798-b8d8-37b9e5136c13\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"
Sep 16 19:27:01.093278 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:01.093190    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"
Sep 16 19:27:01.375635 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:01.375611    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"]
Sep 16 19:27:01.377720 ip-10-0-142-80 kubenswrapper[2578]: W0916 19:27:01.377692    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5100699a_2aa4_4798_b8d8_37b9e5136c13.slice/crio-80a6dffa9fc554a4366efc756b8f43812598a5cd3339fa09100dac6e78829108 WatchSource:0}: Error finding container 80a6dffa9fc554a4366efc756b8f43812598a5cd3339fa09100dac6e78829108: Status 404 returned error can't find the container with id 80a6dffa9fc554a4366efc756b8f43812598a5cd3339fa09100dac6e78829108
Sep 16 19:27:02.244373 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:02.243523    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99" event={"ID":"5100699a-2aa4-4798-b8d8-37b9e5136c13","Type":"ContainerStarted","Data":"f18a2e90c2a5aeebd7c10557857b044d3d63bc0c1b764d42d2d20400a87a7307"}
Sep 16 19:27:02.244373 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:02.243596    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99" event={"ID":"5100699a-2aa4-4798-b8d8-37b9e5136c13","Type":"ContainerStarted","Data":"80a6dffa9fc554a4366efc756b8f43812598a5cd3339fa09100dac6e78829108"}
Sep 16 19:27:05.263277 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:05.263240    2578 generic.go:356] "Generic (PLEG): container finished" podID="5100699a-2aa4-4798-b8d8-37b9e5136c13" containerID="f18a2e90c2a5aeebd7c10557857b044d3d63bc0c1b764d42d2d20400a87a7307" exitCode=0
Sep 16 19:27:05.263277 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:05.263278    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99" event={"ID":"5100699a-2aa4-4798-b8d8-37b9e5136c13","Type":"ContainerDied","Data":"f18a2e90c2a5aeebd7c10557857b044d3d63bc0c1b764d42d2d20400a87a7307"}
Sep 16 19:27:06.274221 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:06.274182    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99" event={"ID":"5100699a-2aa4-4798-b8d8-37b9e5136c13","Type":"ContainerStarted","Data":"7bc684e715e65b80efc22f5317d3415fbf2b27e8d29bc42a1456413c6852f4b0"}
Sep 16 19:27:06.304544 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:06.304491    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99" podStartSLOduration=3.436495818 podStartE2EDuration="6.304472342s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.867976524s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:27:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:27:06.300334812 +0000 UTC m=+2794.535268255" watchObservedRunningTime="2026-09-16 19:27:06.304472342 +0000 UTC m=+2794.539405785"
Sep 16 19:27:07.093729 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:07.093688    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"
Sep 16 19:27:11.093799 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:11.093755    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"
Sep 16 19:27:11.095563 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:11.095536    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99" podUID="5100699a-2aa4-4798-b8d8-37b9e5136c13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Sep 16 19:27:21.093885 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:21.093842    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99" podUID="5100699a-2aa4-4798-b8d8-37b9e5136c13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Sep 16 19:27:24.890961 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:24.890917    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"]
Sep 16 19:27:24.894772 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:24.894747    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:24.897673 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:24.897646    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Sep 16 19:27:24.897812 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:24.897719    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 16 19:27:24.897812 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:24.897791    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 16 19:27:24.897812 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:24.897725    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-khzrt\""
Sep 16 19:27:24.907352 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:24.907329    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"]
Sep 16 19:27:25.023186 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:25.023148    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srcp5\" (UniqueName: \"kubernetes.io/projected/7858bfec-206b-429a-a965-aacc60221ef4-kube-api-access-srcp5\") pod \"router-gateway-1-openshift-default-65765d8947-m4kpg\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:25.023186 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:25.023192    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7858bfec-206b-429a-a965-aacc60221ef4-workload-certs\") pod \"router-gateway-1-openshift-default-65765d8947-m4kpg\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:25.023450 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:25.023215    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7858bfec-206b-429a-a965-aacc60221ef4-istio-envoy\") pod \"router-gateway-1-openshift-default-65765d8947-m4kpg\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:25.023450 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:25.023272    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7858bfec-206b-429a-a965-aacc60221ef4-credential-socket\") pod \"router-gateway-1-openshift-default-65765d8947-m4kpg\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:25.023450 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:25.023415    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7858bfec-206b-429a-a965-aacc60221ef4-workload-socket\") pod \"router-gateway-1-openshift-default-65765d8947-m4kpg\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:25.023596 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:25.023465    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7858bfec-206b-429a-a965-aacc60221ef4-istio-token\") pod \"router-gateway-1-openshift-default-65765d8947-m4kpg\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:25.023596 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:25.023509    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7858bfec-206b-429a-a965-aacc60221ef4-istio-data\") pod \"router-gateway-1-openshift-default-65765d8947-m4kpg\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:25.023596 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:25.023532    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7858bfec-206b-429a-a965-aacc60221ef4-istio-podinfo\") pod \"router-gateway-1-openshift-default-65765d8947-m4kpg\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:25.023596 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:25.023555    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7858bfec-206b-429a-a965-aacc60221ef4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-65765d8947-m4kpg\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:25.124310 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:25.124260    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7858bfec-206b-429a-a965-aacc60221ef4-workload-socket\") pod \"router-gateway-1-openshift-default-65765d8947-m4kpg\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:25.124516 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:25.124321    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7858bfec-206b-429a-a965-aacc60221ef4-istio-token\") pod \"router-gateway-1-openshift-default-65765d8947-m4kpg\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:25.124516 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:25.124353    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7858bfec-206b-429a-a965-aacc60221ef4-istio-data\") pod \"router-gateway-1-openshift-default-65765d8947-m4kpg\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:25.124516 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:25.124385    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7858bfec-206b-429a-a965-aacc60221ef4-istio-podinfo\") pod \"router-gateway-1-openshift-default-65765d8947-m4kpg\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:25.124516 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:25.124409    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7858bfec-206b-429a-a965-aacc60221ef4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-65765d8947-m4kpg\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:25.124516 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:25.124469    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srcp5\" (UniqueName: \"kubernetes.io/projected/7858bfec-206b-429a-a965-aacc60221ef4-kube-api-access-srcp5\") pod \"router-gateway-1-openshift-default-65765d8947-m4kpg\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:25.124516 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:25.124504    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7858bfec-206b-429a-a965-aacc60221ef4-workload-certs\") pod \"router-gateway-1-openshift-default-65765d8947-m4kpg\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:25.124839 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:25.124532    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7858bfec-206b-429a-a965-aacc60221ef4-istio-envoy\") pod \"router-gateway-1-openshift-default-65765d8947-m4kpg\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:25.124839 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:25.124560    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7858bfec-206b-429a-a965-aacc60221ef4-credential-socket\") pod \"router-gateway-1-openshift-default-65765d8947-m4kpg\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:25.124839 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:25.124733    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7858bfec-206b-429a-a965-aacc60221ef4-workload-socket\") pod \"router-gateway-1-openshift-default-65765d8947-m4kpg\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:25.125005 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:25.124915    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7858bfec-206b-429a-a965-aacc60221ef4-credential-socket\") pod \"router-gateway-1-openshift-default-65765d8947-m4kpg\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:25.125089 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:25.125015    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7858bfec-206b-429a-a965-aacc60221ef4-workload-certs\") pod \"router-gateway-1-openshift-default-65765d8947-m4kpg\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:25.125163 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:25.125107    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7858bfec-206b-429a-a965-aacc60221ef4-istio-data\") pod \"router-gateway-1-openshift-default-65765d8947-m4kpg\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:25.125530 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:25.125504    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7858bfec-206b-429a-a965-aacc60221ef4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-65765d8947-m4kpg\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:25.127477 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:25.127454    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7858bfec-206b-429a-a965-aacc60221ef4-istio-podinfo\") pod \"router-gateway-1-openshift-default-65765d8947-m4kpg\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:25.127596 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:25.127478    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7858bfec-206b-429a-a965-aacc60221ef4-istio-envoy\") pod \"router-gateway-1-openshift-default-65765d8947-m4kpg\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:25.134384 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:25.134357    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7858bfec-206b-429a-a965-aacc60221ef4-istio-token\") pod \"router-gateway-1-openshift-default-65765d8947-m4kpg\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:25.134777 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:25.134751    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srcp5\" (UniqueName: \"kubernetes.io/projected/7858bfec-206b-429a-a965-aacc60221ef4-kube-api-access-srcp5\") pod \"router-gateway-1-openshift-default-65765d8947-m4kpg\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:25.207018 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:25.206978    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:25.502816 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:25.502785    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"]
Sep 16 19:27:25.505895 ip-10-0-142-80 kubenswrapper[2578]: W0916 19:27:25.505859    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7858bfec_206b_429a_a965_aacc60221ef4.slice/crio-b5044b87f905a1e96abeb19de5de5c19d06c556a2b87bea9047d5bd9b34a833c WatchSource:0}: Error finding container b5044b87f905a1e96abeb19de5de5c19d06c556a2b87bea9047d5bd9b34a833c: Status 404 returned error can't find the container with id b5044b87f905a1e96abeb19de5de5c19d06c556a2b87bea9047d5bd9b34a833c
Sep 16 19:27:25.508334 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:25.508293    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 19:27:25.508439 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:25.508403    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 19:27:25.508503 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:25.508446    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 19:27:26.384498 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:26.384454    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg" event={"ID":"7858bfec-206b-429a-a965-aacc60221ef4","Type":"ContainerStarted","Data":"a5eeb7416241d30ba125d9ff7ae1b4a41f47cb3bce3235e5babf2d7244847d44"}
Sep 16 19:27:26.384969 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:26.384514    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg" event={"ID":"7858bfec-206b-429a-a965-aacc60221ef4","Type":"ContainerStarted","Data":"b5044b87f905a1e96abeb19de5de5c19d06c556a2b87bea9047d5bd9b34a833c"}
Sep 16 19:27:26.409142 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:26.409084    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg" podStartSLOduration=2.409064872 podStartE2EDuration="2.409064872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:27:26.406815747 +0000 UTC m=+2814.641749191" watchObservedRunningTime="2026-09-16 19:27:26.409064872 +0000 UTC m=+2814.643998317"
Sep 16 19:27:27.207809 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:27.207774    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:27.212964 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:27.212934    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:27.390063 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:27.390020    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:27.391017 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:27.390996    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:31.094001 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:31.093955    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99" podUID="5100699a-2aa4-4798-b8d8-37b9e5136c13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Sep 16 19:27:41.094294 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:41.094248    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99" podUID="5100699a-2aa4-4798-b8d8-37b9e5136c13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Sep 16 19:27:46.499900 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:46.499855    2578 generic.go:356] "Generic (PLEG): container finished" podID="7858bfec-206b-429a-a965-aacc60221ef4" containerID="a5eeb7416241d30ba125d9ff7ae1b4a41f47cb3bce3235e5babf2d7244847d44" exitCode=0
Sep 16 19:27:46.500409 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:46.499925    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg" event={"ID":"7858bfec-206b-429a-a965-aacc60221ef4","Type":"ContainerDied","Data":"a5eeb7416241d30ba125d9ff7ae1b4a41f47cb3bce3235e5babf2d7244847d44"}
Sep 16 19:27:46.500409 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:46.500328    2578 scope.go:117] "RemoveContainer" containerID="a5eeb7416241d30ba125d9ff7ae1b4a41f47cb3bce3235e5babf2d7244847d44"
Sep 16 19:27:46.501309 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:46.501273    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 19:27:46.501407 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:46.501351    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 19:27:46.501407 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:46.501393    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 19:27:47.506658 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:47.506622    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg" event={"ID":"7858bfec-206b-429a-a965-aacc60221ef4","Type":"ContainerStarted","Data":"f34410e5c1f75e40d96c0b785b7e7e981537b9872a7da9b66c7242b00359b2e8"}
Sep 16 19:27:48.208217 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:48.208179    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:48.208397 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:48.208376    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg" podUID="7858bfec-206b-429a-a965-aacc60221ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 16 19:27:49.207981 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:49.207933    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg" podUID="7858bfec-206b-429a-a965-aacc60221ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 16 19:27:50.208481 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:50.208424    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg" podUID="7858bfec-206b-429a-a965-aacc60221ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 16 19:27:51.094564 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:51.094522    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99" podUID="5100699a-2aa4-4798-b8d8-37b9e5136c13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Sep 16 19:27:51.207610 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:51.207561    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg" podUID="7858bfec-206b-429a-a965-aacc60221ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 16 19:27:52.208233 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:52.208194    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg" podUID="7858bfec-206b-429a-a965-aacc60221ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 16 19:27:53.211873 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:53.211841    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:53.212282 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:53.212268    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:27:53.212938 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:27:53.212914    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:28:01.094325 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:28:01.094277    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99" podUID="5100699a-2aa4-4798-b8d8-37b9e5136c13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Sep 16 19:28:11.093804 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:28:11.093758    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99" podUID="5100699a-2aa4-4798-b8d8-37b9e5136c13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Sep 16 19:28:21.104189 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:28:21.104159    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"
Sep 16 19:28:21.111746 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:28:21.111707    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"
Sep 16 19:28:46.799709 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:28:46.799625    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"]
Sep 16 19:29:01.823597 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:01.823554    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99" podUID="5100699a-2aa4-4798-b8d8-37b9e5136c13" containerName="main" containerID="cri-o://7bc684e715e65b80efc22f5317d3415fbf2b27e8d29bc42a1456413c6852f4b0" gracePeriod=45
Sep 16 19:29:02.104984 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.104871    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99" podUID="5100699a-2aa4-4798-b8d8-37b9e5136c13" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Sep 16 19:29:02.471162 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.471138    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"
Sep 16 19:29:02.564162 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.564134    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5100699a-2aa4-4798-b8d8-37b9e5136c13-dshm\") pod \"5100699a-2aa4-4798-b8d8-37b9e5136c13\" (UID: \"5100699a-2aa4-4798-b8d8-37b9e5136c13\") "
Sep 16 19:29:02.564307 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.564178    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5100699a-2aa4-4798-b8d8-37b9e5136c13-tmp-dir\") pod \"5100699a-2aa4-4798-b8d8-37b9e5136c13\" (UID: \"5100699a-2aa4-4798-b8d8-37b9e5136c13\") "
Sep 16 19:29:02.564307 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.564197    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5100699a-2aa4-4798-b8d8-37b9e5136c13-kserve-provision-location\") pod \"5100699a-2aa4-4798-b8d8-37b9e5136c13\" (UID: \"5100699a-2aa4-4798-b8d8-37b9e5136c13\") "
Sep 16 19:29:02.564307 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.564240    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5100699a-2aa4-4798-b8d8-37b9e5136c13-tls-certs\") pod \"5100699a-2aa4-4798-b8d8-37b9e5136c13\" (UID: \"5100699a-2aa4-4798-b8d8-37b9e5136c13\") "
Sep 16 19:29:02.564307 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.564260    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5100699a-2aa4-4798-b8d8-37b9e5136c13-cabundle-cert\") pod \"5100699a-2aa4-4798-b8d8-37b9e5136c13\" (UID: \"5100699a-2aa4-4798-b8d8-37b9e5136c13\") "
Sep 16 19:29:02.564307 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.564299    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5100699a-2aa4-4798-b8d8-37b9e5136c13-model-cache\") pod \"5100699a-2aa4-4798-b8d8-37b9e5136c13\" (UID: \"5100699a-2aa4-4798-b8d8-37b9e5136c13\") "
Sep 16 19:29:02.564598 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.564320    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppp6l\" (UniqueName: \"kubernetes.io/projected/5100699a-2aa4-4798-b8d8-37b9e5136c13-kube-api-access-ppp6l\") pod \"5100699a-2aa4-4798-b8d8-37b9e5136c13\" (UID: \"5100699a-2aa4-4798-b8d8-37b9e5136c13\") "
Sep 16 19:29:02.564598 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.564545    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5100699a-2aa4-4798-b8d8-37b9e5136c13-home\") pod \"5100699a-2aa4-4798-b8d8-37b9e5136c13\" (UID: \"5100699a-2aa4-4798-b8d8-37b9e5136c13\") "
Sep 16 19:29:02.565078 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.564993    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5100699a-2aa4-4798-b8d8-37b9e5136c13-model-cache" (OuterVolumeSpecName: "model-cache") pod "5100699a-2aa4-4798-b8d8-37b9e5136c13" (UID: "5100699a-2aa4-4798-b8d8-37b9e5136c13"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:02.565210 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.565174    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5100699a-2aa4-4798-b8d8-37b9e5136c13-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5100699a-2aa4-4798-b8d8-37b9e5136c13" (UID: "5100699a-2aa4-4798-b8d8-37b9e5136c13"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:29:02.565658 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.565630    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5100699a-2aa4-4798-b8d8-37b9e5136c13-home" (OuterVolumeSpecName: "home") pod "5100699a-2aa4-4798-b8d8-37b9e5136c13" (UID: "5100699a-2aa4-4798-b8d8-37b9e5136c13"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:02.566688 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.566665    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5100699a-2aa4-4798-b8d8-37b9e5136c13-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5100699a-2aa4-4798-b8d8-37b9e5136c13" (UID: "5100699a-2aa4-4798-b8d8-37b9e5136c13"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:29:02.566848 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.566831    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5100699a-2aa4-4798-b8d8-37b9e5136c13-dshm" (OuterVolumeSpecName: "dshm") pod "5100699a-2aa4-4798-b8d8-37b9e5136c13" (UID: "5100699a-2aa4-4798-b8d8-37b9e5136c13"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:02.566979 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.566964    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5100699a-2aa4-4798-b8d8-37b9e5136c13-kube-api-access-ppp6l" (OuterVolumeSpecName: "kube-api-access-ppp6l") pod "5100699a-2aa4-4798-b8d8-37b9e5136c13" (UID: "5100699a-2aa4-4798-b8d8-37b9e5136c13"). InnerVolumeSpecName "kube-api-access-ppp6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:29:02.580690 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.580653    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5100699a-2aa4-4798-b8d8-37b9e5136c13-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5100699a-2aa4-4798-b8d8-37b9e5136c13" (UID: "5100699a-2aa4-4798-b8d8-37b9e5136c13"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:02.587126 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.587105    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5100699a-2aa4-4798-b8d8-37b9e5136c13-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5100699a-2aa4-4798-b8d8-37b9e5136c13" (UID: "5100699a-2aa4-4798-b8d8-37b9e5136c13"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:02.665680 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.665619    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5100699a-2aa4-4798-b8d8-37b9e5136c13-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:29:02.665680 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.665642    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5100699a-2aa4-4798-b8d8-37b9e5136c13-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:29:02.665680 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.665653    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5100699a-2aa4-4798-b8d8-37b9e5136c13-kserve-provision-location\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:29:02.665680 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.665662    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5100699a-2aa4-4798-b8d8-37b9e5136c13-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:29:02.665680 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.665671    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5100699a-2aa4-4798-b8d8-37b9e5136c13-cabundle-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:29:02.665680 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.665678    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5100699a-2aa4-4798-b8d8-37b9e5136c13-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:29:02.665952 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.665686    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppp6l\" (UniqueName: \"kubernetes.io/projected/5100699a-2aa4-4798-b8d8-37b9e5136c13-kube-api-access-ppp6l\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:29:02.665952 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.665694    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5100699a-2aa4-4798-b8d8-37b9e5136c13-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:29:02.909647 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.909610    2578 generic.go:356] "Generic (PLEG): container finished" podID="5100699a-2aa4-4798-b8d8-37b9e5136c13" containerID="7bc684e715e65b80efc22f5317d3415fbf2b27e8d29bc42a1456413c6852f4b0" exitCode=0
Sep 16 19:29:02.910148 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.909664    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"
Sep 16 19:29:02.910148 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.909697    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99" event={"ID":"5100699a-2aa4-4798-b8d8-37b9e5136c13","Type":"ContainerDied","Data":"7bc684e715e65b80efc22f5317d3415fbf2b27e8d29bc42a1456413c6852f4b0"}
Sep 16 19:29:02.910148 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.909767    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99" event={"ID":"5100699a-2aa4-4798-b8d8-37b9e5136c13","Type":"ContainerDied","Data":"80a6dffa9fc554a4366efc756b8f43812598a5cd3339fa09100dac6e78829108"}
Sep 16 19:29:02.910148 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.909803    2578 scope.go:117] "RemoveContainer" containerID="7bc684e715e65b80efc22f5317d3415fbf2b27e8d29bc42a1456413c6852f4b0"
Sep 16 19:29:02.918695 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.918676    2578 scope.go:117] "RemoveContainer" containerID="f18a2e90c2a5aeebd7c10557857b044d3d63bc0c1b764d42d2d20400a87a7307"
Sep 16 19:29:02.926629 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.926611    2578 scope.go:117] "RemoveContainer" containerID="7bc684e715e65b80efc22f5317d3415fbf2b27e8d29bc42a1456413c6852f4b0"
Sep 16 19:29:02.926910 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:29:02.926891    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bc684e715e65b80efc22f5317d3415fbf2b27e8d29bc42a1456413c6852f4b0\": container with ID starting with 7bc684e715e65b80efc22f5317d3415fbf2b27e8d29bc42a1456413c6852f4b0 not found: ID does not exist" containerID="7bc684e715e65b80efc22f5317d3415fbf2b27e8d29bc42a1456413c6852f4b0"
Sep 16 19:29:02.926964 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.926919    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc684e715e65b80efc22f5317d3415fbf2b27e8d29bc42a1456413c6852f4b0"} err="failed to get container status \"7bc684e715e65b80efc22f5317d3415fbf2b27e8d29bc42a1456413c6852f4b0\": rpc error: code = NotFound desc = could not find container \"7bc684e715e65b80efc22f5317d3415fbf2b27e8d29bc42a1456413c6852f4b0\": container with ID starting with 7bc684e715e65b80efc22f5317d3415fbf2b27e8d29bc42a1456413c6852f4b0 not found: ID does not exist"
Sep 16 19:29:02.926964 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.926935    2578 scope.go:117] "RemoveContainer" containerID="f18a2e90c2a5aeebd7c10557857b044d3d63bc0c1b764d42d2d20400a87a7307"
Sep 16 19:29:02.927287 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:29:02.927267    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f18a2e90c2a5aeebd7c10557857b044d3d63bc0c1b764d42d2d20400a87a7307\": container with ID starting with f18a2e90c2a5aeebd7c10557857b044d3d63bc0c1b764d42d2d20400a87a7307 not found: ID does not exist" containerID="f18a2e90c2a5aeebd7c10557857b044d3d63bc0c1b764d42d2d20400a87a7307"
Sep 16 19:29:02.927354 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.927293    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f18a2e90c2a5aeebd7c10557857b044d3d63bc0c1b764d42d2d20400a87a7307"} err="failed to get container status \"f18a2e90c2a5aeebd7c10557857b044d3d63bc0c1b764d42d2d20400a87a7307\": rpc error: code = NotFound desc = could not find container \"f18a2e90c2a5aeebd7c10557857b044d3d63bc0c1b764d42d2d20400a87a7307\": container with ID starting with f18a2e90c2a5aeebd7c10557857b044d3d63bc0c1b764d42d2d20400a87a7307 not found: ID does not exist"
Sep 16 19:29:02.936840 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.936817    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"]
Sep 16 19:29:02.939244 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:02.939222    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bd956c69-4bh99"]
Sep 16 19:29:04.217409 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:04.217375    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5100699a-2aa4-4798-b8d8-37b9e5136c13" path="/var/lib/kubelet/pods/5100699a-2aa4-4798-b8d8-37b9e5136c13/volumes"
Sep 16 19:29:45.474569 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.474532    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"]
Sep 16 19:29:45.475179 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.474871    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5100699a-2aa4-4798-b8d8-37b9e5136c13" containerName="main"
Sep 16 19:29:45.475179 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.474883    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5100699a-2aa4-4798-b8d8-37b9e5136c13" containerName="main"
Sep 16 19:29:45.475179 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.474892    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5100699a-2aa4-4798-b8d8-37b9e5136c13" containerName="storage-initializer"
Sep 16 19:29:45.475179 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.474898    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5100699a-2aa4-4798-b8d8-37b9e5136c13" containerName="storage-initializer"
Sep 16 19:29:45.475179 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.474964    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5100699a-2aa4-4798-b8d8-37b9e5136c13" containerName="main"
Sep 16 19:29:45.477987 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.477964    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"
Sep 16 19:29:45.481083 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.481059    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 16 19:29:45.481241 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.481098    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 16 19:29:45.481321 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.481128    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:29:45.481379 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.481147    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-jwfgz\""
Sep 16 19:29:45.481437 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.481163    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 16 19:29:45.481437 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.481186    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 16 19:29:45.490614 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.490589    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"]
Sep 16 19:29:45.522587 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.522539    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85\" (UID: \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"
Sep 16 19:29:45.522787 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.522604    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp55z\" (UniqueName: \"kubernetes.io/projected/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-kube-api-access-xp55z\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85\" (UID: \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"
Sep 16 19:29:45.522787 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.522637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85\" (UID: \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"
Sep 16 19:29:45.522915 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.522739    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85\" (UID: \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"
Sep 16 19:29:45.523010 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.522992    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85\" (UID: \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"
Sep 16 19:29:45.523120 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.523023    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85\" (UID: \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"
Sep 16 19:29:45.523120 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.523088    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85\" (UID: \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"
Sep 16 19:29:45.523222 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.523118    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85\" (UID: \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"
Sep 16 19:29:45.623974 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.623630    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85\" (UID: \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"
Sep 16 19:29:45.623974 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.623790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp55z\" (UniqueName: \"kubernetes.io/projected/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-kube-api-access-xp55z\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85\" (UID: \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"
Sep 16 19:29:45.623974 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.623828    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85\" (UID: \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"
Sep 16 19:29:45.623974 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.623860    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85\" (UID: \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"
Sep 16 19:29:45.623974 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.623901    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85\" (UID: \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"
Sep 16 19:29:45.623974 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.623936    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85\" (UID: \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"
Sep 16 19:29:45.624465 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.624006    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85\" (UID: \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"
Sep 16 19:29:45.624465 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.624066    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85\" (UID: \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"
Sep 16 19:29:45.624465 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.624414    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85\" (UID: \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"
Sep 16 19:29:45.624623 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.624488    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85\" (UID: \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"
Sep 16 19:29:45.624985 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.624959    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85\" (UID: \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"
Sep 16 19:29:45.625138 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.625114    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85\" (UID: \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"
Sep 16 19:29:45.625214 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.625152    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85\" (UID: \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"
Sep 16 19:29:45.626974 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.626943    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85\" (UID: \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"
Sep 16 19:29:45.627105 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.626987    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85\" (UID: \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"
Sep 16 19:29:45.634399 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.634369    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp55z\" (UniqueName: \"kubernetes.io/projected/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-kube-api-access-xp55z\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85\" (UID: \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"
Sep 16 19:29:45.788995 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:45.788895    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"
Sep 16 19:29:46.075445 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:46.075335    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"]
Sep 16 19:29:46.077764 ip-10-0-142-80 kubenswrapper[2578]: W0916 19:29:46.077736    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e45fe09_21e1_4ff7_81ef_21ef8efdab62.slice/crio-9a5c89d5a9ec99776ca9c486b4dbdc8c3d32b155fd989d39c6c1b4dad5ef2fe3 WatchSource:0}: Error finding container 9a5c89d5a9ec99776ca9c486b4dbdc8c3d32b155fd989d39c6c1b4dad5ef2fe3: Status 404 returned error can't find the container with id 9a5c89d5a9ec99776ca9c486b4dbdc8c3d32b155fd989d39c6c1b4dad5ef2fe3
Sep 16 19:29:46.126658 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:46.126621    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85" event={"ID":"8e45fe09-21e1-4ff7-81ef-21ef8efdab62","Type":"ContainerStarted","Data":"9a5c89d5a9ec99776ca9c486b4dbdc8c3d32b155fd989d39c6c1b4dad5ef2fe3"}
Sep 16 19:29:47.139065 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:47.138200    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85" event={"ID":"8e45fe09-21e1-4ff7-81ef-21ef8efdab62","Type":"ContainerStarted","Data":"6633ebb4696d9f0f7f1764dfb443408181882750cc6058305f31f7e8c8617107"}
Sep 16 19:29:50.159159 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:50.159117    2578 generic.go:356] "Generic (PLEG): container finished" podID="8e45fe09-21e1-4ff7-81ef-21ef8efdab62" containerID="6633ebb4696d9f0f7f1764dfb443408181882750cc6058305f31f7e8c8617107" exitCode=0
Sep 16 19:29:50.159598 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:50.159166    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85" event={"ID":"8e45fe09-21e1-4ff7-81ef-21ef8efdab62","Type":"ContainerDied","Data":"6633ebb4696d9f0f7f1764dfb443408181882750cc6058305f31f7e8c8617107"}
Sep 16 19:29:51.167938 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:51.167901    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85" event={"ID":"8e45fe09-21e1-4ff7-81ef-21ef8efdab62","Type":"ContainerStarted","Data":"4606d265dcb1b9126b48c9b6e747dad15eaff1358e9ca3541e16a7ce316e0d1d"}
Sep 16 19:29:51.193944 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:51.193892    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85" podStartSLOduration=2.959184462 podStartE2EDuration="6.193873516s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.234689054s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:29:51.192330238 +0000 UTC m=+2959.427263706" watchObservedRunningTime="2026-09-16 19:29:51.193873516 +0000 UTC m=+2959.428806960"
Sep 16 19:29:51.789646 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:51.789602    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"
Sep 16 19:29:55.789142 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:55.789030    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"
Sep 16 19:29:55.790851 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:29:55.790808    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85" podUID="8e45fe09-21e1-4ff7-81ef-21ef8efdab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 16 19:30:05.789354 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:05.789302    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85" podUID="8e45fe09-21e1-4ff7-81ef-21ef8efdab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 16 19:30:08.113634 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:08.113582    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"]
Sep 16 19:30:08.114116 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:08.113798    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg" podUID="7858bfec-206b-429a-a965-aacc60221ef4" containerName="istio-proxy" containerID="cri-o://f34410e5c1f75e40d96c0b785b7e7e981537b9872a7da9b66c7242b00359b2e8" gracePeriod=30
Sep 16 19:30:08.215129 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:08.215088    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg" podUID="7858bfec-206b-429a-a965-aacc60221ef4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:30:13.292707 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:13.292672    2578 generic.go:356] "Generic (PLEG): container finished" podID="7858bfec-206b-429a-a965-aacc60221ef4" containerID="f34410e5c1f75e40d96c0b785b7e7e981537b9872a7da9b66c7242b00359b2e8" exitCode=0
Sep 16 19:30:13.293122 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:13.292741    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg" event={"ID":"7858bfec-206b-429a-a965-aacc60221ef4","Type":"ContainerDied","Data":"f34410e5c1f75e40d96c0b785b7e7e981537b9872a7da9b66c7242b00359b2e8"}
Sep 16 19:30:13.293122 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:13.292798    2578 scope.go:117] "RemoveContainer" containerID="a5eeb7416241d30ba125d9ff7ae1b4a41f47cb3bce3235e5babf2d7244847d44"
Sep 16 19:30:13.366813 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:13.366778    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:30:13.504200 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:13.504166    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7858bfec-206b-429a-a965-aacc60221ef4-workload-socket\") pod \"7858bfec-206b-429a-a965-aacc60221ef4\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") "
Sep 16 19:30:13.504200 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:13.504208    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7858bfec-206b-429a-a965-aacc60221ef4-istiod-ca-cert\") pod \"7858bfec-206b-429a-a965-aacc60221ef4\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") "
Sep 16 19:30:13.504446 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:13.504233    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7858bfec-206b-429a-a965-aacc60221ef4-istio-podinfo\") pod \"7858bfec-206b-429a-a965-aacc60221ef4\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") "
Sep 16 19:30:13.504446 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:13.504282    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7858bfec-206b-429a-a965-aacc60221ef4-istio-data\") pod \"7858bfec-206b-429a-a965-aacc60221ef4\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") "
Sep 16 19:30:13.504446 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:13.504334    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srcp5\" (UniqueName: \"kubernetes.io/projected/7858bfec-206b-429a-a965-aacc60221ef4-kube-api-access-srcp5\") pod \"7858bfec-206b-429a-a965-aacc60221ef4\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") "
Sep 16 19:30:13.504446 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:13.504355    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7858bfec-206b-429a-a965-aacc60221ef4-credential-socket\") pod \"7858bfec-206b-429a-a965-aacc60221ef4\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") "
Sep 16 19:30:13.504446 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:13.504393    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7858bfec-206b-429a-a965-aacc60221ef4-istio-token\") pod \"7858bfec-206b-429a-a965-aacc60221ef4\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") "
Sep 16 19:30:13.504699 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:13.504456    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7858bfec-206b-429a-a965-aacc60221ef4-istio-envoy\") pod \"7858bfec-206b-429a-a965-aacc60221ef4\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") "
Sep 16 19:30:13.504699 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:13.504476    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7858bfec-206b-429a-a965-aacc60221ef4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7858bfec-206b-429a-a965-aacc60221ef4" (UID: "7858bfec-206b-429a-a965-aacc60221ef4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:13.504699 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:13.504517    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7858bfec-206b-429a-a965-aacc60221ef4-workload-certs\") pod \"7858bfec-206b-429a-a965-aacc60221ef4\" (UID: \"7858bfec-206b-429a-a965-aacc60221ef4\") "
Sep 16 19:30:13.504699 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:13.504604    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7858bfec-206b-429a-a965-aacc60221ef4-istio-data" (OuterVolumeSpecName: "istio-data") pod "7858bfec-206b-429a-a965-aacc60221ef4" (UID: "7858bfec-206b-429a-a965-aacc60221ef4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:13.504699 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:13.504620    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7858bfec-206b-429a-a965-aacc60221ef4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7858bfec-206b-429a-a965-aacc60221ef4" (UID: "7858bfec-206b-429a-a965-aacc60221ef4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:30:13.504957 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:13.504789    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7858bfec-206b-429a-a965-aacc60221ef4-istio-data\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:30:13.504957 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:13.504808    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7858bfec-206b-429a-a965-aacc60221ef4-workload-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:30:13.504957 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:13.504824    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7858bfec-206b-429a-a965-aacc60221ef4-istiod-ca-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:30:13.504957 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:13.504837    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7858bfec-206b-429a-a965-aacc60221ef4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7858bfec-206b-429a-a965-aacc60221ef4" (UID: "7858bfec-206b-429a-a965-aacc60221ef4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:13.504957 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:13.504854    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7858bfec-206b-429a-a965-aacc60221ef4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7858bfec-206b-429a-a965-aacc60221ef4" (UID: "7858bfec-206b-429a-a965-aacc60221ef4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:13.506764 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:13.506735    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7858bfec-206b-429a-a965-aacc60221ef4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7858bfec-206b-429a-a965-aacc60221ef4" (UID: "7858bfec-206b-429a-a965-aacc60221ef4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:13.506893 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:13.506837    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7858bfec-206b-429a-a965-aacc60221ef4-istio-token" (OuterVolumeSpecName: "istio-token") pod "7858bfec-206b-429a-a965-aacc60221ef4" (UID: "7858bfec-206b-429a-a965-aacc60221ef4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:30:13.506893 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:13.506878    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7858bfec-206b-429a-a965-aacc60221ef4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7858bfec-206b-429a-a965-aacc60221ef4" (UID: "7858bfec-206b-429a-a965-aacc60221ef4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 19:30:13.506986 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:13.506887    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7858bfec-206b-429a-a965-aacc60221ef4-kube-api-access-srcp5" (OuterVolumeSpecName: "kube-api-access-srcp5") pod "7858bfec-206b-429a-a965-aacc60221ef4" (UID: "7858bfec-206b-429a-a965-aacc60221ef4"). InnerVolumeSpecName "kube-api-access-srcp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:30:13.605733 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:13.605688    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7858bfec-206b-429a-a965-aacc60221ef4-istio-envoy\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:30:13.605733 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:13.605725    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7858bfec-206b-429a-a965-aacc60221ef4-workload-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:30:13.605733 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:13.605738    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7858bfec-206b-429a-a965-aacc60221ef4-istio-podinfo\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:30:13.605986 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:13.605750    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srcp5\" (UniqueName: \"kubernetes.io/projected/7858bfec-206b-429a-a965-aacc60221ef4-kube-api-access-srcp5\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:30:13.605986 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:13.605763    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7858bfec-206b-429a-a965-aacc60221ef4-credential-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:30:13.605986 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:13.605776    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7858bfec-206b-429a-a965-aacc60221ef4-istio-token\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:30:14.299760 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:14.299731    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"
Sep 16 19:30:14.300250 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:14.299769    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg" event={"ID":"7858bfec-206b-429a-a965-aacc60221ef4","Type":"ContainerDied","Data":"b5044b87f905a1e96abeb19de5de5c19d06c556a2b87bea9047d5bd9b34a833c"}
Sep 16 19:30:14.300250 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:14.299834    2578 scope.go:117] "RemoveContainer" containerID="f34410e5c1f75e40d96c0b785b7e7e981537b9872a7da9b66c7242b00359b2e8"
Sep 16 19:30:14.324798 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:14.324762    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"]
Sep 16 19:30:14.329624 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:14.329594    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-m4kpg"]
Sep 16 19:30:15.789890 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:15.789848    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85" podUID="8e45fe09-21e1-4ff7-81ef-21ef8efdab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 16 19:30:16.218215 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:16.218178    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7858bfec-206b-429a-a965-aacc60221ef4" path="/var/lib/kubelet/pods/7858bfec-206b-429a-a965-aacc60221ef4/volumes"
Sep 16 19:30:25.789646 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:25.789593    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85" podUID="8e45fe09-21e1-4ff7-81ef-21ef8efdab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 16 19:30:35.789892 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:35.789843    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85" podUID="8e45fe09-21e1-4ff7-81ef-21ef8efdab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 16 19:30:45.789394 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:45.789344    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85" podUID="8e45fe09-21e1-4ff7-81ef-21ef8efdab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 16 19:30:55.789397 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:30:55.789339    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85" podUID="8e45fe09-21e1-4ff7-81ef-21ef8efdab62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 16 19:31:05.799545 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:05.799509    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"
Sep 16 19:31:05.807627 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:05.807604    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"
Sep 16 19:31:07.749698 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:07.749662    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"]
Sep 16 19:31:22.772349 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:22.772301    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85" podUID="8e45fe09-21e1-4ff7-81ef-21ef8efdab62" containerName="main" containerID="cri-o://4606d265dcb1b9126b48c9b6e747dad15eaff1358e9ca3541e16a7ce316e0d1d" gracePeriod=45
Sep 16 19:31:23.334431 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.334400    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"
Sep 16 19:31:23.394199 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.394169    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-model-cache\") pod \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\" (UID: \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\") "
Sep 16 19:31:23.394199 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.394204    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-home\") pod \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\" (UID: \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\") "
Sep 16 19:31:23.394463 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.394260    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-tmp-dir\") pod \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\" (UID: \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\") "
Sep 16 19:31:23.394463 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.394277    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xp55z\" (UniqueName: \"kubernetes.io/projected/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-kube-api-access-xp55z\") pod \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\" (UID: \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\") "
Sep 16 19:31:23.394463 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.394297    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-cabundle-cert\") pod \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\" (UID: \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\") "
Sep 16 19:31:23.394463 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.394313    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-tls-certs\") pod \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\" (UID: \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\") "
Sep 16 19:31:23.394463 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.394332    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-dshm\") pod \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\" (UID: \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\") "
Sep 16 19:31:23.394463 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.394349    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-kserve-provision-location\") pod \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\" (UID: \"8e45fe09-21e1-4ff7-81ef-21ef8efdab62\") "
Sep 16 19:31:23.394463 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.394445    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-model-cache" (OuterVolumeSpecName: "model-cache") pod "8e45fe09-21e1-4ff7-81ef-21ef8efdab62" (UID: "8e45fe09-21e1-4ff7-81ef-21ef8efdab62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:23.394805 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.394610    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:31:23.394805 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.394623    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8e45fe09-21e1-4ff7-81ef-21ef8efdab62" (UID: "8e45fe09-21e1-4ff7-81ef-21ef8efdab62"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:31:23.394805 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.394727    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-home" (OuterVolumeSpecName: "home") pod "8e45fe09-21e1-4ff7-81ef-21ef8efdab62" (UID: "8e45fe09-21e1-4ff7-81ef-21ef8efdab62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:23.394937 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.394820    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8e45fe09-21e1-4ff7-81ef-21ef8efdab62" (UID: "8e45fe09-21e1-4ff7-81ef-21ef8efdab62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:23.396672 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.396647    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8e45fe09-21e1-4ff7-81ef-21ef8efdab62" (UID: "8e45fe09-21e1-4ff7-81ef-21ef8efdab62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:31:23.396826 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.396798    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-dshm" (OuterVolumeSpecName: "dshm") pod "8e45fe09-21e1-4ff7-81ef-21ef8efdab62" (UID: "8e45fe09-21e1-4ff7-81ef-21ef8efdab62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:23.396949 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.396868    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-kube-api-access-xp55z" (OuterVolumeSpecName: "kube-api-access-xp55z") pod "8e45fe09-21e1-4ff7-81ef-21ef8efdab62" (UID: "8e45fe09-21e1-4ff7-81ef-21ef8efdab62"). InnerVolumeSpecName "kube-api-access-xp55z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:31:23.417773 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.417694    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8e45fe09-21e1-4ff7-81ef-21ef8efdab62" (UID: "8e45fe09-21e1-4ff7-81ef-21ef8efdab62"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:23.495415 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.495374    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:31:23.495415 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.495409    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:31:23.495415 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.495420    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp55z\" (UniqueName: \"kubernetes.io/projected/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-kube-api-access-xp55z\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:31:23.495658 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.495429    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-cabundle-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:31:23.495658 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.495437    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:31:23.495658 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.495446    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:31:23.495658 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.495456    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e45fe09-21e1-4ff7-81ef-21ef8efdab62-kserve-provision-location\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:31:23.679804 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.679765    2578 generic.go:356] "Generic (PLEG): container finished" podID="8e45fe09-21e1-4ff7-81ef-21ef8efdab62" containerID="4606d265dcb1b9126b48c9b6e747dad15eaff1358e9ca3541e16a7ce316e0d1d" exitCode=0
Sep 16 19:31:23.679974 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.679825    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"
Sep 16 19:31:23.679974 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.679833    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85" event={"ID":"8e45fe09-21e1-4ff7-81ef-21ef8efdab62","Type":"ContainerDied","Data":"4606d265dcb1b9126b48c9b6e747dad15eaff1358e9ca3541e16a7ce316e0d1d"}
Sep 16 19:31:23.679974 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.679890    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85" event={"ID":"8e45fe09-21e1-4ff7-81ef-21ef8efdab62","Type":"ContainerDied","Data":"9a5c89d5a9ec99776ca9c486b4dbdc8c3d32b155fd989d39c6c1b4dad5ef2fe3"}
Sep 16 19:31:23.679974 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.679921    2578 scope.go:117] "RemoveContainer" containerID="4606d265dcb1b9126b48c9b6e747dad15eaff1358e9ca3541e16a7ce316e0d1d"
Sep 16 19:31:23.689200 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.689176    2578 scope.go:117] "RemoveContainer" containerID="6633ebb4696d9f0f7f1764dfb443408181882750cc6058305f31f7e8c8617107"
Sep 16 19:31:23.697806 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.697784    2578 scope.go:117] "RemoveContainer" containerID="4606d265dcb1b9126b48c9b6e747dad15eaff1358e9ca3541e16a7ce316e0d1d"
Sep 16 19:31:23.698157 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:31:23.698128    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4606d265dcb1b9126b48c9b6e747dad15eaff1358e9ca3541e16a7ce316e0d1d\": container with ID starting with 4606d265dcb1b9126b48c9b6e747dad15eaff1358e9ca3541e16a7ce316e0d1d not found: ID does not exist" containerID="4606d265dcb1b9126b48c9b6e747dad15eaff1358e9ca3541e16a7ce316e0d1d"
Sep 16 19:31:23.698285 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.698166    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4606d265dcb1b9126b48c9b6e747dad15eaff1358e9ca3541e16a7ce316e0d1d"} err="failed to get container status \"4606d265dcb1b9126b48c9b6e747dad15eaff1358e9ca3541e16a7ce316e0d1d\": rpc error: code = NotFound desc = could not find container \"4606d265dcb1b9126b48c9b6e747dad15eaff1358e9ca3541e16a7ce316e0d1d\": container with ID starting with 4606d265dcb1b9126b48c9b6e747dad15eaff1358e9ca3541e16a7ce316e0d1d not found: ID does not exist"
Sep 16 19:31:23.698285 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.698186    2578 scope.go:117] "RemoveContainer" containerID="6633ebb4696d9f0f7f1764dfb443408181882750cc6058305f31f7e8c8617107"
Sep 16 19:31:23.698527 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:31:23.698502    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6633ebb4696d9f0f7f1764dfb443408181882750cc6058305f31f7e8c8617107\": container with ID starting with 6633ebb4696d9f0f7f1764dfb443408181882750cc6058305f31f7e8c8617107 not found: ID does not exist" containerID="6633ebb4696d9f0f7f1764dfb443408181882750cc6058305f31f7e8c8617107"
Sep 16 19:31:23.698600 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.698539    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6633ebb4696d9f0f7f1764dfb443408181882750cc6058305f31f7e8c8617107"} err="failed to get container status \"6633ebb4696d9f0f7f1764dfb443408181882750cc6058305f31f7e8c8617107\": rpc error: code = NotFound desc = could not find container \"6633ebb4696d9f0f7f1764dfb443408181882750cc6058305f31f7e8c8617107\": container with ID starting with 6633ebb4696d9f0f7f1764dfb443408181882750cc6058305f31f7e8c8617107 not found: ID does not exist"
Sep 16 19:31:23.704930 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.704897    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"]
Sep 16 19:31:23.712814 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:23.712788    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-65594479c8-bgt85"]
Sep 16 19:31:24.217511 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:31:24.217477    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e45fe09-21e1-4ff7-81ef-21ef8efdab62" path="/var/lib/kubelet/pods/8e45fe09-21e1-4ff7-81ef-21ef8efdab62/volumes"
Sep 16 19:32:10.018258 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.018219    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl"]
Sep 16 19:32:10.018787 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.018581    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7858bfec-206b-429a-a965-aacc60221ef4" containerName="istio-proxy"
Sep 16 19:32:10.018787 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.018599    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7858bfec-206b-429a-a965-aacc60221ef4" containerName="istio-proxy"
Sep 16 19:32:10.018787 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.018614    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e45fe09-21e1-4ff7-81ef-21ef8efdab62" containerName="main"
Sep 16 19:32:10.018787 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.018619    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e45fe09-21e1-4ff7-81ef-21ef8efdab62" containerName="main"
Sep 16 19:32:10.018787 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.018627    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e45fe09-21e1-4ff7-81ef-21ef8efdab62" containerName="storage-initializer"
Sep 16 19:32:10.018787 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.018638    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e45fe09-21e1-4ff7-81ef-21ef8efdab62" containerName="storage-initializer"
Sep 16 19:32:10.018787 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.018710    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7858bfec-206b-429a-a965-aacc60221ef4" containerName="istio-proxy"
Sep 16 19:32:10.018787 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.018721    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e45fe09-21e1-4ff7-81ef-21ef8efdab62" containerName="main"
Sep 16 19:32:10.019105 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.018800    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7858bfec-206b-429a-a965-aacc60221ef4" containerName="istio-proxy"
Sep 16 19:32:10.019105 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.018807    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7858bfec-206b-429a-a965-aacc60221ef4" containerName="istio-proxy"
Sep 16 19:32:10.019105 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.018863    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7858bfec-206b-429a-a965-aacc60221ef4" containerName="istio-proxy"
Sep 16 19:32:10.021956 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.021937    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl"
Sep 16 19:32:10.025068 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.025025    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"seaweedfs-s3-creds\""
Sep 16 19:32:10.025191 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.025108    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Sep 16 19:32:10.025259 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.025184    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Sep 16 19:32:10.025259 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.025184    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-c6n9x\""
Sep 16 19:32:10.025405 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.025383    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Sep 16 19:32:10.038017 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.037994    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl"]
Sep 16 19:32:10.175478 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.175439    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d519aa76-9272-4364-b918-9e40b9d2ee0d-tls-certs\") pod \"e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl\" (UID: \"d519aa76-9272-4364-b918-9e40b9d2ee0d\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl"
Sep 16 19:32:10.175478 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.175485    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d519aa76-9272-4364-b918-9e40b9d2ee0d-dshm\") pod \"e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl\" (UID: \"d519aa76-9272-4364-b918-9e40b9d2ee0d\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl"
Sep 16 19:32:10.175725 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.175511    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d519aa76-9272-4364-b918-9e40b9d2ee0d-home\") pod \"e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl\" (UID: \"d519aa76-9272-4364-b918-9e40b9d2ee0d\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl"
Sep 16 19:32:10.175725 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.175593    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn4hw\" (UniqueName: \"kubernetes.io/projected/d519aa76-9272-4364-b918-9e40b9d2ee0d-kube-api-access-cn4hw\") pod \"e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl\" (UID: \"d519aa76-9272-4364-b918-9e40b9d2ee0d\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl"
Sep 16 19:32:10.175725 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.175638    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d519aa76-9272-4364-b918-9e40b9d2ee0d-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl\" (UID: \"d519aa76-9272-4364-b918-9e40b9d2ee0d\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl"
Sep 16 19:32:10.175869 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.175740    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d519aa76-9272-4364-b918-9e40b9d2ee0d-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl\" (UID: \"d519aa76-9272-4364-b918-9e40b9d2ee0d\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl"
Sep 16 19:32:10.175869 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.175784    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d519aa76-9272-4364-b918-9e40b9d2ee0d-model-cache\") pod \"e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl\" (UID: \"d519aa76-9272-4364-b918-9e40b9d2ee0d\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl"
Sep 16 19:32:10.276323 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.276236    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn4hw\" (UniqueName: \"kubernetes.io/projected/d519aa76-9272-4364-b918-9e40b9d2ee0d-kube-api-access-cn4hw\") pod \"e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl\" (UID: \"d519aa76-9272-4364-b918-9e40b9d2ee0d\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl"
Sep 16 19:32:10.276323 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.276291    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d519aa76-9272-4364-b918-9e40b9d2ee0d-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl\" (UID: \"d519aa76-9272-4364-b918-9e40b9d2ee0d\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl"
Sep 16 19:32:10.276549 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.276343    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d519aa76-9272-4364-b918-9e40b9d2ee0d-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl\" (UID: \"d519aa76-9272-4364-b918-9e40b9d2ee0d\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl"
Sep 16 19:32:10.276549 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.276369    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d519aa76-9272-4364-b918-9e40b9d2ee0d-model-cache\") pod \"e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl\" (UID: \"d519aa76-9272-4364-b918-9e40b9d2ee0d\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl"
Sep 16 19:32:10.276549 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.276398    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d519aa76-9272-4364-b918-9e40b9d2ee0d-tls-certs\") pod \"e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl\" (UID: \"d519aa76-9272-4364-b918-9e40b9d2ee0d\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl"
Sep 16 19:32:10.276549 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.276434    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d519aa76-9272-4364-b918-9e40b9d2ee0d-dshm\") pod \"e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl\" (UID: \"d519aa76-9272-4364-b918-9e40b9d2ee0d\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl"
Sep 16 19:32:10.276549 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.276485    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d519aa76-9272-4364-b918-9e40b9d2ee0d-home\") pod \"e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl\" (UID: \"d519aa76-9272-4364-b918-9e40b9d2ee0d\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl"
Sep 16 19:32:10.276824 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.276712    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d519aa76-9272-4364-b918-9e40b9d2ee0d-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl\" (UID: \"d519aa76-9272-4364-b918-9e40b9d2ee0d\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl"
Sep 16 19:32:10.276881 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.276834    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d519aa76-9272-4364-b918-9e40b9d2ee0d-model-cache\") pod \"e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl\" (UID: \"d519aa76-9272-4364-b918-9e40b9d2ee0d\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl"
Sep 16 19:32:10.276936 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.276881    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d519aa76-9272-4364-b918-9e40b9d2ee0d-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl\" (UID: \"d519aa76-9272-4364-b918-9e40b9d2ee0d\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl"
Sep 16 19:32:10.276975 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.276946    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d519aa76-9272-4364-b918-9e40b9d2ee0d-home\") pod \"e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl\" (UID: \"d519aa76-9272-4364-b918-9e40b9d2ee0d\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl"
Sep 16 19:32:10.278837 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.278818    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d519aa76-9272-4364-b918-9e40b9d2ee0d-dshm\") pod \"e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl\" (UID: \"d519aa76-9272-4364-b918-9e40b9d2ee0d\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl"
Sep 16 19:32:10.279067 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.279033    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d519aa76-9272-4364-b918-9e40b9d2ee0d-tls-certs\") pod \"e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl\" (UID: \"d519aa76-9272-4364-b918-9e40b9d2ee0d\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl"
Sep 16 19:32:10.287527 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.287497    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn4hw\" (UniqueName: \"kubernetes.io/projected/d519aa76-9272-4364-b918-9e40b9d2ee0d-kube-api-access-cn4hw\") pod \"e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl\" (UID: \"d519aa76-9272-4364-b918-9e40b9d2ee0d\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl"
Sep 16 19:32:10.335597 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.335569    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl"
Sep 16 19:32:10.618062 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.618021    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl"]
Sep 16 19:32:10.619685 ip-10-0-142-80 kubenswrapper[2578]: W0916 19:32:10.619660    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd519aa76_9272_4364_b918_9e40b9d2ee0d.slice/crio-013fe2c9d94baf702ed662b327e1945137774463d080d449d4f3aaea04641e0b WatchSource:0}: Error finding container 013fe2c9d94baf702ed662b327e1945137774463d080d449d4f3aaea04641e0b: Status 404 returned error can't find the container with id 013fe2c9d94baf702ed662b327e1945137774463d080d449d4f3aaea04641e0b
Sep 16 19:32:10.912813 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.912735    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl" event={"ID":"d519aa76-9272-4364-b918-9e40b9d2ee0d","Type":"ContainerStarted","Data":"b67b81db00b0ca975dc0c36e99d2a29e55b1491ba25e39150644492b9964aa56"}
Sep 16 19:32:10.912813 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:10.912792    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl" event={"ID":"d519aa76-9272-4364-b918-9e40b9d2ee0d","Type":"ContainerStarted","Data":"013fe2c9d94baf702ed662b327e1945137774463d080d449d4f3aaea04641e0b"}
Sep 16 19:32:13.220528 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:13.220490    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl"]
Sep 16 19:32:13.220970 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:13.220730    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl" podUID="d519aa76-9272-4364-b918-9e40b9d2ee0d" containerName="storage-initializer" containerID="cri-o://b67b81db00b0ca975dc0c36e99d2a29e55b1491ba25e39150644492b9964aa56" gracePeriod=60
Sep 16 19:32:14.867444 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:14.867416    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl"
Sep 16 19:32:14.935227 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:14.935195    2578 generic.go:356] "Generic (PLEG): container finished" podID="d519aa76-9272-4364-b918-9e40b9d2ee0d" containerID="b67b81db00b0ca975dc0c36e99d2a29e55b1491ba25e39150644492b9964aa56" exitCode=0
Sep 16 19:32:14.935392 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:14.935243    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl"
Sep 16 19:32:14.935392 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:14.935278    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl" event={"ID":"d519aa76-9272-4364-b918-9e40b9d2ee0d","Type":"ContainerDied","Data":"b67b81db00b0ca975dc0c36e99d2a29e55b1491ba25e39150644492b9964aa56"}
Sep 16 19:32:14.935392 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:14.935337    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl" event={"ID":"d519aa76-9272-4364-b918-9e40b9d2ee0d","Type":"ContainerDied","Data":"013fe2c9d94baf702ed662b327e1945137774463d080d449d4f3aaea04641e0b"}
Sep 16 19:32:14.935392 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:14.935363    2578 scope.go:117] "RemoveContainer" containerID="b67b81db00b0ca975dc0c36e99d2a29e55b1491ba25e39150644492b9964aa56"
Sep 16 19:32:14.944713 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:14.944697    2578 scope.go:117] "RemoveContainer" containerID="b67b81db00b0ca975dc0c36e99d2a29e55b1491ba25e39150644492b9964aa56"
Sep 16 19:32:14.944984 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:32:14.944965    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b67b81db00b0ca975dc0c36e99d2a29e55b1491ba25e39150644492b9964aa56\": container with ID starting with b67b81db00b0ca975dc0c36e99d2a29e55b1491ba25e39150644492b9964aa56 not found: ID does not exist" containerID="b67b81db00b0ca975dc0c36e99d2a29e55b1491ba25e39150644492b9964aa56"
Sep 16 19:32:14.945075 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:14.944997    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b67b81db00b0ca975dc0c36e99d2a29e55b1491ba25e39150644492b9964aa56"} err="failed to get container status \"b67b81db00b0ca975dc0c36e99d2a29e55b1491ba25e39150644492b9964aa56\": rpc error: code = NotFound desc = could not find container \"b67b81db00b0ca975dc0c36e99d2a29e55b1491ba25e39150644492b9964aa56\": container with ID starting with b67b81db00b0ca975dc0c36e99d2a29e55b1491ba25e39150644492b9964aa56 not found: ID does not exist"
Sep 16 19:32:15.021085 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:15.021014    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d519aa76-9272-4364-b918-9e40b9d2ee0d-tmp-dir\") pod \"d519aa76-9272-4364-b918-9e40b9d2ee0d\" (UID: \"d519aa76-9272-4364-b918-9e40b9d2ee0d\") "
Sep 16 19:32:15.021085 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:15.021072    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d519aa76-9272-4364-b918-9e40b9d2ee0d-tls-certs\") pod \"d519aa76-9272-4364-b918-9e40b9d2ee0d\" (UID: \"d519aa76-9272-4364-b918-9e40b9d2ee0d\") "
Sep 16 19:32:15.021242 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:15.021109    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d519aa76-9272-4364-b918-9e40b9d2ee0d-kserve-provision-location\") pod \"d519aa76-9272-4364-b918-9e40b9d2ee0d\" (UID: \"d519aa76-9272-4364-b918-9e40b9d2ee0d\") "
Sep 16 19:32:15.021242 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:15.021143    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d519aa76-9272-4364-b918-9e40b9d2ee0d-home\") pod \"d519aa76-9272-4364-b918-9e40b9d2ee0d\" (UID: \"d519aa76-9272-4364-b918-9e40b9d2ee0d\") "
Sep 16 19:32:15.021242 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:15.021161    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d519aa76-9272-4364-b918-9e40b9d2ee0d-dshm\") pod \"d519aa76-9272-4364-b918-9e40b9d2ee0d\" (UID: \"d519aa76-9272-4364-b918-9e40b9d2ee0d\") "
Sep 16 19:32:15.021242 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:15.021232    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cn4hw\" (UniqueName: \"kubernetes.io/projected/d519aa76-9272-4364-b918-9e40b9d2ee0d-kube-api-access-cn4hw\") pod \"d519aa76-9272-4364-b918-9e40b9d2ee0d\" (UID: \"d519aa76-9272-4364-b918-9e40b9d2ee0d\") "
Sep 16 19:32:15.021448 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:15.021249    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d519aa76-9272-4364-b918-9e40b9d2ee0d-model-cache\") pod \"d519aa76-9272-4364-b918-9e40b9d2ee0d\" (UID: \"d519aa76-9272-4364-b918-9e40b9d2ee0d\") "
Sep 16 19:32:15.021448 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:15.021287    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d519aa76-9272-4364-b918-9e40b9d2ee0d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d519aa76-9272-4364-b918-9e40b9d2ee0d" (UID: "d519aa76-9272-4364-b918-9e40b9d2ee0d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:15.021448 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:15.021410    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d519aa76-9272-4364-b918-9e40b9d2ee0d-home" (OuterVolumeSpecName: "home") pod "d519aa76-9272-4364-b918-9e40b9d2ee0d" (UID: "d519aa76-9272-4364-b918-9e40b9d2ee0d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:15.021607 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:15.021491    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d519aa76-9272-4364-b918-9e40b9d2ee0d-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:32:15.021607 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:15.021508    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d519aa76-9272-4364-b918-9e40b9d2ee0d-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:32:15.021607 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:15.021560    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d519aa76-9272-4364-b918-9e40b9d2ee0d-model-cache" (OuterVolumeSpecName: "model-cache") pod "d519aa76-9272-4364-b918-9e40b9d2ee0d" (UID: "d519aa76-9272-4364-b918-9e40b9d2ee0d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:15.023368 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:15.023335    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d519aa76-9272-4364-b918-9e40b9d2ee0d-dshm" (OuterVolumeSpecName: "dshm") pod "d519aa76-9272-4364-b918-9e40b9d2ee0d" (UID: "d519aa76-9272-4364-b918-9e40b9d2ee0d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:15.023472 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:15.023412    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d519aa76-9272-4364-b918-9e40b9d2ee0d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d519aa76-9272-4364-b918-9e40b9d2ee0d" (UID: "d519aa76-9272-4364-b918-9e40b9d2ee0d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:32:15.023544 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:15.023513    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d519aa76-9272-4364-b918-9e40b9d2ee0d-kube-api-access-cn4hw" (OuterVolumeSpecName: "kube-api-access-cn4hw") pod "d519aa76-9272-4364-b918-9e40b9d2ee0d" (UID: "d519aa76-9272-4364-b918-9e40b9d2ee0d"). InnerVolumeSpecName "kube-api-access-cn4hw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:32:15.093420 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:15.093384    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d519aa76-9272-4364-b918-9e40b9d2ee0d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d519aa76-9272-4364-b918-9e40b9d2ee0d" (UID: "d519aa76-9272-4364-b918-9e40b9d2ee0d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:15.122102 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:15.122071    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn4hw\" (UniqueName: \"kubernetes.io/projected/d519aa76-9272-4364-b918-9e40b9d2ee0d-kube-api-access-cn4hw\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:32:15.122102 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:15.122099    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d519aa76-9272-4364-b918-9e40b9d2ee0d-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:32:15.122259 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:15.122110    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d519aa76-9272-4364-b918-9e40b9d2ee0d-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:32:15.122259 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:15.122118    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d519aa76-9272-4364-b918-9e40b9d2ee0d-kserve-provision-location\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:32:15.122259 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:15.122127    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d519aa76-9272-4364-b918-9e40b9d2ee0d-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:32:15.279054 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:15.278968    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl"]
Sep 16 19:32:15.282732 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:15.282707    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-7d4cb8bb45-h86gl"]
Sep 16 19:32:16.217558 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:16.217524    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d519aa76-9272-4364-b918-9e40b9d2ee0d" path="/var/lib/kubelet/pods/d519aa76-9272-4364-b918-9e40b9d2ee0d/volumes"
Sep 16 19:32:33.813314 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:33.813281    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8"]
Sep 16 19:32:33.813712 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:33.813598    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d519aa76-9272-4364-b918-9e40b9d2ee0d" containerName="storage-initializer"
Sep 16 19:32:33.813712 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:33.813610    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d519aa76-9272-4364-b918-9e40b9d2ee0d" containerName="storage-initializer"
Sep 16 19:32:33.813712 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:33.813667    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d519aa76-9272-4364-b918-9e40b9d2ee0d" containerName="storage-initializer"
Sep 16 19:32:33.818345 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:33.818328    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8"
Sep 16 19:32:33.823840 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:33.823813    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"e2e-wk-cfg-implicit-svc-kserve-self-signed-certs\""
Sep 16 19:32:33.823973 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:33.823813    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"openshift-service-ca.crt\""
Sep 16 19:32:33.823973 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:33.823881    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"kube-root-ca.crt\""
Sep 16 19:32:33.823973 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:33.823918    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"default-dockercfg-v6ksb\""
Sep 16 19:32:33.823973 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:33.823867    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"e2e-wk-cfg-implicit-svc-epp-sa-dockercfg-qzqst\""
Sep 16 19:32:33.829496 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:33.829470    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8"]
Sep 16 19:32:33.852705 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:33.852683    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pqqk\" (UniqueName: \"kubernetes.io/projected/f64ac41a-2d96-4363-821c-bedb8faf2791-kube-api-access-7pqqk\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8\" (UID: \"f64ac41a-2d96-4363-821c-bedb8faf2791\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8"
Sep 16 19:32:33.852811 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:33.852721    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f64ac41a-2d96-4363-821c-bedb8faf2791-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8\" (UID: \"f64ac41a-2d96-4363-821c-bedb8faf2791\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8"
Sep 16 19:32:33.953290 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:33.953263    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f64ac41a-2d96-4363-821c-bedb8faf2791-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8\" (UID: \"f64ac41a-2d96-4363-821c-bedb8faf2791\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8"
Sep 16 19:32:33.953439 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:33.953359    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pqqk\" (UniqueName: \"kubernetes.io/projected/f64ac41a-2d96-4363-821c-bedb8faf2791-kube-api-access-7pqqk\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8\" (UID: \"f64ac41a-2d96-4363-821c-bedb8faf2791\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8"
Sep 16 19:32:33.955718 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:33.955687    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f64ac41a-2d96-4363-821c-bedb8faf2791-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8\" (UID: \"f64ac41a-2d96-4363-821c-bedb8faf2791\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8"
Sep 16 19:32:33.963520 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:33.963497    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pqqk\" (UniqueName: \"kubernetes.io/projected/f64ac41a-2d96-4363-821c-bedb8faf2791-kube-api-access-7pqqk\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8\" (UID: \"f64ac41a-2d96-4363-821c-bedb8faf2791\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8"
Sep 16 19:32:34.128988 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:34.128894    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8"
Sep 16 19:32:34.411680 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:34.406316    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8"]
Sep 16 19:32:35.038139 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:35.038103    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8" event={"ID":"f64ac41a-2d96-4363-821c-bedb8faf2791","Type":"ContainerStarted","Data":"615c06259a9e761f224f3d732780ef1ea2f12c39c3180e90cc611fd6e9255fa3"}
Sep 16 19:32:35.038551 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:35.038155    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8" event={"ID":"f64ac41a-2d96-4363-821c-bedb8faf2791","Type":"ContainerStarted","Data":"dcc1e566cbed60fe4ede20038115f5787a0b204c1a7db70e1eaae030ffa7f6b4"}
Sep 16 19:32:35.038551 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:35.038214    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8"
Sep 16 19:32:35.063245 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:35.063197    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8" podStartSLOduration=2.063182773 podStartE2EDuration="2.063182773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:32:35.062090043 +0000 UTC m=+3123.297023488" watchObservedRunningTime="2026-09-16 19:32:35.063182773 +0000 UTC m=+3123.298116216"
Sep 16 19:32:35.177180 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:35.177144    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8"]
Sep 16 19:32:36.042794 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:36.042761    2578 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8" secret="" err="secret \"e2e-wk-cfg-implicit-svc-epp-sa-dockercfg-qzqst\" not found"
Sep 16 19:32:36.069715 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:32:36.069682    2578 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:32:36.069873 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:32:36.069755    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f64ac41a-2d96-4363-821c-bedb8faf2791-tls-certs podName:f64ac41a-2d96-4363-821c-bedb8faf2791 nodeName:}" failed. No retries permitted until 2026-09-16 19:32:36.569735799 +0000 UTC m=+3124.804669235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f64ac41a-2d96-4363-821c-bedb8faf2791-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8" (UID: "f64ac41a-2d96-4363-821c-bedb8faf2791") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:32:36.573826 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:32:36.573792    2578 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:32:36.573993 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:32:36.573856    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f64ac41a-2d96-4363-821c-bedb8faf2791-tls-certs podName:f64ac41a-2d96-4363-821c-bedb8faf2791 nodeName:}" failed. No retries permitted until 2026-09-16 19:32:37.573842965 +0000 UTC m=+3125.808776386 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f64ac41a-2d96-4363-821c-bedb8faf2791-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8" (UID: "f64ac41a-2d96-4363-821c-bedb8faf2791") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:32:37.584867 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:32:37.584833    2578 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:32:37.585380 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:32:37.584918    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f64ac41a-2d96-4363-821c-bedb8faf2791-tls-certs podName:f64ac41a-2d96-4363-821c-bedb8faf2791 nodeName:}" failed. No retries permitted until 2026-09-16 19:32:39.584899516 +0000 UTC m=+3127.819832952 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f64ac41a-2d96-4363-821c-bedb8faf2791-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8" (UID: "f64ac41a-2d96-4363-821c-bedb8faf2791") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:32:39.602532 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:32:39.602490    2578 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:32:39.602983 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:32:39.602588    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f64ac41a-2d96-4363-821c-bedb8faf2791-tls-certs podName:f64ac41a-2d96-4363-821c-bedb8faf2791 nodeName:}" failed. No retries permitted until 2026-09-16 19:32:43.602570676 +0000 UTC m=+3131.837504099 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f64ac41a-2d96-4363-821c-bedb8faf2791-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8" (UID: "f64ac41a-2d96-4363-821c-bedb8faf2791") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:32:41.299447 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:41.299412    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr"]
Sep 16 19:32:41.302881 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:41.302859    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr"
Sep 16 19:32:41.307171 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:41.307143    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-2fs62\""
Sep 16 19:32:41.307304 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:41.307211    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-h7zhm\""
Sep 16 19:32:41.307304 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:41.307143    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 16 19:32:41.307304 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:41.307267    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 16 19:32:41.307471 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:41.307322    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 16 19:32:41.314794 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:41.314774    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr"]
Sep 16 19:32:41.415799 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:41.415635    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgrjv\" (UniqueName: \"kubernetes.io/projected/fefcabfd-9926-4a66-a41b-87b84beea9b4-kube-api-access-tgrjv\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr\" (UID: \"fefcabfd-9926-4a66-a41b-87b84beea9b4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr"
Sep 16 19:32:41.415799 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:41.415698    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fefcabfd-9926-4a66-a41b-87b84beea9b4-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr\" (UID: \"fefcabfd-9926-4a66-a41b-87b84beea9b4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr"
Sep 16 19:32:41.516699 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:41.516666    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgrjv\" (UniqueName: \"kubernetes.io/projected/fefcabfd-9926-4a66-a41b-87b84beea9b4-kube-api-access-tgrjv\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr\" (UID: \"fefcabfd-9926-4a66-a41b-87b84beea9b4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr"
Sep 16 19:32:41.516838 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:41.516729    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fefcabfd-9926-4a66-a41b-87b84beea9b4-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr\" (UID: \"fefcabfd-9926-4a66-a41b-87b84beea9b4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr"
Sep 16 19:32:41.519176 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:41.519157    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fefcabfd-9926-4a66-a41b-87b84beea9b4-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr\" (UID: \"fefcabfd-9926-4a66-a41b-87b84beea9b4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr"
Sep 16 19:32:41.525651 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:41.525626    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgrjv\" (UniqueName: \"kubernetes.io/projected/fefcabfd-9926-4a66-a41b-87b84beea9b4-kube-api-access-tgrjv\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr\" (UID: \"fefcabfd-9926-4a66-a41b-87b84beea9b4\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr"
Sep 16 19:32:41.615704 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:41.615638    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr"
Sep 16 19:32:41.896609 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:41.896386    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr"]
Sep 16 19:32:42.080462 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:42.080406    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr" event={"ID":"fefcabfd-9926-4a66-a41b-87b84beea9b4","Type":"ContainerStarted","Data":"0d6879285f99ad35d3818345f33817ed57aec5bd8523480bce2fd1ddbdc34ef1"}
Sep 16 19:32:42.080648 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:42.080492    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr" event={"ID":"fefcabfd-9926-4a66-a41b-87b84beea9b4","Type":"ContainerStarted","Data":"584c0e838837269f0103cb78f617a09f47b5a9f4d9d5c7b15b71b78b51a126fc"}
Sep 16 19:32:42.080648 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:42.080545    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr"
Sep 16 19:32:42.099547 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:42.099496    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr" podStartSLOduration=1.099479406 podStartE2EDuration="1.099479406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:32:42.098299306 +0000 UTC m=+3130.333232750" watchObservedRunningTime="2026-09-16 19:32:42.099479406 +0000 UTC m=+3130.334412849"
Sep 16 19:32:43.634421 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:32:43.634386    2578 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:32:43.634853 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:32:43.634467    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f64ac41a-2d96-4363-821c-bedb8faf2791-tls-certs podName:f64ac41a-2d96-4363-821c-bedb8faf2791 nodeName:}" failed. No retries permitted until 2026-09-16 19:32:51.634451967 +0000 UTC m=+3139.869385390 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f64ac41a-2d96-4363-821c-bedb8faf2791-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8" (UID: "f64ac41a-2d96-4363-821c-bedb8faf2791") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:32:51.695883 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:32:51.695849    2578 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:32:51.698376 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:32:51.695922    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f64ac41a-2d96-4363-821c-bedb8faf2791-tls-certs podName:f64ac41a-2d96-4363-821c-bedb8faf2791 nodeName:}" failed. No retries permitted until 2026-09-16 19:33:07.695907876 +0000 UTC m=+3155.930841298 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f64ac41a-2d96-4363-821c-bedb8faf2791-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8" (UID: "f64ac41a-2d96-4363-821c-bedb8faf2791") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:32:52.067991 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:32:52.067902    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8" podUID="f64ac41a-2d96-4363-821c-bedb8faf2791" containerName="main" containerID="cri-o://615c06259a9e761f224f3d732780ef1ea2f12c39c3180e90cc611fd6e9255fa3" gracePeriod=45
Sep 16 19:33:07.052071 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:33:07.051962    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8" podUID="f64ac41a-2d96-4363-821c-bedb8faf2791" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:33:07.737427 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:33:07.737391    2578 secret.go:189] Couldn't get secret e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-self-signed-certs: secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:33:07.737609 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:33:07.737463    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f64ac41a-2d96-4363-821c-bedb8faf2791-tls-certs podName:f64ac41a-2d96-4363-821c-bedb8faf2791 nodeName:}" failed. No retries permitted until 2026-09-16 19:33:39.737449743 +0000 UTC m=+3187.972383164 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f64ac41a-2d96-4363-821c-bedb8faf2791-tls-certs") pod "e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8" (UID: "f64ac41a-2d96-4363-821c-bedb8faf2791") : secret "e2e-wk-cfg-implicit-svc-kserve-self-signed-certs" not found
Sep 16 19:33:13.086956 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:33:13.086919    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr"
Sep 16 19:33:17.050779 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:33:17.050742    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8" podUID="f64ac41a-2d96-4363-821c-bedb8faf2791" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:33:22.290835 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:33:22.290794    2578 generic.go:356] "Generic (PLEG): container finished" podID="f64ac41a-2d96-4363-821c-bedb8faf2791" containerID="615c06259a9e761f224f3d732780ef1ea2f12c39c3180e90cc611fd6e9255fa3" exitCode=0
Sep 16 19:33:22.291303 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:33:22.290866    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8" event={"ID":"f64ac41a-2d96-4363-821c-bedb8faf2791","Type":"ContainerDied","Data":"615c06259a9e761f224f3d732780ef1ea2f12c39c3180e90cc611fd6e9255fa3"}
Sep 16 19:33:22.325317 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:33:22.325286    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8"
Sep 16 19:33:22.452256 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:33:22.452211    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f64ac41a-2d96-4363-821c-bedb8faf2791-tls-certs\") pod \"f64ac41a-2d96-4363-821c-bedb8faf2791\" (UID: \"f64ac41a-2d96-4363-821c-bedb8faf2791\") "
Sep 16 19:33:22.452454 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:33:22.452301    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pqqk\" (UniqueName: \"kubernetes.io/projected/f64ac41a-2d96-4363-821c-bedb8faf2791-kube-api-access-7pqqk\") pod \"f64ac41a-2d96-4363-821c-bedb8faf2791\" (UID: \"f64ac41a-2d96-4363-821c-bedb8faf2791\") "
Sep 16 19:33:22.454557 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:33:22.454525    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f64ac41a-2d96-4363-821c-bedb8faf2791-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f64ac41a-2d96-4363-821c-bedb8faf2791" (UID: "f64ac41a-2d96-4363-821c-bedb8faf2791"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:33:22.454557 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:33:22.454525    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f64ac41a-2d96-4363-821c-bedb8faf2791-kube-api-access-7pqqk" (OuterVolumeSpecName: "kube-api-access-7pqqk") pod "f64ac41a-2d96-4363-821c-bedb8faf2791" (UID: "f64ac41a-2d96-4363-821c-bedb8faf2791"). InnerVolumeSpecName "kube-api-access-7pqqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:33:22.553154 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:33:22.553108    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pqqk\" (UniqueName: \"kubernetes.io/projected/f64ac41a-2d96-4363-821c-bedb8faf2791-kube-api-access-7pqqk\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:33:22.553154 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:33:22.553146    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f64ac41a-2d96-4363-821c-bedb8faf2791-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:33:23.297132 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:33:23.297102    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8"
Sep 16 19:33:23.297132 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:33:23.297127    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8" event={"ID":"f64ac41a-2d96-4363-821c-bedb8faf2791","Type":"ContainerDied","Data":"dcc1e566cbed60fe4ede20038115f5787a0b204c1a7db70e1eaae030ffa7f6b4"}
Sep 16 19:33:23.297628 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:33:23.297181    2578 scope.go:117] "RemoveContainer" containerID="615c06259a9e761f224f3d732780ef1ea2f12c39c3180e90cc611fd6e9255fa3"
Sep 16 19:33:23.331744 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:33:23.331710    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8"]
Sep 16 19:33:23.338251 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:33:23.338228    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddrvvz8"]
Sep 16 19:33:24.217620 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:33:24.217578    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f64ac41a-2d96-4363-821c-bedb8faf2791" path="/var/lib/kubelet/pods/f64ac41a-2d96-4363-821c-bedb8faf2791/volumes"
Sep 16 19:34:33.412527 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:34:33.412432    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr"]
Sep 16 19:34:43.087451 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:34:43.087414    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr" podUID="fefcabfd-9926-4a66-a41b-87b84beea9b4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:34:48.431534 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:34:48.431490    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr" podUID="fefcabfd-9926-4a66-a41b-87b84beea9b4" containerName="main" containerID="cri-o://0d6879285f99ad35d3818345f33817ed57aec5bd8523480bce2fd1ddbdc34ef1" gracePeriod=45
Sep 16 19:34:53.087446 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:34:53.087408    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr" podUID="fefcabfd-9926-4a66-a41b-87b84beea9b4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:34:53.674478 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:34:53.674442    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx"]
Sep 16 19:34:53.674874 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:34:53.674854    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f64ac41a-2d96-4363-821c-bedb8faf2791" containerName="main"
Sep 16 19:34:53.674972 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:34:53.674878    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f64ac41a-2d96-4363-821c-bedb8faf2791" containerName="main"
Sep 16 19:34:53.675028 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:34:53.674992    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f64ac41a-2d96-4363-821c-bedb8faf2791" containerName="main"
Sep 16 19:34:53.677691 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:34:53.677668    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx"
Sep 16 19:34:53.680426 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:34:53.680403    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-vp4l9\""
Sep 16 19:34:53.686891 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:34:53.686863    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx"]
Sep 16 19:34:53.767496 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:34:53.767458    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb837dc5-1bdb-4757-aa34-fb590cb45f08-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx\" (UID: \"cb837dc5-1bdb-4757-aa34-fb590cb45f08\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx"
Sep 16 19:34:53.767674 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:34:53.767503    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q7j8\" (UniqueName: \"kubernetes.io/projected/cb837dc5-1bdb-4757-aa34-fb590cb45f08-kube-api-access-5q7j8\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx\" (UID: \"cb837dc5-1bdb-4757-aa34-fb590cb45f08\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx"
Sep 16 19:34:53.868231 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:34:53.868187    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q7j8\" (UniqueName: \"kubernetes.io/projected/cb837dc5-1bdb-4757-aa34-fb590cb45f08-kube-api-access-5q7j8\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx\" (UID: \"cb837dc5-1bdb-4757-aa34-fb590cb45f08\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx"
Sep 16 19:34:53.868416 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:34:53.868333    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb837dc5-1bdb-4757-aa34-fb590cb45f08-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx\" (UID: \"cb837dc5-1bdb-4757-aa34-fb590cb45f08\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx"
Sep 16 19:34:53.870918 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:34:53.870882    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb837dc5-1bdb-4757-aa34-fb590cb45f08-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx\" (UID: \"cb837dc5-1bdb-4757-aa34-fb590cb45f08\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx"
Sep 16 19:34:53.877980 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:34:53.877951    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q7j8\" (UniqueName: \"kubernetes.io/projected/cb837dc5-1bdb-4757-aa34-fb590cb45f08-kube-api-access-5q7j8\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx\" (UID: \"cb837dc5-1bdb-4757-aa34-fb590cb45f08\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx"
Sep 16 19:34:53.988701 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:34:53.988601    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx"
Sep 16 19:34:54.266967 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:34:54.266731    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx"]
Sep 16 19:34:54.269585 ip-10-0-142-80 kubenswrapper[2578]: W0916 19:34:54.269551    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb837dc5_1bdb_4757_aa34_fb590cb45f08.slice/crio-57100578bb011b7e9652013743f4cc88d33f6aec6b8f17ecc93d9fbce10f81bd WatchSource:0}: Error finding container 57100578bb011b7e9652013743f4cc88d33f6aec6b8f17ecc93d9fbce10f81bd: Status 404 returned error can't find the container with id 57100578bb011b7e9652013743f4cc88d33f6aec6b8f17ecc93d9fbce10f81bd
Sep 16 19:34:54.761935 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:34:54.761892    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx" event={"ID":"cb837dc5-1bdb-4757-aa34-fb590cb45f08","Type":"ContainerStarted","Data":"c483ec4c9b5da360a761b46edf03c0f858ebec144e85222184ddd8f9603c1e29"}
Sep 16 19:34:54.762162 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:34:54.762000    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx"
Sep 16 19:34:54.762162 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:34:54.762026    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx" event={"ID":"cb837dc5-1bdb-4757-aa34-fb590cb45f08","Type":"ContainerStarted","Data":"57100578bb011b7e9652013743f4cc88d33f6aec6b8f17ecc93d9fbce10f81bd"}
Sep 16 19:34:54.790168 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:34:54.790109    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx" podStartSLOduration=1.790094388 podStartE2EDuration="1.790094388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:34:54.787233478 +0000 UTC m=+3263.022166923" watchObservedRunningTime="2026-09-16 19:34:54.790094388 +0000 UTC m=+3263.025027832"
Sep 16 19:35:03.087320 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:35:03.087273    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr" podUID="fefcabfd-9926-4a66-a41b-87b84beea9b4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:35:03.087777 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:35:03.087421    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr"
Sep 16 19:35:13.087462 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:35:13.087422    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr" podUID="fefcabfd-9926-4a66-a41b-87b84beea9b4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:35:18.672625 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:35:18.672599    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr"
Sep 16 19:35:18.778229 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:35:18.778121    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fefcabfd-9926-4a66-a41b-87b84beea9b4-tls-certs\") pod \"fefcabfd-9926-4a66-a41b-87b84beea9b4\" (UID: \"fefcabfd-9926-4a66-a41b-87b84beea9b4\") "
Sep 16 19:35:18.778229 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:35:18.778177    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgrjv\" (UniqueName: \"kubernetes.io/projected/fefcabfd-9926-4a66-a41b-87b84beea9b4-kube-api-access-tgrjv\") pod \"fefcabfd-9926-4a66-a41b-87b84beea9b4\" (UID: \"fefcabfd-9926-4a66-a41b-87b84beea9b4\") "
Sep 16 19:35:18.780480 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:35:18.780441    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fefcabfd-9926-4a66-a41b-87b84beea9b4-kube-api-access-tgrjv" (OuterVolumeSpecName: "kube-api-access-tgrjv") pod "fefcabfd-9926-4a66-a41b-87b84beea9b4" (UID: "fefcabfd-9926-4a66-a41b-87b84beea9b4"). InnerVolumeSpecName "kube-api-access-tgrjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:35:18.780611 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:35:18.780506    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fefcabfd-9926-4a66-a41b-87b84beea9b4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fefcabfd-9926-4a66-a41b-87b84beea9b4" (UID: "fefcabfd-9926-4a66-a41b-87b84beea9b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:35:18.879368 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:35:18.879328    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fefcabfd-9926-4a66-a41b-87b84beea9b4-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:35:18.879368 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:35:18.879360    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgrjv\" (UniqueName: \"kubernetes.io/projected/fefcabfd-9926-4a66-a41b-87b84beea9b4-kube-api-access-tgrjv\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:35:18.891241 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:35:18.891201    2578 generic.go:356] "Generic (PLEG): container finished" podID="fefcabfd-9926-4a66-a41b-87b84beea9b4" containerID="0d6879285f99ad35d3818345f33817ed57aec5bd8523480bce2fd1ddbdc34ef1" exitCode=0
Sep 16 19:35:18.891423 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:35:18.891245    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr"
Sep 16 19:35:18.891423 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:35:18.891288    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr" event={"ID":"fefcabfd-9926-4a66-a41b-87b84beea9b4","Type":"ContainerDied","Data":"0d6879285f99ad35d3818345f33817ed57aec5bd8523480bce2fd1ddbdc34ef1"}
Sep 16 19:35:18.891423 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:35:18.891350    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr" event={"ID":"fefcabfd-9926-4a66-a41b-87b84beea9b4","Type":"ContainerDied","Data":"584c0e838837269f0103cb78f617a09f47b5a9f4d9d5c7b15b71b78b51a126fc"}
Sep 16 19:35:18.891423 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:35:18.891383    2578 scope.go:117] "RemoveContainer" containerID="0d6879285f99ad35d3818345f33817ed57aec5bd8523480bce2fd1ddbdc34ef1"
Sep 16 19:35:18.900727 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:35:18.900708    2578 scope.go:117] "RemoveContainer" containerID="0d6879285f99ad35d3818345f33817ed57aec5bd8523480bce2fd1ddbdc34ef1"
Sep 16 19:35:18.901033 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:35:18.901014    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d6879285f99ad35d3818345f33817ed57aec5bd8523480bce2fd1ddbdc34ef1\": container with ID starting with 0d6879285f99ad35d3818345f33817ed57aec5bd8523480bce2fd1ddbdc34ef1 not found: ID does not exist" containerID="0d6879285f99ad35d3818345f33817ed57aec5bd8523480bce2fd1ddbdc34ef1"
Sep 16 19:35:18.901102 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:35:18.901060    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d6879285f99ad35d3818345f33817ed57aec5bd8523480bce2fd1ddbdc34ef1"} err="failed to get container status \"0d6879285f99ad35d3818345f33817ed57aec5bd8523480bce2fd1ddbdc34ef1\": rpc error: code = NotFound desc = could not find container \"0d6879285f99ad35d3818345f33817ed57aec5bd8523480bce2fd1ddbdc34ef1\": container with ID starting with 0d6879285f99ad35d3818345f33817ed57aec5bd8523480bce2fd1ddbdc34ef1 not found: ID does not exist"
Sep 16 19:35:18.916677 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:35:18.916648    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr"]
Sep 16 19:35:18.921200 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:35:18.921176    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-v4kjr"]
Sep 16 19:35:20.217115 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:35:20.217084    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fefcabfd-9926-4a66-a41b-87b84beea9b4" path="/var/lib/kubelet/pods/fefcabfd-9926-4a66-a41b-87b84beea9b4/volumes"
Sep 16 19:35:25.768785 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:35:25.768752    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx"
Sep 16 19:36:45.901947 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:36:45.901906    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx"]
Sep 16 19:36:55.768446 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:36:55.768398    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx" podUID="cb837dc5-1bdb-4757-aa34-fb590cb45f08" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:37:00.921531 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:00.921482    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx" podUID="cb837dc5-1bdb-4757-aa34-fb590cb45f08" containerName="main" containerID="cri-o://c483ec4c9b5da360a761b46edf03c0f858ebec144e85222184ddd8f9603c1e29" gracePeriod=45
Sep 16 19:37:05.769307 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:05.769265    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx" podUID="cb837dc5-1bdb-4757-aa34-fb590cb45f08" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:37:15.769091 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:15.769017    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx" podUID="cb837dc5-1bdb-4757-aa34-fb590cb45f08" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:37:15.769574 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:15.769213    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx"
Sep 16 19:37:25.768719 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:25.768620    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx" podUID="cb837dc5-1bdb-4757-aa34-fb590cb45f08" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:37:31.160270 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:31.160243    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx"
Sep 16 19:37:31.276875 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:31.276767    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb837dc5-1bdb-4757-aa34-fb590cb45f08-tls-certs\") pod \"cb837dc5-1bdb-4757-aa34-fb590cb45f08\" (UID: \"cb837dc5-1bdb-4757-aa34-fb590cb45f08\") "
Sep 16 19:37:31.276875 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:31.276864    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5q7j8\" (UniqueName: \"kubernetes.io/projected/cb837dc5-1bdb-4757-aa34-fb590cb45f08-kube-api-access-5q7j8\") pod \"cb837dc5-1bdb-4757-aa34-fb590cb45f08\" (UID: \"cb837dc5-1bdb-4757-aa34-fb590cb45f08\") "
Sep 16 19:37:31.278909 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:31.278882    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb837dc5-1bdb-4757-aa34-fb590cb45f08-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cb837dc5-1bdb-4757-aa34-fb590cb45f08" (UID: "cb837dc5-1bdb-4757-aa34-fb590cb45f08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:37:31.279030 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:31.278908    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb837dc5-1bdb-4757-aa34-fb590cb45f08-kube-api-access-5q7j8" (OuterVolumeSpecName: "kube-api-access-5q7j8") pod "cb837dc5-1bdb-4757-aa34-fb590cb45f08" (UID: "cb837dc5-1bdb-4757-aa34-fb590cb45f08"). InnerVolumeSpecName "kube-api-access-5q7j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:37:31.377497 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:31.377463    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb837dc5-1bdb-4757-aa34-fb590cb45f08-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:37:31.377497 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:31.377493    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q7j8\" (UniqueName: \"kubernetes.io/projected/cb837dc5-1bdb-4757-aa34-fb590cb45f08-kube-api-access-5q7j8\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:37:31.567423 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:31.567311    2578 generic.go:356] "Generic (PLEG): container finished" podID="cb837dc5-1bdb-4757-aa34-fb590cb45f08" containerID="c483ec4c9b5da360a761b46edf03c0f858ebec144e85222184ddd8f9603c1e29" exitCode=0
Sep 16 19:37:31.567423 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:31.567381    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx"
Sep 16 19:37:31.567423 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:31.567385    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx" event={"ID":"cb837dc5-1bdb-4757-aa34-fb590cb45f08","Type":"ContainerDied","Data":"c483ec4c9b5da360a761b46edf03c0f858ebec144e85222184ddd8f9603c1e29"}
Sep 16 19:37:31.567707 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:31.567457    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx" event={"ID":"cb837dc5-1bdb-4757-aa34-fb590cb45f08","Type":"ContainerDied","Data":"57100578bb011b7e9652013743f4cc88d33f6aec6b8f17ecc93d9fbce10f81bd"}
Sep 16 19:37:31.567707 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:31.567483    2578 scope.go:117] "RemoveContainer" containerID="c483ec4c9b5da360a761b46edf03c0f858ebec144e85222184ddd8f9603c1e29"
Sep 16 19:37:31.577186 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:31.577169    2578 scope.go:117] "RemoveContainer" containerID="c483ec4c9b5da360a761b46edf03c0f858ebec144e85222184ddd8f9603c1e29"
Sep 16 19:37:31.577435 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:37:31.577410    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c483ec4c9b5da360a761b46edf03c0f858ebec144e85222184ddd8f9603c1e29\": container with ID starting with c483ec4c9b5da360a761b46edf03c0f858ebec144e85222184ddd8f9603c1e29 not found: ID does not exist" containerID="c483ec4c9b5da360a761b46edf03c0f858ebec144e85222184ddd8f9603c1e29"
Sep 16 19:37:31.577498 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:31.577440    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c483ec4c9b5da360a761b46edf03c0f858ebec144e85222184ddd8f9603c1e29"} err="failed to get container status \"c483ec4c9b5da360a761b46edf03c0f858ebec144e85222184ddd8f9603c1e29\": rpc error: code = NotFound desc = could not find container \"c483ec4c9b5da360a761b46edf03c0f858ebec144e85222184ddd8f9603c1e29\": container with ID starting with c483ec4c9b5da360a761b46edf03c0f858ebec144e85222184ddd8f9603c1e29 not found: ID does not exist"
Sep 16 19:37:31.592164 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:31.592144    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx"]
Sep 16 19:37:31.596301 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:31.596282    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-82btx"]
Sep 16 19:37:32.218395 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:32.218364    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb837dc5-1bdb-4757-aa34-fb590cb45f08" path="/var/lib/kubelet/pods/cb837dc5-1bdb-4757-aa34-fb590cb45f08/volumes"
Sep 16 19:37:48.989687 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:48.989649    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"]
Sep 16 19:37:48.990108 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:48.989974    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb837dc5-1bdb-4757-aa34-fb590cb45f08" containerName="main"
Sep 16 19:37:48.990108 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:48.990002    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb837dc5-1bdb-4757-aa34-fb590cb45f08" containerName="main"
Sep 16 19:37:48.990108 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:48.990016    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fefcabfd-9926-4a66-a41b-87b84beea9b4" containerName="main"
Sep 16 19:37:48.990108 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:48.990021    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fefcabfd-9926-4a66-a41b-87b84beea9b4" containerName="main"
Sep 16 19:37:48.990108 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:48.990099    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fefcabfd-9926-4a66-a41b-87b84beea9b4" containerName="main"
Sep 16 19:37:48.990108 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:48.990110    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb837dc5-1bdb-4757-aa34-fb590cb45f08" containerName="main"
Sep 16 19:37:48.993033 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:48.993007    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"
Sep 16 19:37:49.000528 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.000499    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-zl7sx\""
Sep 16 19:37:49.000662 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.000507    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Sep 16 19:37:49.000662 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.000554    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Sep 16 19:37:49.000662 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.000570    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:37:49.000662 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.000593    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Sep 16 19:37:49.000662 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.000619    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Sep 16 19:37:49.004816 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.004791    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"]
Sep 16 19:37:49.128863 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.128833    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f8919b9-22d7-435f-bebb-6382e0b970df-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr\" (UID: \"2f8919b9-22d7-435f-bebb-6382e0b970df\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"
Sep 16 19:37:49.129022 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.128872    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f8919b9-22d7-435f-bebb-6382e0b970df-tmp-dir\") pod \"lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr\" (UID: \"2f8919b9-22d7-435f-bebb-6382e0b970df\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"
Sep 16 19:37:49.129022 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.128896    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f8919b9-22d7-435f-bebb-6382e0b970df-model-cache\") pod \"lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr\" (UID: \"2f8919b9-22d7-435f-bebb-6382e0b970df\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"
Sep 16 19:37:49.129022 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.128937    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8htk\" (UniqueName: \"kubernetes.io/projected/2f8919b9-22d7-435f-bebb-6382e0b970df-kube-api-access-h8htk\") pod \"lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr\" (UID: \"2f8919b9-22d7-435f-bebb-6382e0b970df\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"
Sep 16 19:37:49.129022 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.128980    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f8919b9-22d7-435f-bebb-6382e0b970df-dshm\") pod \"lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr\" (UID: \"2f8919b9-22d7-435f-bebb-6382e0b970df\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"
Sep 16 19:37:49.129022 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.129000    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f8919b9-22d7-435f-bebb-6382e0b970df-home\") pod \"lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr\" (UID: \"2f8919b9-22d7-435f-bebb-6382e0b970df\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"
Sep 16 19:37:49.129022 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.129017    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f8919b9-22d7-435f-bebb-6382e0b970df-cabundle-cert\") pod \"lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr\" (UID: \"2f8919b9-22d7-435f-bebb-6382e0b970df\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"
Sep 16 19:37:49.129344 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.129124    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f8919b9-22d7-435f-bebb-6382e0b970df-tls-certs\") pod \"lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr\" (UID: \"2f8919b9-22d7-435f-bebb-6382e0b970df\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"
Sep 16 19:37:49.229470 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.229442    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8htk\" (UniqueName: \"kubernetes.io/projected/2f8919b9-22d7-435f-bebb-6382e0b970df-kube-api-access-h8htk\") pod \"lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr\" (UID: \"2f8919b9-22d7-435f-bebb-6382e0b970df\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"
Sep 16 19:37:49.229616 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.229477    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f8919b9-22d7-435f-bebb-6382e0b970df-dshm\") pod \"lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr\" (UID: \"2f8919b9-22d7-435f-bebb-6382e0b970df\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"
Sep 16 19:37:49.229616 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.229495    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f8919b9-22d7-435f-bebb-6382e0b970df-home\") pod \"lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr\" (UID: \"2f8919b9-22d7-435f-bebb-6382e0b970df\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"
Sep 16 19:37:49.229616 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.229511    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f8919b9-22d7-435f-bebb-6382e0b970df-cabundle-cert\") pod \"lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr\" (UID: \"2f8919b9-22d7-435f-bebb-6382e0b970df\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"
Sep 16 19:37:49.229616 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.229560    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f8919b9-22d7-435f-bebb-6382e0b970df-tls-certs\") pod \"lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr\" (UID: \"2f8919b9-22d7-435f-bebb-6382e0b970df\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"
Sep 16 19:37:49.229616 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.229596    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f8919b9-22d7-435f-bebb-6382e0b970df-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr\" (UID: \"2f8919b9-22d7-435f-bebb-6382e0b970df\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"
Sep 16 19:37:49.229880 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.229626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f8919b9-22d7-435f-bebb-6382e0b970df-tmp-dir\") pod \"lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr\" (UID: \"2f8919b9-22d7-435f-bebb-6382e0b970df\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"
Sep 16 19:37:49.229880 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.229658    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f8919b9-22d7-435f-bebb-6382e0b970df-model-cache\") pod \"lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr\" (UID: \"2f8919b9-22d7-435f-bebb-6382e0b970df\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"
Sep 16 19:37:49.230021 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.229985    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f8919b9-22d7-435f-bebb-6382e0b970df-home\") pod \"lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr\" (UID: \"2f8919b9-22d7-435f-bebb-6382e0b970df\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"
Sep 16 19:37:49.230103 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.230068    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f8919b9-22d7-435f-bebb-6382e0b970df-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr\" (UID: \"2f8919b9-22d7-435f-bebb-6382e0b970df\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"
Sep 16 19:37:49.230162 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.230104    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f8919b9-22d7-435f-bebb-6382e0b970df-model-cache\") pod \"lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr\" (UID: \"2f8919b9-22d7-435f-bebb-6382e0b970df\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"
Sep 16 19:37:49.230162 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.230134    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f8919b9-22d7-435f-bebb-6382e0b970df-tmp-dir\") pod \"lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr\" (UID: \"2f8919b9-22d7-435f-bebb-6382e0b970df\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"
Sep 16 19:37:49.230346 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.230328    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f8919b9-22d7-435f-bebb-6382e0b970df-cabundle-cert\") pod \"lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr\" (UID: \"2f8919b9-22d7-435f-bebb-6382e0b970df\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"
Sep 16 19:37:49.231804 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.231784    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f8919b9-22d7-435f-bebb-6382e0b970df-dshm\") pod \"lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr\" (UID: \"2f8919b9-22d7-435f-bebb-6382e0b970df\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"
Sep 16 19:37:49.232145 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.232125    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f8919b9-22d7-435f-bebb-6382e0b970df-tls-certs\") pod \"lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr\" (UID: \"2f8919b9-22d7-435f-bebb-6382e0b970df\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"
Sep 16 19:37:49.238785 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.238764    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8htk\" (UniqueName: \"kubernetes.io/projected/2f8919b9-22d7-435f-bebb-6382e0b970df-kube-api-access-h8htk\") pod \"lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr\" (UID: \"2f8919b9-22d7-435f-bebb-6382e0b970df\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"
Sep 16 19:37:49.303895 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.303819    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"
Sep 16 19:37:49.594594 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.594556    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"]
Sep 16 19:37:49.663255 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:49.663226    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr" event={"ID":"2f8919b9-22d7-435f-bebb-6382e0b970df","Type":"ContainerStarted","Data":"0c96b5e85169c6bd0f543a382fee6a53561bd70e21940b9c3849061f7f406895"}
Sep 16 19:37:50.673550 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:50.673505    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr" event={"ID":"2f8919b9-22d7-435f-bebb-6382e0b970df","Type":"ContainerStarted","Data":"82e986f55bd6e4928abee8193bea5b14ce09cc11720944ffbf3fda96eff13745"}
Sep 16 19:37:52.684659 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:52.684625    2578 generic.go:356] "Generic (PLEG): container finished" podID="2f8919b9-22d7-435f-bebb-6382e0b970df" containerID="82e986f55bd6e4928abee8193bea5b14ce09cc11720944ffbf3fda96eff13745" exitCode=0
Sep 16 19:37:52.685116 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:52.684678    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr" event={"ID":"2f8919b9-22d7-435f-bebb-6382e0b970df","Type":"ContainerDied","Data":"82e986f55bd6e4928abee8193bea5b14ce09cc11720944ffbf3fda96eff13745"}
Sep 16 19:37:53.692736 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:53.692700    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr" event={"ID":"2f8919b9-22d7-435f-bebb-6382e0b970df","Type":"ContainerStarted","Data":"989fd4c010a442b4003f6253f8308e0d3453f386ea5cf5f08506d7f6d8346111"}
Sep 16 19:37:53.719253 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:53.719202    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr" podStartSLOduration=3.112938858 podStartE2EDuration="5.719187243s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.606248385s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:37:53.716573795 +0000 UTC m=+3441.951507239" watchObservedRunningTime="2026-09-16 19:37:53.719187243 +0000 UTC m=+3441.954120685"
Sep 16 19:37:54.304390 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:54.304356    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"
Sep 16 19:37:59.304719 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:59.304678    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"
Sep 16 19:37:59.306260 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:37:59.306231    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr" podUID="2f8919b9-22d7-435f-bebb-6382e0b970df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 16 19:38:09.304470 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:38:09.304422    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr" podUID="2f8919b9-22d7-435f-bebb-6382e0b970df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 16 19:38:19.305260 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:38:19.305207    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr" podUID="2f8919b9-22d7-435f-bebb-6382e0b970df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 16 19:38:29.305339 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:38:29.305290    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr" podUID="2f8919b9-22d7-435f-bebb-6382e0b970df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 16 19:38:39.305017 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:38:39.304964    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr" podUID="2f8919b9-22d7-435f-bebb-6382e0b970df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 16 19:38:49.305205 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:38:49.305157    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr" podUID="2f8919b9-22d7-435f-bebb-6382e0b970df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 16 19:38:59.304868 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:38:59.304764    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr" podUID="2f8919b9-22d7-435f-bebb-6382e0b970df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 16 19:39:09.314214 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:09.314181    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"
Sep 16 19:39:09.321671 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:09.321650    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"
Sep 16 19:39:33.338173 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:33.338144    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr_2f8919b9-22d7-435f-bebb-6382e0b970df/storage-initializer/0.log"
Sep 16 19:39:33.348732 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:33.348709    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr_2f8919b9-22d7-435f-bebb-6382e0b970df/main/0.log"
Sep 16 19:39:34.985326 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:34.985295    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"]
Sep 16 19:39:50.007537 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:50.007481    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr" podUID="2f8919b9-22d7-435f-bebb-6382e0b970df" containerName="main" containerID="cri-o://989fd4c010a442b4003f6253f8308e0d3453f386ea5cf5f08506d7f6d8346111" gracePeriod=45
Sep 16 19:39:50.315341 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:50.315243    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr" podUID="2f8919b9-22d7-435f-bebb-6382e0b970df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 16 19:39:50.651212 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:50.651188    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"
Sep 16 19:39:50.730096 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:50.730068    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f8919b9-22d7-435f-bebb-6382e0b970df-cabundle-cert\") pod \"2f8919b9-22d7-435f-bebb-6382e0b970df\" (UID: \"2f8919b9-22d7-435f-bebb-6382e0b970df\") "
Sep 16 19:39:50.730227 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:50.730104    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f8919b9-22d7-435f-bebb-6382e0b970df-home\") pod \"2f8919b9-22d7-435f-bebb-6382e0b970df\" (UID: \"2f8919b9-22d7-435f-bebb-6382e0b970df\") "
Sep 16 19:39:50.730227 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:50.730131    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f8919b9-22d7-435f-bebb-6382e0b970df-model-cache\") pod \"2f8919b9-22d7-435f-bebb-6382e0b970df\" (UID: \"2f8919b9-22d7-435f-bebb-6382e0b970df\") "
Sep 16 19:39:50.730227 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:50.730155    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f8919b9-22d7-435f-bebb-6382e0b970df-kserve-provision-location\") pod \"2f8919b9-22d7-435f-bebb-6382e0b970df\" (UID: \"2f8919b9-22d7-435f-bebb-6382e0b970df\") "
Sep 16 19:39:50.730227 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:50.730187    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f8919b9-22d7-435f-bebb-6382e0b970df-tls-certs\") pod \"2f8919b9-22d7-435f-bebb-6382e0b970df\" (UID: \"2f8919b9-22d7-435f-bebb-6382e0b970df\") "
Sep 16 19:39:50.730443 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:50.730244    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f8919b9-22d7-435f-bebb-6382e0b970df-tmp-dir\") pod \"2f8919b9-22d7-435f-bebb-6382e0b970df\" (UID: \"2f8919b9-22d7-435f-bebb-6382e0b970df\") "
Sep 16 19:39:50.730443 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:50.730290    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f8919b9-22d7-435f-bebb-6382e0b970df-dshm\") pod \"2f8919b9-22d7-435f-bebb-6382e0b970df\" (UID: \"2f8919b9-22d7-435f-bebb-6382e0b970df\") "
Sep 16 19:39:50.730443 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:50.730328    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8htk\" (UniqueName: \"kubernetes.io/projected/2f8919b9-22d7-435f-bebb-6382e0b970df-kube-api-access-h8htk\") pod \"2f8919b9-22d7-435f-bebb-6382e0b970df\" (UID: \"2f8919b9-22d7-435f-bebb-6382e0b970df\") "
Sep 16 19:39:50.730592 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:50.730459    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8919b9-22d7-435f-bebb-6382e0b970df-model-cache" (OuterVolumeSpecName: "model-cache") pod "2f8919b9-22d7-435f-bebb-6382e0b970df" (UID: "2f8919b9-22d7-435f-bebb-6382e0b970df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:39:50.730592 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:50.730468    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f8919b9-22d7-435f-bebb-6382e0b970df-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2f8919b9-22d7-435f-bebb-6382e0b970df" (UID: "2f8919b9-22d7-435f-bebb-6382e0b970df"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:39:50.730707 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:50.730607    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8919b9-22d7-435f-bebb-6382e0b970df-home" (OuterVolumeSpecName: "home") pod "2f8919b9-22d7-435f-bebb-6382e0b970df" (UID: "2f8919b9-22d7-435f-bebb-6382e0b970df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:39:50.730707 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:50.730683    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f8919b9-22d7-435f-bebb-6382e0b970df-cabundle-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:39:50.730707 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:50.730703    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f8919b9-22d7-435f-bebb-6382e0b970df-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:39:50.732555 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:50.732527    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f8919b9-22d7-435f-bebb-6382e0b970df-kube-api-access-h8htk" (OuterVolumeSpecName: "kube-api-access-h8htk") pod "2f8919b9-22d7-435f-bebb-6382e0b970df" (UID: "2f8919b9-22d7-435f-bebb-6382e0b970df"). InnerVolumeSpecName "kube-api-access-h8htk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:39:50.732703 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:50.732685    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f8919b9-22d7-435f-bebb-6382e0b970df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f8919b9-22d7-435f-bebb-6382e0b970df" (UID: "2f8919b9-22d7-435f-bebb-6382e0b970df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:39:50.732756 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:50.732703    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8919b9-22d7-435f-bebb-6382e0b970df-dshm" (OuterVolumeSpecName: "dshm") pod "2f8919b9-22d7-435f-bebb-6382e0b970df" (UID: "2f8919b9-22d7-435f-bebb-6382e0b970df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:39:50.742107 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:50.742085    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8919b9-22d7-435f-bebb-6382e0b970df-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2f8919b9-22d7-435f-bebb-6382e0b970df" (UID: "2f8919b9-22d7-435f-bebb-6382e0b970df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:39:50.752230 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:50.752209    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8919b9-22d7-435f-bebb-6382e0b970df-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2f8919b9-22d7-435f-bebb-6382e0b970df" (UID: "2f8919b9-22d7-435f-bebb-6382e0b970df"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:39:50.831334 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:50.831276    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f8919b9-22d7-435f-bebb-6382e0b970df-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:39:50.831334 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:50.831299    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f8919b9-22d7-435f-bebb-6382e0b970df-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:39:50.831334 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:50.831310    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8htk\" (UniqueName: \"kubernetes.io/projected/2f8919b9-22d7-435f-bebb-6382e0b970df-kube-api-access-h8htk\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:39:50.831334 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:50.831319    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f8919b9-22d7-435f-bebb-6382e0b970df-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:39:50.831334 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:50.831327    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f8919b9-22d7-435f-bebb-6382e0b970df-kserve-provision-location\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:39:50.831334 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:50.831335    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f8919b9-22d7-435f-bebb-6382e0b970df-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:39:51.299553 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:51.299520    2578 generic.go:356] "Generic (PLEG): container finished" podID="2f8919b9-22d7-435f-bebb-6382e0b970df" containerID="989fd4c010a442b4003f6253f8308e0d3453f386ea5cf5f08506d7f6d8346111" exitCode=0
Sep 16 19:39:51.300060 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:51.299599    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"
Sep 16 19:39:51.300060 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:51.299609    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr" event={"ID":"2f8919b9-22d7-435f-bebb-6382e0b970df","Type":"ContainerDied","Data":"989fd4c010a442b4003f6253f8308e0d3453f386ea5cf5f08506d7f6d8346111"}
Sep 16 19:39:51.300060 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:51.299689    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr" event={"ID":"2f8919b9-22d7-435f-bebb-6382e0b970df","Type":"ContainerDied","Data":"0c96b5e85169c6bd0f543a382fee6a53561bd70e21940b9c3849061f7f406895"}
Sep 16 19:39:51.300060 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:51.299730    2578 scope.go:117] "RemoveContainer" containerID="989fd4c010a442b4003f6253f8308e0d3453f386ea5cf5f08506d7f6d8346111"
Sep 16 19:39:51.309320 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:51.309303    2578 scope.go:117] "RemoveContainer" containerID="82e986f55bd6e4928abee8193bea5b14ce09cc11720944ffbf3fda96eff13745"
Sep 16 19:39:51.317548 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:51.317531    2578 scope.go:117] "RemoveContainer" containerID="989fd4c010a442b4003f6253f8308e0d3453f386ea5cf5f08506d7f6d8346111"
Sep 16 19:39:51.317821 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:39:51.317799    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"989fd4c010a442b4003f6253f8308e0d3453f386ea5cf5f08506d7f6d8346111\": container with ID starting with 989fd4c010a442b4003f6253f8308e0d3453f386ea5cf5f08506d7f6d8346111 not found: ID does not exist" containerID="989fd4c010a442b4003f6253f8308e0d3453f386ea5cf5f08506d7f6d8346111"
Sep 16 19:39:51.317868 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:51.317831    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"989fd4c010a442b4003f6253f8308e0d3453f386ea5cf5f08506d7f6d8346111"} err="failed to get container status \"989fd4c010a442b4003f6253f8308e0d3453f386ea5cf5f08506d7f6d8346111\": rpc error: code = NotFound desc = could not find container \"989fd4c010a442b4003f6253f8308e0d3453f386ea5cf5f08506d7f6d8346111\": container with ID starting with 989fd4c010a442b4003f6253f8308e0d3453f386ea5cf5f08506d7f6d8346111 not found: ID does not exist"
Sep 16 19:39:51.317868 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:51.317849    2578 scope.go:117] "RemoveContainer" containerID="82e986f55bd6e4928abee8193bea5b14ce09cc11720944ffbf3fda96eff13745"
Sep 16 19:39:51.318133 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:39:51.318112    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82e986f55bd6e4928abee8193bea5b14ce09cc11720944ffbf3fda96eff13745\": container with ID starting with 82e986f55bd6e4928abee8193bea5b14ce09cc11720944ffbf3fda96eff13745 not found: ID does not exist" containerID="82e986f55bd6e4928abee8193bea5b14ce09cc11720944ffbf3fda96eff13745"
Sep 16 19:39:51.318200 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:51.318141    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82e986f55bd6e4928abee8193bea5b14ce09cc11720944ffbf3fda96eff13745"} err="failed to get container status \"82e986f55bd6e4928abee8193bea5b14ce09cc11720944ffbf3fda96eff13745\": rpc error: code = NotFound desc = could not find container \"82e986f55bd6e4928abee8193bea5b14ce09cc11720944ffbf3fda96eff13745\": container with ID starting with 82e986f55bd6e4928abee8193bea5b14ce09cc11720944ffbf3fda96eff13745 not found: ID does not exist"
Sep 16 19:39:51.325392 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:51.325370    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"]
Sep 16 19:39:51.331441 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:51.331411    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-b8cd9ffb6-2s5kr"]
Sep 16 19:39:52.217617 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:39:52.217590    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f8919b9-22d7-435f-bebb-6382e0b970df" path="/var/lib/kubelet/pods/2f8919b9-22d7-435f-bebb-6382e0b970df/volumes"
Sep 16 19:40:02.437055 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:40:02.437009    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz"]
Sep 16 19:40:02.437449 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:40:02.437311    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f8919b9-22d7-435f-bebb-6382e0b970df" containerName="main"
Sep 16 19:40:02.437449 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:40:02.437324    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f8919b9-22d7-435f-bebb-6382e0b970df" containerName="main"
Sep 16 19:40:02.437449 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:40:02.437333    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f8919b9-22d7-435f-bebb-6382e0b970df" containerName="storage-initializer"
Sep 16 19:40:02.437449 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:40:02.437339    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f8919b9-22d7-435f-bebb-6382e0b970df" containerName="storage-initializer"
Sep 16 19:40:02.437449 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:40:02.437397    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f8919b9-22d7-435f-bebb-6382e0b970df" containerName="main"
Sep 16 19:40:02.440210 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:40:02.440191    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz"
Sep 16 19:40:02.443862 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:40:02.443836    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 16 19:40:02.443862 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:40:02.443855    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 16 19:40:02.444060 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:40:02.443847    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-fmg6t\""
Sep 16 19:40:02.444060 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:40:02.443881    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 16 19:40:02.444060 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:40:02.443836    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-zmx4k\""
Sep 16 19:40:02.451135 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:40:02.451111    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz"]
Sep 16 19:40:02.613498 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:40:02.613468    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53c9b53c-7d3d-4ad7-afc6-8f81499fef4f-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz\" (UID: \"53c9b53c-7d3d-4ad7-afc6-8f81499fef4f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz"
Sep 16 19:40:02.613616 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:40:02.613567    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5rxr\" (UniqueName: \"kubernetes.io/projected/53c9b53c-7d3d-4ad7-afc6-8f81499fef4f-kube-api-access-p5rxr\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz\" (UID: \"53c9b53c-7d3d-4ad7-afc6-8f81499fef4f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz"
Sep 16 19:40:02.714211 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:40:02.714134    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5rxr\" (UniqueName: \"kubernetes.io/projected/53c9b53c-7d3d-4ad7-afc6-8f81499fef4f-kube-api-access-p5rxr\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz\" (UID: \"53c9b53c-7d3d-4ad7-afc6-8f81499fef4f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz"
Sep 16 19:40:02.714211 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:40:02.714175    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53c9b53c-7d3d-4ad7-afc6-8f81499fef4f-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz\" (UID: \"53c9b53c-7d3d-4ad7-afc6-8f81499fef4f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz"
Sep 16 19:40:02.716603 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:40:02.716574    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53c9b53c-7d3d-4ad7-afc6-8f81499fef4f-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz\" (UID: \"53c9b53c-7d3d-4ad7-afc6-8f81499fef4f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz"
Sep 16 19:40:02.723002 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:40:02.722971    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5rxr\" (UniqueName: \"kubernetes.io/projected/53c9b53c-7d3d-4ad7-afc6-8f81499fef4f-kube-api-access-p5rxr\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz\" (UID: \"53c9b53c-7d3d-4ad7-afc6-8f81499fef4f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz"
Sep 16 19:40:02.751307 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:40:02.751278    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz"
Sep 16 19:40:03.026617 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:40:03.026590    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz"]
Sep 16 19:40:03.028394 ip-10-0-142-80 kubenswrapper[2578]: W0916 19:40:03.028363    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53c9b53c_7d3d_4ad7_afc6_8f81499fef4f.slice/crio-b0db5dab788259dcdb2793aa1a7e7ebfa5830bf31ce83ca27e9ada541fc7c52e WatchSource:0}: Error finding container b0db5dab788259dcdb2793aa1a7e7ebfa5830bf31ce83ca27e9ada541fc7c52e: Status 404 returned error can't find the container with id b0db5dab788259dcdb2793aa1a7e7ebfa5830bf31ce83ca27e9ada541fc7c52e
Sep 16 19:40:03.368642 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:40:03.368544    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz" event={"ID":"53c9b53c-7d3d-4ad7-afc6-8f81499fef4f","Type":"ContainerStarted","Data":"bae498724005d8854dbb01d17aa8a578a97e2a04a0c5c2c0b71666194a225728"}
Sep 16 19:40:03.368642 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:40:03.368604    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz" event={"ID":"53c9b53c-7d3d-4ad7-afc6-8f81499fef4f","Type":"ContainerStarted","Data":"b0db5dab788259dcdb2793aa1a7e7ebfa5830bf31ce83ca27e9ada541fc7c52e"}
Sep 16 19:40:03.368870 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:40:03.368651    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz"
Sep 16 19:40:03.386736 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:40:03.386690    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz" podStartSLOduration=1.386677648 podStartE2EDuration="1.386677648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:40:03.38473733 +0000 UTC m=+3571.619670773" watchObservedRunningTime="2026-09-16 19:40:03.386677648 +0000 UTC m=+3571.621611092"
Sep 16 19:40:34.375803 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:40:34.375767    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz"
Sep 16 19:42:19.767661 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:42:19.767621    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz"]
Sep 16 19:42:24.376128 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:42:24.376077    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz" podUID="53c9b53c-7d3d-4ad7-afc6-8f81499fef4f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:42:34.375391 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:42:34.375335    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz" podUID="53c9b53c-7d3d-4ad7-afc6-8f81499fef4f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:42:34.786437 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:42:34.786374    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz" podUID="53c9b53c-7d3d-4ad7-afc6-8f81499fef4f" containerName="main" containerID="cri-o://bae498724005d8854dbb01d17aa8a578a97e2a04a0c5c2c0b71666194a225728" gracePeriod=45
Sep 16 19:42:44.381581 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:42:44.381535    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz" podUID="53c9b53c-7d3d-4ad7-afc6-8f81499fef4f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:42:44.382189 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:42:44.381671    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz"
Sep 16 19:42:54.376122 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:42:54.376083    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz" podUID="53c9b53c-7d3d-4ad7-afc6-8f81499fef4f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:43:04.376313 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:04.376270    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz" podUID="53c9b53c-7d3d-4ad7-afc6-8f81499fef4f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:43:05.039546 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:05.039520    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz"
Sep 16 19:43:05.113394 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:05.113293    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5rxr\" (UniqueName: \"kubernetes.io/projected/53c9b53c-7d3d-4ad7-afc6-8f81499fef4f-kube-api-access-p5rxr\") pod \"53c9b53c-7d3d-4ad7-afc6-8f81499fef4f\" (UID: \"53c9b53c-7d3d-4ad7-afc6-8f81499fef4f\") "
Sep 16 19:43:05.113394 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:05.113386    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53c9b53c-7d3d-4ad7-afc6-8f81499fef4f-tls-certs\") pod \"53c9b53c-7d3d-4ad7-afc6-8f81499fef4f\" (UID: \"53c9b53c-7d3d-4ad7-afc6-8f81499fef4f\") "
Sep 16 19:43:05.115536 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:05.115504    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53c9b53c-7d3d-4ad7-afc6-8f81499fef4f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "53c9b53c-7d3d-4ad7-afc6-8f81499fef4f" (UID: "53c9b53c-7d3d-4ad7-afc6-8f81499fef4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:43:05.115621 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:05.115579    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53c9b53c-7d3d-4ad7-afc6-8f81499fef4f-kube-api-access-p5rxr" (OuterVolumeSpecName: "kube-api-access-p5rxr") pod "53c9b53c-7d3d-4ad7-afc6-8f81499fef4f" (UID: "53c9b53c-7d3d-4ad7-afc6-8f81499fef4f"). InnerVolumeSpecName "kube-api-access-p5rxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:43:05.214409 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:05.214370    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53c9b53c-7d3d-4ad7-afc6-8f81499fef4f-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:43:05.214409 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:05.214402    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5rxr\" (UniqueName: \"kubernetes.io/projected/53c9b53c-7d3d-4ad7-afc6-8f81499fef4f-kube-api-access-p5rxr\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:43:05.267758 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:05.267717    2578 generic.go:356] "Generic (PLEG): container finished" podID="53c9b53c-7d3d-4ad7-afc6-8f81499fef4f" containerID="bae498724005d8854dbb01d17aa8a578a97e2a04a0c5c2c0b71666194a225728" exitCode=0
Sep 16 19:43:05.267920 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:05.267760    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz"
Sep 16 19:43:05.267920 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:05.267813    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz" event={"ID":"53c9b53c-7d3d-4ad7-afc6-8f81499fef4f","Type":"ContainerDied","Data":"bae498724005d8854dbb01d17aa8a578a97e2a04a0c5c2c0b71666194a225728"}
Sep 16 19:43:05.267920 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:05.267857    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz" event={"ID":"53c9b53c-7d3d-4ad7-afc6-8f81499fef4f","Type":"ContainerDied","Data":"b0db5dab788259dcdb2793aa1a7e7ebfa5830bf31ce83ca27e9ada541fc7c52e"}
Sep 16 19:43:05.267920 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:05.267878    2578 scope.go:117] "RemoveContainer" containerID="bae498724005d8854dbb01d17aa8a578a97e2a04a0c5c2c0b71666194a225728"
Sep 16 19:43:05.280353 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:05.280333    2578 scope.go:117] "RemoveContainer" containerID="bae498724005d8854dbb01d17aa8a578a97e2a04a0c5c2c0b71666194a225728"
Sep 16 19:43:05.280670 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:43:05.280644    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bae498724005d8854dbb01d17aa8a578a97e2a04a0c5c2c0b71666194a225728\": container with ID starting with bae498724005d8854dbb01d17aa8a578a97e2a04a0c5c2c0b71666194a225728 not found: ID does not exist" containerID="bae498724005d8854dbb01d17aa8a578a97e2a04a0c5c2c0b71666194a225728"
Sep 16 19:43:05.280750 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:05.280681    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bae498724005d8854dbb01d17aa8a578a97e2a04a0c5c2c0b71666194a225728"} err="failed to get container status \"bae498724005d8854dbb01d17aa8a578a97e2a04a0c5c2c0b71666194a225728\": rpc error: code = NotFound desc = could not find container \"bae498724005d8854dbb01d17aa8a578a97e2a04a0c5c2c0b71666194a225728\": container with ID starting with bae498724005d8854dbb01d17aa8a578a97e2a04a0c5c2c0b71666194a225728 not found: ID does not exist"
Sep 16 19:43:05.296856 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:05.296802    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz"]
Sep 16 19:43:05.298909 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:05.298875    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-rfzdz"]
Sep 16 19:43:06.221230 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:06.219708    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53c9b53c-7d3d-4ad7-afc6-8f81499fef4f" path="/var/lib/kubelet/pods/53c9b53c-7d3d-4ad7-afc6-8f81499fef4f/volumes"
Sep 16 19:43:51.535029 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.534970    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"]
Sep 16 19:43:51.535672 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.535561    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53c9b53c-7d3d-4ad7-afc6-8f81499fef4f" containerName="main"
Sep 16 19:43:51.535672 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.535584    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="53c9b53c-7d3d-4ad7-afc6-8f81499fef4f" containerName="main"
Sep 16 19:43:51.535672 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.535670    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="53c9b53c-7d3d-4ad7-afc6-8f81499fef4f" containerName="main"
Sep 16 19:43:51.539194 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.539174    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:43:51.541929 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.541906    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-b4scm\""
Sep 16 19:43:51.542083 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.541957    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 16 19:43:51.542083 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.542052    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 16 19:43:51.542206 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.542166    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 16 19:43:51.542206 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.542181    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 16 19:43:51.542952 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.542936    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-mknvd\""
Sep 16 19:43:51.543062 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.542992    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:43:51.555612 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.555585    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"]
Sep 16 19:43:51.603130 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.603090    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c61b9272-3920-49e7-a52d-924957bf8b20-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k\" (UID: \"c61b9272-3920-49e7-a52d-924957bf8b20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:43:51.603130 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.603128    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c61b9272-3920-49e7-a52d-924957bf8b20-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k\" (UID: \"c61b9272-3920-49e7-a52d-924957bf8b20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:43:51.603357 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.603158    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c61b9272-3920-49e7-a52d-924957bf8b20-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k\" (UID: \"c61b9272-3920-49e7-a52d-924957bf8b20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:43:51.603357 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.603210    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c61b9272-3920-49e7-a52d-924957bf8b20-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k\" (UID: \"c61b9272-3920-49e7-a52d-924957bf8b20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:43:51.603357 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.603249    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c61b9272-3920-49e7-a52d-924957bf8b20-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k\" (UID: \"c61b9272-3920-49e7-a52d-924957bf8b20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:43:51.603357 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.603279    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c61b9272-3920-49e7-a52d-924957bf8b20-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k\" (UID: \"c61b9272-3920-49e7-a52d-924957bf8b20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:43:51.603357 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.603298    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvh8v\" (UniqueName: \"kubernetes.io/projected/c61b9272-3920-49e7-a52d-924957bf8b20-kube-api-access-jvh8v\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k\" (UID: \"c61b9272-3920-49e7-a52d-924957bf8b20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:43:51.603357 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.603342    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c61b9272-3920-49e7-a52d-924957bf8b20-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k\" (UID: \"c61b9272-3920-49e7-a52d-924957bf8b20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:43:51.704784 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.704746    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c61b9272-3920-49e7-a52d-924957bf8b20-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k\" (UID: \"c61b9272-3920-49e7-a52d-924957bf8b20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:43:51.704784 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.704786    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c61b9272-3920-49e7-a52d-924957bf8b20-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k\" (UID: \"c61b9272-3920-49e7-a52d-924957bf8b20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:43:51.705025 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.704813    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c61b9272-3920-49e7-a52d-924957bf8b20-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k\" (UID: \"c61b9272-3920-49e7-a52d-924957bf8b20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:43:51.705025 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.704851    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c61b9272-3920-49e7-a52d-924957bf8b20-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k\" (UID: \"c61b9272-3920-49e7-a52d-924957bf8b20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:43:51.705025 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.704903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c61b9272-3920-49e7-a52d-924957bf8b20-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k\" (UID: \"c61b9272-3920-49e7-a52d-924957bf8b20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:43:51.705025 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.704946    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c61b9272-3920-49e7-a52d-924957bf8b20-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k\" (UID: \"c61b9272-3920-49e7-a52d-924957bf8b20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:43:51.705025 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.704976    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvh8v\" (UniqueName: \"kubernetes.io/projected/c61b9272-3920-49e7-a52d-924957bf8b20-kube-api-access-jvh8v\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k\" (UID: \"c61b9272-3920-49e7-a52d-924957bf8b20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:43:51.705025 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.705003    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c61b9272-3920-49e7-a52d-924957bf8b20-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k\" (UID: \"c61b9272-3920-49e7-a52d-924957bf8b20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:43:51.705374 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.705189    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c61b9272-3920-49e7-a52d-924957bf8b20-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k\" (UID: \"c61b9272-3920-49e7-a52d-924957bf8b20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:43:51.705374 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.705258    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c61b9272-3920-49e7-a52d-924957bf8b20-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k\" (UID: \"c61b9272-3920-49e7-a52d-924957bf8b20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:43:51.705478 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.705406    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c61b9272-3920-49e7-a52d-924957bf8b20-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k\" (UID: \"c61b9272-3920-49e7-a52d-924957bf8b20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:43:51.705478 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.705438    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c61b9272-3920-49e7-a52d-924957bf8b20-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k\" (UID: \"c61b9272-3920-49e7-a52d-924957bf8b20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:43:51.705722 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.705702    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c61b9272-3920-49e7-a52d-924957bf8b20-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k\" (UID: \"c61b9272-3920-49e7-a52d-924957bf8b20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:43:51.707284 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.707260    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c61b9272-3920-49e7-a52d-924957bf8b20-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k\" (UID: \"c61b9272-3920-49e7-a52d-924957bf8b20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:43:51.707426 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.707409    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c61b9272-3920-49e7-a52d-924957bf8b20-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k\" (UID: \"c61b9272-3920-49e7-a52d-924957bf8b20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:43:51.715398 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.715374    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvh8v\" (UniqueName: \"kubernetes.io/projected/c61b9272-3920-49e7-a52d-924957bf8b20-kube-api-access-jvh8v\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k\" (UID: \"c61b9272-3920-49e7-a52d-924957bf8b20\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:43:51.849836 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:51.849738    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:43:52.134636 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:52.134603    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"]
Sep 16 19:43:52.136507 ip-10-0-142-80 kubenswrapper[2578]: W0916 19:43:52.136472    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61b9272_3920_49e7_a52d_924957bf8b20.slice/crio-ad500fced3ab75d31299ffdc363746d557021622b223be21b4c324414b676826 WatchSource:0}: Error finding container ad500fced3ab75d31299ffdc363746d557021622b223be21b4c324414b676826: Status 404 returned error can't find the container with id ad500fced3ab75d31299ffdc363746d557021622b223be21b4c324414b676826
Sep 16 19:43:52.512122 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:52.512076    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k" event={"ID":"c61b9272-3920-49e7-a52d-924957bf8b20","Type":"ContainerStarted","Data":"8f20db4aebf82e5bb2376deab554061068fd2250339b87b220ee84ff630313ea"}
Sep 16 19:43:52.512311 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:52.512151    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:43:52.512311 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:52.512177    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k" event={"ID":"c61b9272-3920-49e7-a52d-924957bf8b20","Type":"ContainerStarted","Data":"ad500fced3ab75d31299ffdc363746d557021622b223be21b4c324414b676826"}
Sep 16 19:43:53.522306 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:53.522254    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k" event={"ID":"c61b9272-3920-49e7-a52d-924957bf8b20","Type":"ContainerStarted","Data":"b61607809786ba9449e980dc1806c1521ef5acda8549f1044da6f428afd69366"}
Sep 16 19:43:55.535292 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:55.535254    2578 generic.go:356] "Generic (PLEG): container finished" podID="c61b9272-3920-49e7-a52d-924957bf8b20" containerID="b61607809786ba9449e980dc1806c1521ef5acda8549f1044da6f428afd69366" exitCode=0
Sep 16 19:43:55.535748 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:55.535330    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k" event={"ID":"c61b9272-3920-49e7-a52d-924957bf8b20","Type":"ContainerDied","Data":"b61607809786ba9449e980dc1806c1521ef5acda8549f1044da6f428afd69366"}
Sep 16 19:43:56.550512 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:56.550471    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k" event={"ID":"c61b9272-3920-49e7-a52d-924957bf8b20","Type":"ContainerStarted","Data":"6386b2a6be28f133e1e67adcb4acbcf7afde595fd9a4640ae76f3dd61bb4d851"}
Sep 16 19:43:56.577803 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:43:56.577736    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k" podStartSLOduration=3.261899669 podStartE2EDuration="5.577713872s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.315814203s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:43:56.576699299 +0000 UTC m=+3804.811632741" watchObservedRunningTime="2026-09-16 19:43:56.577713872 +0000 UTC m=+3804.812647308"
Sep 16 19:44:01.850725 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:44:01.850677    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:44:01.851173 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:44:01.850771    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:44:01.851947 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:44:01.851921    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k" podUID="c61b9272-3920-49e7-a52d-924957bf8b20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8001/health\": dial tcp 10.133.0.70:8001: connect: connection refused"
Sep 16 19:44:11.850808 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:44:11.850751    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k" podUID="c61b9272-3920-49e7-a52d-924957bf8b20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8001/health\": dial tcp 10.133.0.70:8001: connect: connection refused"
Sep 16 19:44:11.862782 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:44:11.862750    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:44:21.850478 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:44:21.850420    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k" podUID="c61b9272-3920-49e7-a52d-924957bf8b20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8001/health\": dial tcp 10.133.0.70:8001: connect: connection refused"
Sep 16 19:44:31.850650 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:44:31.850603    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k" podUID="c61b9272-3920-49e7-a52d-924957bf8b20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8001/health\": dial tcp 10.133.0.70:8001: connect: connection refused"
Sep 16 19:44:41.851099 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:44:41.851020    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k" podUID="c61b9272-3920-49e7-a52d-924957bf8b20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8001/health\": dial tcp 10.133.0.70:8001: connect: connection refused"
Sep 16 19:44:51.850844 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:44:51.850792    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k" podUID="c61b9272-3920-49e7-a52d-924957bf8b20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8001/health\": dial tcp 10.133.0.70:8001: connect: connection refused"
Sep 16 19:45:01.851301 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:45:01.851183    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k" podUID="c61b9272-3920-49e7-a52d-924957bf8b20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8001/health\": dial tcp 10.133.0.70:8001: connect: connection refused"
Sep 16 19:45:11.850516 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:45:11.850308    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k" podUID="c61b9272-3920-49e7-a52d-924957bf8b20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8001/health\": dial tcp 10.133.0.70:8001: connect: connection refused"
Sep 16 19:45:21.850693 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:45:21.850638    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k" podUID="c61b9272-3920-49e7-a52d-924957bf8b20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8001/health\": dial tcp 10.133.0.70:8001: connect: connection refused"
Sep 16 19:45:31.850829 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:45:31.850784    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k" podUID="c61b9272-3920-49e7-a52d-924957bf8b20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8001/health\": dial tcp 10.133.0.70:8001: connect: connection refused"
Sep 16 19:45:41.850984 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:45:41.850930    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k" podUID="c61b9272-3920-49e7-a52d-924957bf8b20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8001/health\": dial tcp 10.133.0.70:8001: connect: connection refused"
Sep 16 19:45:51.865694 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:45:51.865658    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:45:51.885243 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:45:51.885216    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:46:11.494416 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:11.494327    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"]
Sep 16 19:46:26.518841 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:26.518720    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k" podUID="c61b9272-3920-49e7-a52d-924957bf8b20" containerName="main" containerID="cri-o://6386b2a6be28f133e1e67adcb4acbcf7afde595fd9a4640ae76f3dd61bb4d851" gracePeriod=45
Sep 16 19:46:27.060712 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.060664    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k" podUID="c61b9272-3920-49e7-a52d-924957bf8b20" containerName="llm-d-routing-sidecar" containerID="cri-o://8f20db4aebf82e5bb2376deab554061068fd2250339b87b220ee84ff630313ea" gracePeriod=45
Sep 16 19:46:27.306318 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.306293    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:46:27.328679 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.328650    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c61b9272-3920-49e7-a52d-924957bf8b20-home\") pod \"c61b9272-3920-49e7-a52d-924957bf8b20\" (UID: \"c61b9272-3920-49e7-a52d-924957bf8b20\") "
Sep 16 19:46:27.328822 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.328699    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c61b9272-3920-49e7-a52d-924957bf8b20-model-cache\") pod \"c61b9272-3920-49e7-a52d-924957bf8b20\" (UID: \"c61b9272-3920-49e7-a52d-924957bf8b20\") "
Sep 16 19:46:27.328822 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.328723    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c61b9272-3920-49e7-a52d-924957bf8b20-dshm\") pod \"c61b9272-3920-49e7-a52d-924957bf8b20\" (UID: \"c61b9272-3920-49e7-a52d-924957bf8b20\") "
Sep 16 19:46:27.328822 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.328760    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvh8v\" (UniqueName: \"kubernetes.io/projected/c61b9272-3920-49e7-a52d-924957bf8b20-kube-api-access-jvh8v\") pod \"c61b9272-3920-49e7-a52d-924957bf8b20\" (UID: \"c61b9272-3920-49e7-a52d-924957bf8b20\") "
Sep 16 19:46:27.328995 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.328826    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c61b9272-3920-49e7-a52d-924957bf8b20-cabundle-cert\") pod \"c61b9272-3920-49e7-a52d-924957bf8b20\" (UID: \"c61b9272-3920-49e7-a52d-924957bf8b20\") "
Sep 16 19:46:27.328995 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.328859    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c61b9272-3920-49e7-a52d-924957bf8b20-tmp-dir\") pod \"c61b9272-3920-49e7-a52d-924957bf8b20\" (UID: \"c61b9272-3920-49e7-a52d-924957bf8b20\") "
Sep 16 19:46:27.328995 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.328904    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c61b9272-3920-49e7-a52d-924957bf8b20-tls-certs\") pod \"c61b9272-3920-49e7-a52d-924957bf8b20\" (UID: \"c61b9272-3920-49e7-a52d-924957bf8b20\") "
Sep 16 19:46:27.328995 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.328929    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c61b9272-3920-49e7-a52d-924957bf8b20-kserve-provision-location\") pod \"c61b9272-3920-49e7-a52d-924957bf8b20\" (UID: \"c61b9272-3920-49e7-a52d-924957bf8b20\") "
Sep 16 19:46:27.329179 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.328988    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61b9272-3920-49e7-a52d-924957bf8b20-model-cache" (OuterVolumeSpecName: "model-cache") pod "c61b9272-3920-49e7-a52d-924957bf8b20" (UID: "c61b9272-3920-49e7-a52d-924957bf8b20"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:46:27.329244 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.329215    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c61b9272-3920-49e7-a52d-924957bf8b20-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:46:27.329421 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.329254    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61b9272-3920-49e7-a52d-924957bf8b20-home" (OuterVolumeSpecName: "home") pod "c61b9272-3920-49e7-a52d-924957bf8b20" (UID: "c61b9272-3920-49e7-a52d-924957bf8b20"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:46:27.329706 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.329660    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c61b9272-3920-49e7-a52d-924957bf8b20-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c61b9272-3920-49e7-a52d-924957bf8b20" (UID: "c61b9272-3920-49e7-a52d-924957bf8b20"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:46:27.332487 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.332459    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c61b9272-3920-49e7-a52d-924957bf8b20-kube-api-access-jvh8v" (OuterVolumeSpecName: "kube-api-access-jvh8v") pod "c61b9272-3920-49e7-a52d-924957bf8b20" (UID: "c61b9272-3920-49e7-a52d-924957bf8b20"). InnerVolumeSpecName "kube-api-access-jvh8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:46:27.333699 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.333654    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c61b9272-3920-49e7-a52d-924957bf8b20-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c61b9272-3920-49e7-a52d-924957bf8b20" (UID: "c61b9272-3920-49e7-a52d-924957bf8b20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:46:27.333699 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.333666    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61b9272-3920-49e7-a52d-924957bf8b20-dshm" (OuterVolumeSpecName: "dshm") pod "c61b9272-3920-49e7-a52d-924957bf8b20" (UID: "c61b9272-3920-49e7-a52d-924957bf8b20"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:46:27.346682 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.346648    2578 generic.go:356] "Generic (PLEG): container finished" podID="c61b9272-3920-49e7-a52d-924957bf8b20" containerID="6386b2a6be28f133e1e67adcb4acbcf7afde595fd9a4640ae76f3dd61bb4d851" exitCode=0
Sep 16 19:46:27.346682 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.346672    2578 generic.go:356] "Generic (PLEG): container finished" podID="c61b9272-3920-49e7-a52d-924957bf8b20" containerID="8f20db4aebf82e5bb2376deab554061068fd2250339b87b220ee84ff630313ea" exitCode=0
Sep 16 19:46:27.346839 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.346710    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"
Sep 16 19:46:27.346839 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.346738    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k" event={"ID":"c61b9272-3920-49e7-a52d-924957bf8b20","Type":"ContainerDied","Data":"6386b2a6be28f133e1e67adcb4acbcf7afde595fd9a4640ae76f3dd61bb4d851"}
Sep 16 19:46:27.346839 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.346806    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k" event={"ID":"c61b9272-3920-49e7-a52d-924957bf8b20","Type":"ContainerDied","Data":"8f20db4aebf82e5bb2376deab554061068fd2250339b87b220ee84ff630313ea"}
Sep 16 19:46:27.346839 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.346830    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k" event={"ID":"c61b9272-3920-49e7-a52d-924957bf8b20","Type":"ContainerDied","Data":"ad500fced3ab75d31299ffdc363746d557021622b223be21b4c324414b676826"}
Sep 16 19:46:27.346992 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.346858    2578 scope.go:117] "RemoveContainer" containerID="6386b2a6be28f133e1e67adcb4acbcf7afde595fd9a4640ae76f3dd61bb4d851"
Sep 16 19:46:27.347758 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.347729    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61b9272-3920-49e7-a52d-924957bf8b20-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c61b9272-3920-49e7-a52d-924957bf8b20" (UID: "c61b9272-3920-49e7-a52d-924957bf8b20"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:46:27.355590 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.355565    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c61b9272-3920-49e7-a52d-924957bf8b20-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c61b9272-3920-49e7-a52d-924957bf8b20" (UID: "c61b9272-3920-49e7-a52d-924957bf8b20"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:46:27.356394 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.356376    2578 scope.go:117] "RemoveContainer" containerID="b61607809786ba9449e980dc1806c1521ef5acda8549f1044da6f428afd69366"
Sep 16 19:46:27.364467 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.364452    2578 scope.go:117] "RemoveContainer" containerID="8f20db4aebf82e5bb2376deab554061068fd2250339b87b220ee84ff630313ea"
Sep 16 19:46:27.371958 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.371943    2578 scope.go:117] "RemoveContainer" containerID="6386b2a6be28f133e1e67adcb4acbcf7afde595fd9a4640ae76f3dd61bb4d851"
Sep 16 19:46:27.372217 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:46:27.372198    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6386b2a6be28f133e1e67adcb4acbcf7afde595fd9a4640ae76f3dd61bb4d851\": container with ID starting with 6386b2a6be28f133e1e67adcb4acbcf7afde595fd9a4640ae76f3dd61bb4d851 not found: ID does not exist" containerID="6386b2a6be28f133e1e67adcb4acbcf7afde595fd9a4640ae76f3dd61bb4d851"
Sep 16 19:46:27.372276 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.372223    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6386b2a6be28f133e1e67adcb4acbcf7afde595fd9a4640ae76f3dd61bb4d851"} err="failed to get container status \"6386b2a6be28f133e1e67adcb4acbcf7afde595fd9a4640ae76f3dd61bb4d851\": rpc error: code = NotFound desc = could not find container \"6386b2a6be28f133e1e67adcb4acbcf7afde595fd9a4640ae76f3dd61bb4d851\": container with ID starting with 6386b2a6be28f133e1e67adcb4acbcf7afde595fd9a4640ae76f3dd61bb4d851 not found: ID does not exist"
Sep 16 19:46:27.372276 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.372239    2578 scope.go:117] "RemoveContainer" containerID="b61607809786ba9449e980dc1806c1521ef5acda8549f1044da6f428afd69366"
Sep 16 19:46:27.372476 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:46:27.372459    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b61607809786ba9449e980dc1806c1521ef5acda8549f1044da6f428afd69366\": container with ID starting with b61607809786ba9449e980dc1806c1521ef5acda8549f1044da6f428afd69366 not found: ID does not exist" containerID="b61607809786ba9449e980dc1806c1521ef5acda8549f1044da6f428afd69366"
Sep 16 19:46:27.372518 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.372483    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b61607809786ba9449e980dc1806c1521ef5acda8549f1044da6f428afd69366"} err="failed to get container status \"b61607809786ba9449e980dc1806c1521ef5acda8549f1044da6f428afd69366\": rpc error: code = NotFound desc = could not find container \"b61607809786ba9449e980dc1806c1521ef5acda8549f1044da6f428afd69366\": container with ID starting with b61607809786ba9449e980dc1806c1521ef5acda8549f1044da6f428afd69366 not found: ID does not exist"
Sep 16 19:46:27.372518 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.372503    2578 scope.go:117] "RemoveContainer" containerID="8f20db4aebf82e5bb2376deab554061068fd2250339b87b220ee84ff630313ea"
Sep 16 19:46:27.372773 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:46:27.372746    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f20db4aebf82e5bb2376deab554061068fd2250339b87b220ee84ff630313ea\": container with ID starting with 8f20db4aebf82e5bb2376deab554061068fd2250339b87b220ee84ff630313ea not found: ID does not exist" containerID="8f20db4aebf82e5bb2376deab554061068fd2250339b87b220ee84ff630313ea"
Sep 16 19:46:27.372821 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.372779    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f20db4aebf82e5bb2376deab554061068fd2250339b87b220ee84ff630313ea"} err="failed to get container status \"8f20db4aebf82e5bb2376deab554061068fd2250339b87b220ee84ff630313ea\": rpc error: code = NotFound desc = could not find container \"8f20db4aebf82e5bb2376deab554061068fd2250339b87b220ee84ff630313ea\": container with ID starting with 8f20db4aebf82e5bb2376deab554061068fd2250339b87b220ee84ff630313ea not found: ID does not exist"
Sep 16 19:46:27.372821 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.372797    2578 scope.go:117] "RemoveContainer" containerID="6386b2a6be28f133e1e67adcb4acbcf7afde595fd9a4640ae76f3dd61bb4d851"
Sep 16 19:46:27.373050 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.373021    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6386b2a6be28f133e1e67adcb4acbcf7afde595fd9a4640ae76f3dd61bb4d851"} err="failed to get container status \"6386b2a6be28f133e1e67adcb4acbcf7afde595fd9a4640ae76f3dd61bb4d851\": rpc error: code = NotFound desc = could not find container \"6386b2a6be28f133e1e67adcb4acbcf7afde595fd9a4640ae76f3dd61bb4d851\": container with ID starting with 6386b2a6be28f133e1e67adcb4acbcf7afde595fd9a4640ae76f3dd61bb4d851 not found: ID does not exist"
Sep 16 19:46:27.373112 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.373054    2578 scope.go:117] "RemoveContainer" containerID="b61607809786ba9449e980dc1806c1521ef5acda8549f1044da6f428afd69366"
Sep 16 19:46:27.373321 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.373302    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b61607809786ba9449e980dc1806c1521ef5acda8549f1044da6f428afd69366"} err="failed to get container status \"b61607809786ba9449e980dc1806c1521ef5acda8549f1044da6f428afd69366\": rpc error: code = NotFound desc = could not find container \"b61607809786ba9449e980dc1806c1521ef5acda8549f1044da6f428afd69366\": container with ID starting with b61607809786ba9449e980dc1806c1521ef5acda8549f1044da6f428afd69366 not found: ID does not exist"
Sep 16 19:46:27.373376 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.373321    2578 scope.go:117] "RemoveContainer" containerID="8f20db4aebf82e5bb2376deab554061068fd2250339b87b220ee84ff630313ea"
Sep 16 19:46:27.373570 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.373550    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f20db4aebf82e5bb2376deab554061068fd2250339b87b220ee84ff630313ea"} err="failed to get container status \"8f20db4aebf82e5bb2376deab554061068fd2250339b87b220ee84ff630313ea\": rpc error: code = NotFound desc = could not find container \"8f20db4aebf82e5bb2376deab554061068fd2250339b87b220ee84ff630313ea\": container with ID starting with 8f20db4aebf82e5bb2376deab554061068fd2250339b87b220ee84ff630313ea not found: ID does not exist"
Sep 16 19:46:27.430112 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.430084    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c61b9272-3920-49e7-a52d-924957bf8b20-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:46:27.430112 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.430107    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c61b9272-3920-49e7-a52d-924957bf8b20-kserve-provision-location\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:46:27.430248 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.430119    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c61b9272-3920-49e7-a52d-924957bf8b20-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:46:27.430248 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.430128    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c61b9272-3920-49e7-a52d-924957bf8b20-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:46:27.430248 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.430136    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvh8v\" (UniqueName: \"kubernetes.io/projected/c61b9272-3920-49e7-a52d-924957bf8b20-kube-api-access-jvh8v\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:46:27.430248 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.430144    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c61b9272-3920-49e7-a52d-924957bf8b20-cabundle-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:46:27.430248 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.430152    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c61b9272-3920-49e7-a52d-924957bf8b20-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:46:27.673937 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.673903    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"]
Sep 16 19:46:27.679199 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:27.679161    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59ddbb77bbsj57k"]
Sep 16 19:46:28.217218 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:46:28.217183    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c61b9272-3920-49e7-a52d-924957bf8b20" path="/var/lib/kubelet/pods/c61b9272-3920-49e7-a52d-924957bf8b20/volumes"
Sep 16 19:47:05.046115 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.046068    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"]
Sep 16 19:47:05.046761 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.046524    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c61b9272-3920-49e7-a52d-924957bf8b20" containerName="main"
Sep 16 19:47:05.046761 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.046545    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c61b9272-3920-49e7-a52d-924957bf8b20" containerName="main"
Sep 16 19:47:05.046761 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.046558    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c61b9272-3920-49e7-a52d-924957bf8b20" containerName="storage-initializer"
Sep 16 19:47:05.046761 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.046566    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c61b9272-3920-49e7-a52d-924957bf8b20" containerName="storage-initializer"
Sep 16 19:47:05.046761 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.046579    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c61b9272-3920-49e7-a52d-924957bf8b20" containerName="llm-d-routing-sidecar"
Sep 16 19:47:05.046761 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.046587    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c61b9272-3920-49e7-a52d-924957bf8b20" containerName="llm-d-routing-sidecar"
Sep 16 19:47:05.046761 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.046686    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c61b9272-3920-49e7-a52d-924957bf8b20" containerName="main"
Sep 16 19:47:05.046761 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.046700    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c61b9272-3920-49e7-a52d-924957bf8b20" containerName="llm-d-routing-sidecar"
Sep 16 19:47:05.050330 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.050301    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:47:05.053626 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.053591    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-pfnm9\""
Sep 16 19:47:05.053795 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.053594    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 16 19:47:05.053795 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.053698    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:47:05.053795 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.053761    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 16 19:47:05.053961 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.053827    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 16 19:47:05.054560 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.054392    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-cc52p\""
Sep 16 19:47:05.054560 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.054560    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 16 19:47:05.070598 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.070564    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"]
Sep 16 19:47:05.120314 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.120290    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa846b31-ab3e-45e2-8490-e4541e0592da-home\") pod \"custom-route-timeout-pd-test-kserve-77558746c5-pbsrm\" (UID: \"aa846b31-ab3e-45e2-8490-e4541e0592da\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:47:05.120418 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.120318    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa846b31-ab3e-45e2-8490-e4541e0592da-dshm\") pod \"custom-route-timeout-pd-test-kserve-77558746c5-pbsrm\" (UID: \"aa846b31-ab3e-45e2-8490-e4541e0592da\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:47:05.120418 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.120344    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa846b31-ab3e-45e2-8490-e4541e0592da-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-77558746c5-pbsrm\" (UID: \"aa846b31-ab3e-45e2-8490-e4541e0592da\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:47:05.120418 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.120361    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5c2k\" (UniqueName: \"kubernetes.io/projected/aa846b31-ab3e-45e2-8490-e4541e0592da-kube-api-access-w5c2k\") pod \"custom-route-timeout-pd-test-kserve-77558746c5-pbsrm\" (UID: \"aa846b31-ab3e-45e2-8490-e4541e0592da\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:47:05.120527 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.120439    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa846b31-ab3e-45e2-8490-e4541e0592da-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-77558746c5-pbsrm\" (UID: \"aa846b31-ab3e-45e2-8490-e4541e0592da\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:47:05.120527 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.120464    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa846b31-ab3e-45e2-8490-e4541e0592da-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-77558746c5-pbsrm\" (UID: \"aa846b31-ab3e-45e2-8490-e4541e0592da\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:47:05.120527 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.120512    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa846b31-ab3e-45e2-8490-e4541e0592da-model-cache\") pod \"custom-route-timeout-pd-test-kserve-77558746c5-pbsrm\" (UID: \"aa846b31-ab3e-45e2-8490-e4541e0592da\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:47:05.120626 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.120541    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa846b31-ab3e-45e2-8490-e4541e0592da-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-77558746c5-pbsrm\" (UID: \"aa846b31-ab3e-45e2-8490-e4541e0592da\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:47:05.221303 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.221269    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa846b31-ab3e-45e2-8490-e4541e0592da-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-77558746c5-pbsrm\" (UID: \"aa846b31-ab3e-45e2-8490-e4541e0592da\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:47:05.221442 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.221305    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa846b31-ab3e-45e2-8490-e4541e0592da-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-77558746c5-pbsrm\" (UID: \"aa846b31-ab3e-45e2-8490-e4541e0592da\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:47:05.221442 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.221341    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa846b31-ab3e-45e2-8490-e4541e0592da-model-cache\") pod \"custom-route-timeout-pd-test-kserve-77558746c5-pbsrm\" (UID: \"aa846b31-ab3e-45e2-8490-e4541e0592da\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:47:05.221442 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.221370    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa846b31-ab3e-45e2-8490-e4541e0592da-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-77558746c5-pbsrm\" (UID: \"aa846b31-ab3e-45e2-8490-e4541e0592da\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:47:05.221442 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.221395    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa846b31-ab3e-45e2-8490-e4541e0592da-home\") pod \"custom-route-timeout-pd-test-kserve-77558746c5-pbsrm\" (UID: \"aa846b31-ab3e-45e2-8490-e4541e0592da\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:47:05.221442 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.221410    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa846b31-ab3e-45e2-8490-e4541e0592da-dshm\") pod \"custom-route-timeout-pd-test-kserve-77558746c5-pbsrm\" (UID: \"aa846b31-ab3e-45e2-8490-e4541e0592da\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:47:05.221693 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.221490    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa846b31-ab3e-45e2-8490-e4541e0592da-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-77558746c5-pbsrm\" (UID: \"aa846b31-ab3e-45e2-8490-e4541e0592da\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:47:05.221693 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.221532    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5c2k\" (UniqueName: \"kubernetes.io/projected/aa846b31-ab3e-45e2-8490-e4541e0592da-kube-api-access-w5c2k\") pod \"custom-route-timeout-pd-test-kserve-77558746c5-pbsrm\" (UID: \"aa846b31-ab3e-45e2-8490-e4541e0592da\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:47:05.221819 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.221797    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa846b31-ab3e-45e2-8490-e4541e0592da-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-77558746c5-pbsrm\" (UID: \"aa846b31-ab3e-45e2-8490-e4541e0592da\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:47:05.221943 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.221924    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa846b31-ab3e-45e2-8490-e4541e0592da-model-cache\") pod \"custom-route-timeout-pd-test-kserve-77558746c5-pbsrm\" (UID: \"aa846b31-ab3e-45e2-8490-e4541e0592da\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:47:05.222010 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.221952    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa846b31-ab3e-45e2-8490-e4541e0592da-home\") pod \"custom-route-timeout-pd-test-kserve-77558746c5-pbsrm\" (UID: \"aa846b31-ab3e-45e2-8490-e4541e0592da\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:47:05.222010 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.221982    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa846b31-ab3e-45e2-8490-e4541e0592da-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-77558746c5-pbsrm\" (UID: \"aa846b31-ab3e-45e2-8490-e4541e0592da\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:47:05.222241 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.222221    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa846b31-ab3e-45e2-8490-e4541e0592da-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-77558746c5-pbsrm\" (UID: \"aa846b31-ab3e-45e2-8490-e4541e0592da\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:47:05.223832 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.223807    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa846b31-ab3e-45e2-8490-e4541e0592da-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-77558746c5-pbsrm\" (UID: \"aa846b31-ab3e-45e2-8490-e4541e0592da\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:47:05.223832 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.223828    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa846b31-ab3e-45e2-8490-e4541e0592da-dshm\") pod \"custom-route-timeout-pd-test-kserve-77558746c5-pbsrm\" (UID: \"aa846b31-ab3e-45e2-8490-e4541e0592da\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:47:05.231071 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.231031    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5c2k\" (UniqueName: \"kubernetes.io/projected/aa846b31-ab3e-45e2-8490-e4541e0592da-kube-api-access-w5c2k\") pod \"custom-route-timeout-pd-test-kserve-77558746c5-pbsrm\" (UID: \"aa846b31-ab3e-45e2-8490-e4541e0592da\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:47:05.329199 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.329137    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s"]
Sep 16 19:47:05.333291 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.333276    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s"
Sep 16 19:47:05.335985 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.335955    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-vkwk6\""
Sep 16 19:47:05.342838 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.342813    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s"]
Sep 16 19:47:05.361753 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.361729    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:47:05.423631 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.423558    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/262c8958-c1ed-44c5-a455-8ec970d639c6-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s\" (UID: \"262c8958-c1ed-44c5-a455-8ec970d639c6\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s"
Sep 16 19:47:05.423769 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.423675    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz6mm\" (UniqueName: \"kubernetes.io/projected/262c8958-c1ed-44c5-a455-8ec970d639c6-kube-api-access-rz6mm\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s\" (UID: \"262c8958-c1ed-44c5-a455-8ec970d639c6\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s"
Sep 16 19:47:05.525001 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.524967    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/262c8958-c1ed-44c5-a455-8ec970d639c6-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s\" (UID: \"262c8958-c1ed-44c5-a455-8ec970d639c6\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s"
Sep 16 19:47:05.525166 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.525061    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz6mm\" (UniqueName: \"kubernetes.io/projected/262c8958-c1ed-44c5-a455-8ec970d639c6-kube-api-access-rz6mm\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s\" (UID: \"262c8958-c1ed-44c5-a455-8ec970d639c6\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s"
Sep 16 19:47:05.527513 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.527492    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/262c8958-c1ed-44c5-a455-8ec970d639c6-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s\" (UID: \"262c8958-c1ed-44c5-a455-8ec970d639c6\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s"
Sep 16 19:47:05.534878 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.534856    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz6mm\" (UniqueName: \"kubernetes.io/projected/262c8958-c1ed-44c5-a455-8ec970d639c6-kube-api-access-rz6mm\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s\" (UID: \"262c8958-c1ed-44c5-a455-8ec970d639c6\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s"
Sep 16 19:47:05.640466 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.640444    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"]
Sep 16 19:47:05.643277 ip-10-0-142-80 kubenswrapper[2578]: W0916 19:47:05.643252    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa846b31_ab3e_45e2_8490_e4541e0592da.slice/crio-dcf58df1996cd1bdbe74f6afdbfd5c38adc296ef70da709fdf26f1241263e9e0 WatchSource:0}: Error finding container dcf58df1996cd1bdbe74f6afdbfd5c38adc296ef70da709fdf26f1241263e9e0: Status 404 returned error can't find the container with id dcf58df1996cd1bdbe74f6afdbfd5c38adc296ef70da709fdf26f1241263e9e0
Sep 16 19:47:05.643396 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.643293    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s"
Sep 16 19:47:05.917925 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:05.917896    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s"]
Sep 16 19:47:05.919439 ip-10-0-142-80 kubenswrapper[2578]: W0916 19:47:05.919397    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod262c8958_c1ed_44c5_a455_8ec970d639c6.slice/crio-884e30620e1878986840c47b7e8049eb8e98c36a40b9ab66fdea8fdd4346245a WatchSource:0}: Error finding container 884e30620e1878986840c47b7e8049eb8e98c36a40b9ab66fdea8fdd4346245a: Status 404 returned error can't find the container with id 884e30620e1878986840c47b7e8049eb8e98c36a40b9ab66fdea8fdd4346245a
Sep 16 19:47:06.536437 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:06.536391    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s" event={"ID":"262c8958-c1ed-44c5-a455-8ec970d639c6","Type":"ContainerStarted","Data":"8a6f6b479f5c024dbdbe3e2e1710084bd3e94e41b62d72d3474283d002e24a38"}
Sep 16 19:47:06.536881 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:06.536461    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s" event={"ID":"262c8958-c1ed-44c5-a455-8ec970d639c6","Type":"ContainerStarted","Data":"884e30620e1878986840c47b7e8049eb8e98c36a40b9ab66fdea8fdd4346245a"}
Sep 16 19:47:06.536881 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:06.536492    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s"
Sep 16 19:47:06.537984 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:06.537955    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm" event={"ID":"aa846b31-ab3e-45e2-8490-e4541e0592da","Type":"ContainerStarted","Data":"849c3824ed29ddb0d1cf8dd4744de35b5706a5f6dd35c9c3f80ffc61d39837ce"}
Sep 16 19:47:06.538130 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:06.538011    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm" event={"ID":"aa846b31-ab3e-45e2-8490-e4541e0592da","Type":"ContainerStarted","Data":"dcf58df1996cd1bdbe74f6afdbfd5c38adc296ef70da709fdf26f1241263e9e0"}
Sep 16 19:47:06.538130 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:06.538053    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:47:06.556251 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:06.556213    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s" podStartSLOduration=1.556199433 podStartE2EDuration="1.556199433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:47:06.553943449 +0000 UTC m=+3994.788876894" watchObservedRunningTime="2026-09-16 19:47:06.556199433 +0000 UTC m=+3994.791132875"
Sep 16 19:47:07.547848 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:07.547806    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm" event={"ID":"aa846b31-ab3e-45e2-8490-e4541e0592da","Type":"ContainerStarted","Data":"7db073d8ef9460f861c2351a233d023a0f80c272e8a0cb68160b7124ac12b98a"}
Sep 16 19:47:09.560857 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:09.560823    2578 generic.go:356] "Generic (PLEG): container finished" podID="aa846b31-ab3e-45e2-8490-e4541e0592da" containerID="7db073d8ef9460f861c2351a233d023a0f80c272e8a0cb68160b7124ac12b98a" exitCode=0
Sep 16 19:47:09.561257 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:09.560897    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm" event={"ID":"aa846b31-ab3e-45e2-8490-e4541e0592da","Type":"ContainerDied","Data":"7db073d8ef9460f861c2351a233d023a0f80c272e8a0cb68160b7124ac12b98a"}
Sep 16 19:47:10.568096 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:10.568055    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm" event={"ID":"aa846b31-ab3e-45e2-8490-e4541e0592da","Type":"ContainerStarted","Data":"26d181ccdc6e0305a7a731e5f18a30cc2724674c98e54455f2e31b847f5f8226"}
Sep 16 19:47:10.596025 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:10.595976    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm" podStartSLOduration=3.549839349 podStartE2EDuration="5.595959787s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.046120438s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:47:10.595454576 +0000 UTC m=+3998.830388016" watchObservedRunningTime="2026-09-16 19:47:10.595959787 +0000 UTC m=+3998.830893242"
Sep 16 19:47:15.362728 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:15.362684    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:47:15.363246 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:15.362757    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:47:15.364251 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:15.364211    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm" podUID="aa846b31-ab3e-45e2-8490-e4541e0592da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8001/health\": dial tcp 10.133.0.71:8001: connect: connection refused"
Sep 16 19:47:15.375140 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:15.375119    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:47:25.362656 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:25.362602    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm" podUID="aa846b31-ab3e-45e2-8490-e4541e0592da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8001/health\": dial tcp 10.133.0.71:8001: connect: connection refused"
Sep 16 19:47:35.362773 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:35.362727    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm" podUID="aa846b31-ab3e-45e2-8490-e4541e0592da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8001/health\": dial tcp 10.133.0.71:8001: connect: connection refused"
Sep 16 19:47:37.549283 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:37.549251    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s"
Sep 16 19:47:45.362679 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:45.362619    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm" podUID="aa846b31-ab3e-45e2-8490-e4541e0592da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8001/health\": dial tcp 10.133.0.71:8001: connect: connection refused"
Sep 16 19:47:55.362212 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:47:55.362116    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm" podUID="aa846b31-ab3e-45e2-8490-e4541e0592da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8001/health\": dial tcp 10.133.0.71:8001: connect: connection refused"
Sep 16 19:48:05.362777 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:48:05.362732    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm" podUID="aa846b31-ab3e-45e2-8490-e4541e0592da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8001/health\": dial tcp 10.133.0.71:8001: connect: connection refused"
Sep 16 19:48:15.362870 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:48:15.362822    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm" podUID="aa846b31-ab3e-45e2-8490-e4541e0592da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8001/health\": dial tcp 10.133.0.71:8001: connect: connection refused"
Sep 16 19:48:25.362889 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:48:25.362824    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm" podUID="aa846b31-ab3e-45e2-8490-e4541e0592da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8001/health\": dial tcp 10.133.0.71:8001: connect: connection refused"
Sep 16 19:48:35.363188 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:48:35.363143    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm" podUID="aa846b31-ab3e-45e2-8490-e4541e0592da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8001/health\": dial tcp 10.133.0.71:8001: connect: connection refused"
Sep 16 19:48:45.362530 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:48:45.362485    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm" podUID="aa846b31-ab3e-45e2-8490-e4541e0592da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8001/health\": dial tcp 10.133.0.71:8001: connect: connection refused"
Sep 16 19:48:55.362140 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:48:55.362081    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm" podUID="aa846b31-ab3e-45e2-8490-e4541e0592da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8001/health\": dial tcp 10.133.0.71:8001: connect: connection refused"
Sep 16 19:49:05.371776 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:05.371738    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:49:05.383614 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:05.383588    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:49:15.441280 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:15.441241    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s"]
Sep 16 19:49:15.458985 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:15.458094    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"]
Sep 16 19:49:17.549695 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:17.549654    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s" podUID="262c8958-c1ed-44c5-a455-8ec970d639c6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:49:27.550283 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:27.550182    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s" podUID="262c8958-c1ed-44c5-a455-8ec970d639c6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:49:30.462244 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:30.462194    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s" podUID="262c8958-c1ed-44c5-a455-8ec970d639c6" containerName="main" containerID="cri-o://8a6f6b479f5c024dbdbe3e2e1710084bd3e94e41b62d72d3474283d002e24a38" gracePeriod=45
Sep 16 19:49:30.480294 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:30.480256    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm" podUID="aa846b31-ab3e-45e2-8490-e4541e0592da" containerName="main" containerID="cri-o://26d181ccdc6e0305a7a731e5f18a30cc2724674c98e54455f2e31b847f5f8226" gracePeriod=45
Sep 16 19:49:31.010642 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.010610    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm" podUID="aa846b31-ab3e-45e2-8490-e4541e0592da" containerName="llm-d-routing-sidecar" containerID="cri-o://849c3824ed29ddb0d1cf8dd4744de35b5706a5f6dd35c9c3f80ffc61d39837ce" gracePeriod=45
Sep 16 19:49:31.254435 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.254412    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:49:31.298848 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.298818    2578 generic.go:356] "Generic (PLEG): container finished" podID="aa846b31-ab3e-45e2-8490-e4541e0592da" containerID="26d181ccdc6e0305a7a731e5f18a30cc2724674c98e54455f2e31b847f5f8226" exitCode=0
Sep 16 19:49:31.298848 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.298846    2578 generic.go:356] "Generic (PLEG): container finished" podID="aa846b31-ab3e-45e2-8490-e4541e0592da" containerID="849c3824ed29ddb0d1cf8dd4744de35b5706a5f6dd35c9c3f80ffc61d39837ce" exitCode=0
Sep 16 19:49:31.299012 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.298876    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"
Sep 16 19:49:31.299012 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.298883    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm" event={"ID":"aa846b31-ab3e-45e2-8490-e4541e0592da","Type":"ContainerDied","Data":"26d181ccdc6e0305a7a731e5f18a30cc2724674c98e54455f2e31b847f5f8226"}
Sep 16 19:49:31.299012 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.298944    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm" event={"ID":"aa846b31-ab3e-45e2-8490-e4541e0592da","Type":"ContainerDied","Data":"849c3824ed29ddb0d1cf8dd4744de35b5706a5f6dd35c9c3f80ffc61d39837ce"}
Sep 16 19:49:31.299012 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.298962    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm" event={"ID":"aa846b31-ab3e-45e2-8490-e4541e0592da","Type":"ContainerDied","Data":"dcf58df1996cd1bdbe74f6afdbfd5c38adc296ef70da709fdf26f1241263e9e0"}
Sep 16 19:49:31.299012 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.298984    2578 scope.go:117] "RemoveContainer" containerID="26d181ccdc6e0305a7a731e5f18a30cc2724674c98e54455f2e31b847f5f8226"
Sep 16 19:49:31.307611 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.307595    2578 scope.go:117] "RemoveContainer" containerID="7db073d8ef9460f861c2351a233d023a0f80c272e8a0cb68160b7124ac12b98a"
Sep 16 19:49:31.315294 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.315277    2578 scope.go:117] "RemoveContainer" containerID="849c3824ed29ddb0d1cf8dd4744de35b5706a5f6dd35c9c3f80ffc61d39837ce"
Sep 16 19:49:31.322278 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.322260    2578 scope.go:117] "RemoveContainer" containerID="26d181ccdc6e0305a7a731e5f18a30cc2724674c98e54455f2e31b847f5f8226"
Sep 16 19:49:31.322523 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:49:31.322505    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26d181ccdc6e0305a7a731e5f18a30cc2724674c98e54455f2e31b847f5f8226\": container with ID starting with 26d181ccdc6e0305a7a731e5f18a30cc2724674c98e54455f2e31b847f5f8226 not found: ID does not exist" containerID="26d181ccdc6e0305a7a731e5f18a30cc2724674c98e54455f2e31b847f5f8226"
Sep 16 19:49:31.322571 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.322535    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26d181ccdc6e0305a7a731e5f18a30cc2724674c98e54455f2e31b847f5f8226"} err="failed to get container status \"26d181ccdc6e0305a7a731e5f18a30cc2724674c98e54455f2e31b847f5f8226\": rpc error: code = NotFound desc = could not find container \"26d181ccdc6e0305a7a731e5f18a30cc2724674c98e54455f2e31b847f5f8226\": container with ID starting with 26d181ccdc6e0305a7a731e5f18a30cc2724674c98e54455f2e31b847f5f8226 not found: ID does not exist"
Sep 16 19:49:31.322571 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.322555    2578 scope.go:117] "RemoveContainer" containerID="7db073d8ef9460f861c2351a233d023a0f80c272e8a0cb68160b7124ac12b98a"
Sep 16 19:49:31.322803 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:49:31.322784    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7db073d8ef9460f861c2351a233d023a0f80c272e8a0cb68160b7124ac12b98a\": container with ID starting with 7db073d8ef9460f861c2351a233d023a0f80c272e8a0cb68160b7124ac12b98a not found: ID does not exist" containerID="7db073d8ef9460f861c2351a233d023a0f80c272e8a0cb68160b7124ac12b98a"
Sep 16 19:49:31.322871 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.322821    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7db073d8ef9460f861c2351a233d023a0f80c272e8a0cb68160b7124ac12b98a"} err="failed to get container status \"7db073d8ef9460f861c2351a233d023a0f80c272e8a0cb68160b7124ac12b98a\": rpc error: code = NotFound desc = could not find container \"7db073d8ef9460f861c2351a233d023a0f80c272e8a0cb68160b7124ac12b98a\": container with ID starting with 7db073d8ef9460f861c2351a233d023a0f80c272e8a0cb68160b7124ac12b98a not found: ID does not exist"
Sep 16 19:49:31.322871 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.322844    2578 scope.go:117] "RemoveContainer" containerID="849c3824ed29ddb0d1cf8dd4744de35b5706a5f6dd35c9c3f80ffc61d39837ce"
Sep 16 19:49:31.323121 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:49:31.323102    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"849c3824ed29ddb0d1cf8dd4744de35b5706a5f6dd35c9c3f80ffc61d39837ce\": container with ID starting with 849c3824ed29ddb0d1cf8dd4744de35b5706a5f6dd35c9c3f80ffc61d39837ce not found: ID does not exist" containerID="849c3824ed29ddb0d1cf8dd4744de35b5706a5f6dd35c9c3f80ffc61d39837ce"
Sep 16 19:49:31.323169 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.323126    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"849c3824ed29ddb0d1cf8dd4744de35b5706a5f6dd35c9c3f80ffc61d39837ce"} err="failed to get container status \"849c3824ed29ddb0d1cf8dd4744de35b5706a5f6dd35c9c3f80ffc61d39837ce\": rpc error: code = NotFound desc = could not find container \"849c3824ed29ddb0d1cf8dd4744de35b5706a5f6dd35c9c3f80ffc61d39837ce\": container with ID starting with 849c3824ed29ddb0d1cf8dd4744de35b5706a5f6dd35c9c3f80ffc61d39837ce not found: ID does not exist"
Sep 16 19:49:31.323169 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.323139    2578 scope.go:117] "RemoveContainer" containerID="26d181ccdc6e0305a7a731e5f18a30cc2724674c98e54455f2e31b847f5f8226"
Sep 16 19:49:31.323361 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.323342    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26d181ccdc6e0305a7a731e5f18a30cc2724674c98e54455f2e31b847f5f8226"} err="failed to get container status \"26d181ccdc6e0305a7a731e5f18a30cc2724674c98e54455f2e31b847f5f8226\": rpc error: code = NotFound desc = could not find container \"26d181ccdc6e0305a7a731e5f18a30cc2724674c98e54455f2e31b847f5f8226\": container with ID starting with 26d181ccdc6e0305a7a731e5f18a30cc2724674c98e54455f2e31b847f5f8226 not found: ID does not exist"
Sep 16 19:49:31.323437 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.323363    2578 scope.go:117] "RemoveContainer" containerID="7db073d8ef9460f861c2351a233d023a0f80c272e8a0cb68160b7124ac12b98a"
Sep 16 19:49:31.323610 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.323594    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7db073d8ef9460f861c2351a233d023a0f80c272e8a0cb68160b7124ac12b98a"} err="failed to get container status \"7db073d8ef9460f861c2351a233d023a0f80c272e8a0cb68160b7124ac12b98a\": rpc error: code = NotFound desc = could not find container \"7db073d8ef9460f861c2351a233d023a0f80c272e8a0cb68160b7124ac12b98a\": container with ID starting with 7db073d8ef9460f861c2351a233d023a0f80c272e8a0cb68160b7124ac12b98a not found: ID does not exist"
Sep 16 19:49:31.323678 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.323613    2578 scope.go:117] "RemoveContainer" containerID="849c3824ed29ddb0d1cf8dd4744de35b5706a5f6dd35c9c3f80ffc61d39837ce"
Sep 16 19:49:31.323850 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.323825    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"849c3824ed29ddb0d1cf8dd4744de35b5706a5f6dd35c9c3f80ffc61d39837ce"} err="failed to get container status \"849c3824ed29ddb0d1cf8dd4744de35b5706a5f6dd35c9c3f80ffc61d39837ce\": rpc error: code = NotFound desc = could not find container \"849c3824ed29ddb0d1cf8dd4744de35b5706a5f6dd35c9c3f80ffc61d39837ce\": container with ID starting with 849c3824ed29ddb0d1cf8dd4744de35b5706a5f6dd35c9c3f80ffc61d39837ce not found: ID does not exist"
Sep 16 19:49:31.397394 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.397374    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa846b31-ab3e-45e2-8490-e4541e0592da-dshm\") pod \"aa846b31-ab3e-45e2-8490-e4541e0592da\" (UID: \"aa846b31-ab3e-45e2-8490-e4541e0592da\") "
Sep 16 19:49:31.397502 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.397402    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa846b31-ab3e-45e2-8490-e4541e0592da-cabundle-cert\") pod \"aa846b31-ab3e-45e2-8490-e4541e0592da\" (UID: \"aa846b31-ab3e-45e2-8490-e4541e0592da\") "
Sep 16 19:49:31.397502 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.397424    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5c2k\" (UniqueName: \"kubernetes.io/projected/aa846b31-ab3e-45e2-8490-e4541e0592da-kube-api-access-w5c2k\") pod \"aa846b31-ab3e-45e2-8490-e4541e0592da\" (UID: \"aa846b31-ab3e-45e2-8490-e4541e0592da\") "
Sep 16 19:49:31.397502 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.397486    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa846b31-ab3e-45e2-8490-e4541e0592da-tmp-dir\") pod \"aa846b31-ab3e-45e2-8490-e4541e0592da\" (UID: \"aa846b31-ab3e-45e2-8490-e4541e0592da\") "
Sep 16 19:49:31.397658 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.397504    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa846b31-ab3e-45e2-8490-e4541e0592da-home\") pod \"aa846b31-ab3e-45e2-8490-e4541e0592da\" (UID: \"aa846b31-ab3e-45e2-8490-e4541e0592da\") "
Sep 16 19:49:31.397658 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.397553    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa846b31-ab3e-45e2-8490-e4541e0592da-kserve-provision-location\") pod \"aa846b31-ab3e-45e2-8490-e4541e0592da\" (UID: \"aa846b31-ab3e-45e2-8490-e4541e0592da\") "
Sep 16 19:49:31.397658 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.397576    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa846b31-ab3e-45e2-8490-e4541e0592da-model-cache\") pod \"aa846b31-ab3e-45e2-8490-e4541e0592da\" (UID: \"aa846b31-ab3e-45e2-8490-e4541e0592da\") "
Sep 16 19:49:31.397658 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.397608    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa846b31-ab3e-45e2-8490-e4541e0592da-tls-certs\") pod \"aa846b31-ab3e-45e2-8490-e4541e0592da\" (UID: \"aa846b31-ab3e-45e2-8490-e4541e0592da\") "
Sep 16 19:49:31.397879 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.397854    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa846b31-ab3e-45e2-8490-e4541e0592da-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "aa846b31-ab3e-45e2-8490-e4541e0592da" (UID: "aa846b31-ab3e-45e2-8490-e4541e0592da"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:49:31.398023 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.397960    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa846b31-ab3e-45e2-8490-e4541e0592da-model-cache" (OuterVolumeSpecName: "model-cache") pod "aa846b31-ab3e-45e2-8490-e4541e0592da" (UID: "aa846b31-ab3e-45e2-8490-e4541e0592da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:31.398276 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.398247    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa846b31-ab3e-45e2-8490-e4541e0592da-home" (OuterVolumeSpecName: "home") pod "aa846b31-ab3e-45e2-8490-e4541e0592da" (UID: "aa846b31-ab3e-45e2-8490-e4541e0592da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:31.399829 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.399800    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa846b31-ab3e-45e2-8490-e4541e0592da-kube-api-access-w5c2k" (OuterVolumeSpecName: "kube-api-access-w5c2k") pod "aa846b31-ab3e-45e2-8490-e4541e0592da" (UID: "aa846b31-ab3e-45e2-8490-e4541e0592da"). InnerVolumeSpecName "kube-api-access-w5c2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:49:31.399931 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.399825    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa846b31-ab3e-45e2-8490-e4541e0592da-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa846b31-ab3e-45e2-8490-e4541e0592da" (UID: "aa846b31-ab3e-45e2-8490-e4541e0592da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:49:31.399931 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.399878    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa846b31-ab3e-45e2-8490-e4541e0592da-dshm" (OuterVolumeSpecName: "dshm") pod "aa846b31-ab3e-45e2-8490-e4541e0592da" (UID: "aa846b31-ab3e-45e2-8490-e4541e0592da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:31.409552 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.409522    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa846b31-ab3e-45e2-8490-e4541e0592da-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aa846b31-ab3e-45e2-8490-e4541e0592da" (UID: "aa846b31-ab3e-45e2-8490-e4541e0592da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:31.421439 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.421417    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa846b31-ab3e-45e2-8490-e4541e0592da-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aa846b31-ab3e-45e2-8490-e4541e0592da" (UID: "aa846b31-ab3e-45e2-8490-e4541e0592da"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:49:31.499087 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.499004    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa846b31-ab3e-45e2-8490-e4541e0592da-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:49:31.499087 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.499028    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa846b31-ab3e-45e2-8490-e4541e0592da-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:49:31.499087 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.499061    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa846b31-ab3e-45e2-8490-e4541e0592da-kserve-provision-location\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:49:31.499087 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.499073    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa846b31-ab3e-45e2-8490-e4541e0592da-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:49:31.499087 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.499086    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa846b31-ab3e-45e2-8490-e4541e0592da-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:49:31.499504 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.499097    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa846b31-ab3e-45e2-8490-e4541e0592da-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:49:31.499504 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.499110    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa846b31-ab3e-45e2-8490-e4541e0592da-cabundle-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:49:31.499504 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.499121    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5c2k\" (UniqueName: \"kubernetes.io/projected/aa846b31-ab3e-45e2-8490-e4541e0592da-kube-api-access-w5c2k\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:49:31.632799 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.632778    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"]
Sep 16 19:49:31.647282 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:31.647261    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-77558746c5-pbsrm"]
Sep 16 19:49:32.216623 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:32.216591    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa846b31-ab3e-45e2-8490-e4541e0592da" path="/var/lib/kubelet/pods/aa846b31-ab3e-45e2-8490-e4541e0592da/volumes"
Sep 16 19:49:37.550187 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:37.550145    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s" podUID="262c8958-c1ed-44c5-a455-8ec970d639c6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:49:37.550620 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:37.550247    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s"
Sep 16 19:49:47.549414 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:47.549368    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s" podUID="262c8958-c1ed-44c5-a455-8ec970d639c6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:49:57.550097 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:49:57.550059    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s" podUID="262c8958-c1ed-44c5-a455-8ec970d639c6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:50:00.702000 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:00.701976    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s"
Sep 16 19:50:00.711771 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:00.711748    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz6mm\" (UniqueName: \"kubernetes.io/projected/262c8958-c1ed-44c5-a455-8ec970d639c6-kube-api-access-rz6mm\") pod \"262c8958-c1ed-44c5-a455-8ec970d639c6\" (UID: \"262c8958-c1ed-44c5-a455-8ec970d639c6\") "
Sep 16 19:50:00.711868 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:00.711785    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/262c8958-c1ed-44c5-a455-8ec970d639c6-tls-certs\") pod \"262c8958-c1ed-44c5-a455-8ec970d639c6\" (UID: \"262c8958-c1ed-44c5-a455-8ec970d639c6\") "
Sep 16 19:50:00.713731 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:00.713701    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/262c8958-c1ed-44c5-a455-8ec970d639c6-kube-api-access-rz6mm" (OuterVolumeSpecName: "kube-api-access-rz6mm") pod "262c8958-c1ed-44c5-a455-8ec970d639c6" (UID: "262c8958-c1ed-44c5-a455-8ec970d639c6"). InnerVolumeSpecName "kube-api-access-rz6mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:50:00.713731 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:00.713710    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/262c8958-c1ed-44c5-a455-8ec970d639c6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "262c8958-c1ed-44c5-a455-8ec970d639c6" (UID: "262c8958-c1ed-44c5-a455-8ec970d639c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:50:00.812626 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:00.812542    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/262c8958-c1ed-44c5-a455-8ec970d639c6-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:50:00.812626 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:00.812574    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz6mm\" (UniqueName: \"kubernetes.io/projected/262c8958-c1ed-44c5-a455-8ec970d639c6-kube-api-access-rz6mm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:50:01.455299 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:01.455263    2578 generic.go:356] "Generic (PLEG): container finished" podID="262c8958-c1ed-44c5-a455-8ec970d639c6" containerID="8a6f6b479f5c024dbdbe3e2e1710084bd3e94e41b62d72d3474283d002e24a38" exitCode=0
Sep 16 19:50:01.455501 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:01.455306    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s"
Sep 16 19:50:01.455501 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:01.455347    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s" event={"ID":"262c8958-c1ed-44c5-a455-8ec970d639c6","Type":"ContainerDied","Data":"8a6f6b479f5c024dbdbe3e2e1710084bd3e94e41b62d72d3474283d002e24a38"}
Sep 16 19:50:01.455501 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:01.455414    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s" event={"ID":"262c8958-c1ed-44c5-a455-8ec970d639c6","Type":"ContainerDied","Data":"884e30620e1878986840c47b7e8049eb8e98c36a40b9ab66fdea8fdd4346245a"}
Sep 16 19:50:01.455501 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:01.455437    2578 scope.go:117] "RemoveContainer" containerID="8a6f6b479f5c024dbdbe3e2e1710084bd3e94e41b62d72d3474283d002e24a38"
Sep 16 19:50:01.464740 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:01.464716    2578 scope.go:117] "RemoveContainer" containerID="8a6f6b479f5c024dbdbe3e2e1710084bd3e94e41b62d72d3474283d002e24a38"
Sep 16 19:50:01.465004 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:50:01.464983    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a6f6b479f5c024dbdbe3e2e1710084bd3e94e41b62d72d3474283d002e24a38\": container with ID starting with 8a6f6b479f5c024dbdbe3e2e1710084bd3e94e41b62d72d3474283d002e24a38 not found: ID does not exist" containerID="8a6f6b479f5c024dbdbe3e2e1710084bd3e94e41b62d72d3474283d002e24a38"
Sep 16 19:50:01.465067 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:01.465014    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a6f6b479f5c024dbdbe3e2e1710084bd3e94e41b62d72d3474283d002e24a38"} err="failed to get container status \"8a6f6b479f5c024dbdbe3e2e1710084bd3e94e41b62d72d3474283d002e24a38\": rpc error: code = NotFound desc = could not find container \"8a6f6b479f5c024dbdbe3e2e1710084bd3e94e41b62d72d3474283d002e24a38\": container with ID starting with 8a6f6b479f5c024dbdbe3e2e1710084bd3e94e41b62d72d3474283d002e24a38 not found: ID does not exist"
Sep 16 19:50:01.493603 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:01.493573    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s"]
Sep 16 19:50:01.508244 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:01.508217    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48mbn6s"]
Sep 16 19:50:02.217917 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:02.217881    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="262c8958-c1ed-44c5-a455-8ec970d639c6" path="/var/lib/kubelet/pods/262c8958-c1ed-44c5-a455-8ec970d639c6/volumes"
Sep 16 19:50:05.593980 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.593947    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"]
Sep 16 19:50:05.594408 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.594288    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa846b31-ab3e-45e2-8490-e4541e0592da" containerName="llm-d-routing-sidecar"
Sep 16 19:50:05.594408 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.594301    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa846b31-ab3e-45e2-8490-e4541e0592da" containerName="llm-d-routing-sidecar"
Sep 16 19:50:05.594408 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.594311    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="262c8958-c1ed-44c5-a455-8ec970d639c6" containerName="main"
Sep 16 19:50:05.594408 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.594317    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="262c8958-c1ed-44c5-a455-8ec970d639c6" containerName="main"
Sep 16 19:50:05.594408 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.594324    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa846b31-ab3e-45e2-8490-e4541e0592da" containerName="storage-initializer"
Sep 16 19:50:05.594408 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.594330    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa846b31-ab3e-45e2-8490-e4541e0592da" containerName="storage-initializer"
Sep 16 19:50:05.594408 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.594340    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa846b31-ab3e-45e2-8490-e4541e0592da" containerName="main"
Sep 16 19:50:05.594408 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.594345    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa846b31-ab3e-45e2-8490-e4541e0592da" containerName="main"
Sep 16 19:50:05.594408 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.594398    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa846b31-ab3e-45e2-8490-e4541e0592da" containerName="llm-d-routing-sidecar"
Sep 16 19:50:05.594408 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.594407    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa846b31-ab3e-45e2-8490-e4541e0592da" containerName="main"
Sep 16 19:50:05.594408 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.594415    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="262c8958-c1ed-44c5-a455-8ec970d639c6" containerName="main"
Sep 16 19:50:05.598521 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.598500    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:05.602869 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.602848    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Sep 16 19:50:05.602973 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.602906    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 16 19:50:05.603959 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.603938    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-k4hsm\""
Sep 16 19:50:05.609007 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.608987    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 16 19:50:05.644876 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.644847    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"]
Sep 16 19:50:05.649885 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.649858    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1920012-14fb-4875-87fb-7ac2da56702e-istio-token\") pod \"router-gateway-2-openshift-default-7f645c8cbd-qgdw2\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:05.649995 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.649907    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1920012-14fb-4875-87fb-7ac2da56702e-workload-socket\") pod \"router-gateway-2-openshift-default-7f645c8cbd-qgdw2\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:05.649995 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.649937    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1920012-14fb-4875-87fb-7ac2da56702e-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-7f645c8cbd-qgdw2\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:05.650147 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.650021    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1920012-14fb-4875-87fb-7ac2da56702e-workload-certs\") pod \"router-gateway-2-openshift-default-7f645c8cbd-qgdw2\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:05.650147 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.650064    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxzqb\" (UniqueName: \"kubernetes.io/projected/e1920012-14fb-4875-87fb-7ac2da56702e-kube-api-access-xxzqb\") pod \"router-gateway-2-openshift-default-7f645c8cbd-qgdw2\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:05.650147 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.650087    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1920012-14fb-4875-87fb-7ac2da56702e-istio-podinfo\") pod \"router-gateway-2-openshift-default-7f645c8cbd-qgdw2\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:05.650147 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.650113    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1920012-14fb-4875-87fb-7ac2da56702e-istio-envoy\") pod \"router-gateway-2-openshift-default-7f645c8cbd-qgdw2\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:05.650319 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.650154    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1920012-14fb-4875-87fb-7ac2da56702e-credential-socket\") pod \"router-gateway-2-openshift-default-7f645c8cbd-qgdw2\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:05.650319 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.650184    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1920012-14fb-4875-87fb-7ac2da56702e-istio-data\") pod \"router-gateway-2-openshift-default-7f645c8cbd-qgdw2\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:05.751519 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.751485    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1920012-14fb-4875-87fb-7ac2da56702e-workload-certs\") pod \"router-gateway-2-openshift-default-7f645c8cbd-qgdw2\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:05.751519 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.751521    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxzqb\" (UniqueName: \"kubernetes.io/projected/e1920012-14fb-4875-87fb-7ac2da56702e-kube-api-access-xxzqb\") pod \"router-gateway-2-openshift-default-7f645c8cbd-qgdw2\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:05.751753 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.751536    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1920012-14fb-4875-87fb-7ac2da56702e-istio-podinfo\") pod \"router-gateway-2-openshift-default-7f645c8cbd-qgdw2\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:05.751753 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.751567    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1920012-14fb-4875-87fb-7ac2da56702e-istio-envoy\") pod \"router-gateway-2-openshift-default-7f645c8cbd-qgdw2\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:05.751753 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.751691    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1920012-14fb-4875-87fb-7ac2da56702e-credential-socket\") pod \"router-gateway-2-openshift-default-7f645c8cbd-qgdw2\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:05.751753 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.751744    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1920012-14fb-4875-87fb-7ac2da56702e-istio-data\") pod \"router-gateway-2-openshift-default-7f645c8cbd-qgdw2\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:05.751976 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.751824    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1920012-14fb-4875-87fb-7ac2da56702e-istio-token\") pod \"router-gateway-2-openshift-default-7f645c8cbd-qgdw2\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:05.751976 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.751862    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1920012-14fb-4875-87fb-7ac2da56702e-workload-socket\") pod \"router-gateway-2-openshift-default-7f645c8cbd-qgdw2\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:05.751976 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.751887    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1920012-14fb-4875-87fb-7ac2da56702e-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-7f645c8cbd-qgdw2\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:05.751976 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.751903    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1920012-14fb-4875-87fb-7ac2da56702e-workload-certs\") pod \"router-gateway-2-openshift-default-7f645c8cbd-qgdw2\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:05.752225 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.752056    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1920012-14fb-4875-87fb-7ac2da56702e-credential-socket\") pod \"router-gateway-2-openshift-default-7f645c8cbd-qgdw2\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:05.752225 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.752185    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1920012-14fb-4875-87fb-7ac2da56702e-istio-data\") pod \"router-gateway-2-openshift-default-7f645c8cbd-qgdw2\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:05.752332 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.752228    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1920012-14fb-4875-87fb-7ac2da56702e-workload-socket\") pod \"router-gateway-2-openshift-default-7f645c8cbd-qgdw2\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:05.752519 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.752499    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1920012-14fb-4875-87fb-7ac2da56702e-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-7f645c8cbd-qgdw2\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:05.754064 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.754017    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1920012-14fb-4875-87fb-7ac2da56702e-istio-envoy\") pod \"router-gateway-2-openshift-default-7f645c8cbd-qgdw2\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:05.754211 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.754194    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1920012-14fb-4875-87fb-7ac2da56702e-istio-podinfo\") pod \"router-gateway-2-openshift-default-7f645c8cbd-qgdw2\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:05.765405 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.765380    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxzqb\" (UniqueName: \"kubernetes.io/projected/e1920012-14fb-4875-87fb-7ac2da56702e-kube-api-access-xxzqb\") pod \"router-gateway-2-openshift-default-7f645c8cbd-qgdw2\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:05.765759 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.765743    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1920012-14fb-4875-87fb-7ac2da56702e-istio-token\") pod \"router-gateway-2-openshift-default-7f645c8cbd-qgdw2\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:05.908266 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:05.908192    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:06.197500 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:06.197476    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"]
Sep 16 19:50:06.199057 ip-10-0-142-80 kubenswrapper[2578]: W0916 19:50:06.199006    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1920012_14fb_4875_87fb_7ac2da56702e.slice/crio-f4793db5b295511e8d2d47f7a169abeee5ebf1528d81e635890ac04818141cdf WatchSource:0}: Error finding container f4793db5b295511e8d2d47f7a169abeee5ebf1528d81e635890ac04818141cdf: Status 404 returned error can't find the container with id f4793db5b295511e8d2d47f7a169abeee5ebf1528d81e635890ac04818141cdf
Sep 16 19:50:06.201004 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:06.200973    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 19:50:06.201114 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:06.201069    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 19:50:06.201175 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:06.201119    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 19:50:06.483431 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:06.483336    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2" event={"ID":"e1920012-14fb-4875-87fb-7ac2da56702e","Type":"ContainerStarted","Data":"fb413967c7562858589268daeeb4c0be9fd6647f083fba64b5ede4cd01716e9b"}
Sep 16 19:50:06.483431 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:06.483404    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2" event={"ID":"e1920012-14fb-4875-87fb-7ac2da56702e","Type":"ContainerStarted","Data":"f4793db5b295511e8d2d47f7a169abeee5ebf1528d81e635890ac04818141cdf"}
Sep 16 19:50:06.511626 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:06.511571    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2" podStartSLOduration=1.511551223 podStartE2EDuration="1.511551223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:50:06.507460927 +0000 UTC m=+4174.742394371" watchObservedRunningTime="2026-09-16 19:50:06.511551223 +0000 UTC m=+4174.746484667"
Sep 16 19:50:06.908333 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:06.908255    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:07.913833 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:07.913802    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:08.493312 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:08.493281    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:08.494140 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:08.494116    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:14.534263 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.534221    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"]
Sep 16 19:50:14.542557 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.542525    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:50:14.546631 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.546596    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-f7z8n\""
Sep 16 19:50:14.546863 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.546840    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 16 19:50:14.547118 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.547072    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:50:14.547970 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.547290    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-xct4r\""
Sep 16 19:50:14.547970 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.547544    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 16 19:50:14.554969 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.554946    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"]
Sep 16 19:50:14.626438 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.626399    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a91be1cf-9dbc-4810-ad79-36ff90efe774-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd\" (UID: \"a91be1cf-9dbc-4810-ad79-36ff90efe774\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:50:14.626701 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.626677    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a91be1cf-9dbc-4810-ad79-36ff90efe774-model-cache\") pod \"router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd\" (UID: \"a91be1cf-9dbc-4810-ad79-36ff90efe774\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:50:14.626836 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.626820    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a91be1cf-9dbc-4810-ad79-36ff90efe774-tls-certs\") pod \"router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd\" (UID: \"a91be1cf-9dbc-4810-ad79-36ff90efe774\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:50:14.626946 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.626930    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spv4r\" (UniqueName: \"kubernetes.io/projected/a91be1cf-9dbc-4810-ad79-36ff90efe774-kube-api-access-spv4r\") pod \"router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd\" (UID: \"a91be1cf-9dbc-4810-ad79-36ff90efe774\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:50:14.627101 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.627084    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a91be1cf-9dbc-4810-ad79-36ff90efe774-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd\" (UID: \"a91be1cf-9dbc-4810-ad79-36ff90efe774\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:50:14.627232 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.627217    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a91be1cf-9dbc-4810-ad79-36ff90efe774-tmp-dir\") pod \"router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd\" (UID: \"a91be1cf-9dbc-4810-ad79-36ff90efe774\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:50:14.627360 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.627343    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a91be1cf-9dbc-4810-ad79-36ff90efe774-home\") pod \"router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd\" (UID: \"a91be1cf-9dbc-4810-ad79-36ff90efe774\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:50:14.627495 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.627480    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a91be1cf-9dbc-4810-ad79-36ff90efe774-dshm\") pod \"router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd\" (UID: \"a91be1cf-9dbc-4810-ad79-36ff90efe774\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:50:14.728685 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.728651    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a91be1cf-9dbc-4810-ad79-36ff90efe774-tmp-dir\") pod \"router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd\" (UID: \"a91be1cf-9dbc-4810-ad79-36ff90efe774\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:50:14.728685 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.728692    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a91be1cf-9dbc-4810-ad79-36ff90efe774-home\") pod \"router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd\" (UID: \"a91be1cf-9dbc-4810-ad79-36ff90efe774\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:50:14.729012 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.728729    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a91be1cf-9dbc-4810-ad79-36ff90efe774-dshm\") pod \"router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd\" (UID: \"a91be1cf-9dbc-4810-ad79-36ff90efe774\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:50:14.729012 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.728769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a91be1cf-9dbc-4810-ad79-36ff90efe774-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd\" (UID: \"a91be1cf-9dbc-4810-ad79-36ff90efe774\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:50:14.729012 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.728813    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a91be1cf-9dbc-4810-ad79-36ff90efe774-model-cache\") pod \"router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd\" (UID: \"a91be1cf-9dbc-4810-ad79-36ff90efe774\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:50:14.729012 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.728847    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a91be1cf-9dbc-4810-ad79-36ff90efe774-tls-certs\") pod \"router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd\" (UID: \"a91be1cf-9dbc-4810-ad79-36ff90efe774\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:50:14.729012 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.728870    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spv4r\" (UniqueName: \"kubernetes.io/projected/a91be1cf-9dbc-4810-ad79-36ff90efe774-kube-api-access-spv4r\") pod \"router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd\" (UID: \"a91be1cf-9dbc-4810-ad79-36ff90efe774\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:50:14.729012 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.728908    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a91be1cf-9dbc-4810-ad79-36ff90efe774-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd\" (UID: \"a91be1cf-9dbc-4810-ad79-36ff90efe774\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:50:14.729353 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.729154    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a91be1cf-9dbc-4810-ad79-36ff90efe774-tmp-dir\") pod \"router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd\" (UID: \"a91be1cf-9dbc-4810-ad79-36ff90efe774\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:50:14.729353 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.729213    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a91be1cf-9dbc-4810-ad79-36ff90efe774-home\") pod \"router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd\" (UID: \"a91be1cf-9dbc-4810-ad79-36ff90efe774\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:50:14.729803 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.729778    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a91be1cf-9dbc-4810-ad79-36ff90efe774-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd\" (UID: \"a91be1cf-9dbc-4810-ad79-36ff90efe774\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:50:14.729971 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.729941    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a91be1cf-9dbc-4810-ad79-36ff90efe774-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd\" (UID: \"a91be1cf-9dbc-4810-ad79-36ff90efe774\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:50:14.730271 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.730246    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a91be1cf-9dbc-4810-ad79-36ff90efe774-model-cache\") pod \"router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd\" (UID: \"a91be1cf-9dbc-4810-ad79-36ff90efe774\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:50:14.731948 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.731927    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a91be1cf-9dbc-4810-ad79-36ff90efe774-tls-certs\") pod \"router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd\" (UID: \"a91be1cf-9dbc-4810-ad79-36ff90efe774\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:50:14.732073 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.731998    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a91be1cf-9dbc-4810-ad79-36ff90efe774-dshm\") pod \"router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd\" (UID: \"a91be1cf-9dbc-4810-ad79-36ff90efe774\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:50:14.740007 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.739986    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spv4r\" (UniqueName: \"kubernetes.io/projected/a91be1cf-9dbc-4810-ad79-36ff90efe774-kube-api-access-spv4r\") pod \"router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd\" (UID: \"a91be1cf-9dbc-4810-ad79-36ff90efe774\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:50:14.859814 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:14.859779    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:50:15.143388 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:15.143353    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"]
Sep 16 19:50:15.145495 ip-10-0-142-80 kubenswrapper[2578]: W0916 19:50:15.145465    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda91be1cf_9dbc_4810_ad79_36ff90efe774.slice/crio-2011f25772ddb479bc34fcbcb8eb6ef9b4c9582c4c3e95df01105d6aece80635 WatchSource:0}: Error finding container 2011f25772ddb479bc34fcbcb8eb6ef9b4c9582c4c3e95df01105d6aece80635: Status 404 returned error can't find the container with id 2011f25772ddb479bc34fcbcb8eb6ef9b4c9582c4c3e95df01105d6aece80635
Sep 16 19:50:15.541406 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:15.541367    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd" event={"ID":"a91be1cf-9dbc-4810-ad79-36ff90efe774","Type":"ContainerStarted","Data":"2ed60a6e53ad672175d4480e559120b97db4f3aaf3f421ed4263784fbc90067e"}
Sep 16 19:50:15.541851 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:15.541452    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd" event={"ID":"a91be1cf-9dbc-4810-ad79-36ff90efe774","Type":"ContainerStarted","Data":"2011f25772ddb479bc34fcbcb8eb6ef9b4c9582c4c3e95df01105d6aece80635"}
Sep 16 19:50:15.541851 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:15.541477    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:50:16.550446 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:16.550409    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd" event={"ID":"a91be1cf-9dbc-4810-ad79-36ff90efe774","Type":"ContainerStarted","Data":"58b4405acaaf22cc5e2eaef72168a1637b610a11f9019b7100407ec47e785885"}
Sep 16 19:50:18.564980 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:18.564945    2578 generic.go:356] "Generic (PLEG): container finished" podID="e1920012-14fb-4875-87fb-7ac2da56702e" containerID="fb413967c7562858589268daeeb4c0be9fd6647f083fba64b5ede4cd01716e9b" exitCode=0
Sep 16 19:50:18.565508 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:18.565018    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2" event={"ID":"e1920012-14fb-4875-87fb-7ac2da56702e","Type":"ContainerDied","Data":"fb413967c7562858589268daeeb4c0be9fd6647f083fba64b5ede4cd01716e9b"}
Sep 16 19:50:18.565508 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:18.565414    2578 scope.go:117] "RemoveContainer" containerID="fb413967c7562858589268daeeb4c0be9fd6647f083fba64b5ede4cd01716e9b"
Sep 16 19:50:18.566409 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:18.566375    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 19:50:18.566550 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:18.566462    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 19:50:18.566550 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:18.566509    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 19:50:18.566951 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:18.566933    2578 generic.go:356] "Generic (PLEG): container finished" podID="a91be1cf-9dbc-4810-ad79-36ff90efe774" containerID="58b4405acaaf22cc5e2eaef72168a1637b610a11f9019b7100407ec47e785885" exitCode=0
Sep 16 19:50:18.567022 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:18.567000    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd" event={"ID":"a91be1cf-9dbc-4810-ad79-36ff90efe774","Type":"ContainerDied","Data":"58b4405acaaf22cc5e2eaef72168a1637b610a11f9019b7100407ec47e785885"}
Sep 16 19:50:19.574615 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:19.574567    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2" event={"ID":"e1920012-14fb-4875-87fb-7ac2da56702e","Type":"ContainerStarted","Data":"49fa20e8c84262c2d1c82ff4231bbe1d082aad549de40838eaf81bb8434ca5a7"}
Sep 16 19:50:19.576816 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:19.576791    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd" event={"ID":"a91be1cf-9dbc-4810-ad79-36ff90efe774","Type":"ContainerStarted","Data":"0a302967b4f5f80b547c375100ba17c208e0c10d0f8eb3259917cc6f3c9ba565"}
Sep 16 19:50:19.622809 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:19.622746    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd" podStartSLOduration=3.451489556 podStartE2EDuration="5.622725894s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.171236338s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:50:19.618420766 +0000 UTC m=+4187.853354200" watchObservedRunningTime="2026-09-16 19:50:19.622725894 +0000 UTC m=+4187.857659339"
Sep 16 19:50:19.909187 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:19.909088    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:19.910311 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:19.910278    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2" podUID="e1920012-14fb-4875-87fb-7ac2da56702e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 16 19:50:20.908723 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:20.908679    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2" podUID="e1920012-14fb-4875-87fb-7ac2da56702e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 16 19:50:21.909522 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:21.909475    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2" podUID="e1920012-14fb-4875-87fb-7ac2da56702e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 16 19:50:22.908700 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:22.908658    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2" podUID="e1920012-14fb-4875-87fb-7ac2da56702e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 16 19:50:23.494093 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:23.494051    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:23.908617 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:23.908518    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2" podUID="e1920012-14fb-4875-87fb-7ac2da56702e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 16 19:50:24.860662 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:24.860620    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:50:24.861157 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:24.860698    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:50:24.861903 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:24.861874    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd" podUID="a91be1cf-9dbc-4810-ad79-36ff90efe774" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8001/health\": dial tcp 10.133.0.74:8001: connect: connection refused"
Sep 16 19:50:24.912785 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:24.912757    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:24.913916 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:24.913895    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:50:34.860475 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:34.860429    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd" podUID="a91be1cf-9dbc-4810-ad79-36ff90efe774" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8001/health\": dial tcp 10.133.0.74:8001: connect: connection refused"
Sep 16 19:50:34.878350 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:34.878317    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:50:44.860797 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:44.860746    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd" podUID="a91be1cf-9dbc-4810-ad79-36ff90efe774" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8001/health\": dial tcp 10.133.0.74:8001: connect: connection refused"
Sep 16 19:50:54.860830 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:50:54.860723    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd" podUID="a91be1cf-9dbc-4810-ad79-36ff90efe774" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8001/health\": dial tcp 10.133.0.74:8001: connect: connection refused"
Sep 16 19:51:04.860440 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:51:04.860362    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd" podUID="a91be1cf-9dbc-4810-ad79-36ff90efe774" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8001/health\": dial tcp 10.133.0.74:8001: connect: connection refused"
Sep 16 19:51:14.860826 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:51:14.860764    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd" podUID="a91be1cf-9dbc-4810-ad79-36ff90efe774" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8001/health\": dial tcp 10.133.0.74:8001: connect: connection refused"
Sep 16 19:51:24.861053 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:51:24.860992    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd" podUID="a91be1cf-9dbc-4810-ad79-36ff90efe774" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8001/health\": dial tcp 10.133.0.74:8001: connect: connection refused"
Sep 16 19:51:34.860676 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:51:34.860630    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd" podUID="a91be1cf-9dbc-4810-ad79-36ff90efe774" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8001/health\": dial tcp 10.133.0.74:8001: connect: connection refused"
Sep 16 19:51:44.860438 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:51:44.860390    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd" podUID="a91be1cf-9dbc-4810-ad79-36ff90efe774" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8001/health\": dial tcp 10.133.0.74:8001: connect: connection refused"
Sep 16 19:51:49.134201 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:51:49.134159    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds"]
Sep 16 19:51:49.138332 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:51:49.138307    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds"
Sep 16 19:51:49.142540 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:51:49.142512    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 16 19:51:49.142696 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:51:49.142674    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-t5d5j\""
Sep 16 19:51:49.142767 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:51:49.142693    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 16 19:51:49.142767 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:51:49.142622    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-jr7t7\""
Sep 16 19:51:49.143181 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:51:49.143161    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 16 19:51:49.145270 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:51:49.145241    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90f20be3-aae7-44c1-9ed6-6aca54210719-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds\" (UID: \"90f20be3-aae7-44c1-9ed6-6aca54210719\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds"
Sep 16 19:51:49.145423 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:51:49.145400    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvd7b\" (UniqueName: \"kubernetes.io/projected/90f20be3-aae7-44c1-9ed6-6aca54210719-kube-api-access-gvd7b\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds\" (UID: \"90f20be3-aae7-44c1-9ed6-6aca54210719\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds"
Sep 16 19:51:49.146223 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:51:49.145921    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds"]
Sep 16 19:51:49.246351 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:51:49.246311    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvd7b\" (UniqueName: \"kubernetes.io/projected/90f20be3-aae7-44c1-9ed6-6aca54210719-kube-api-access-gvd7b\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds\" (UID: \"90f20be3-aae7-44c1-9ed6-6aca54210719\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds"
Sep 16 19:51:49.246551 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:51:49.246377    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90f20be3-aae7-44c1-9ed6-6aca54210719-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds\" (UID: \"90f20be3-aae7-44c1-9ed6-6aca54210719\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds"
Sep 16 19:51:49.248800 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:51:49.248773    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90f20be3-aae7-44c1-9ed6-6aca54210719-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds\" (UID: \"90f20be3-aae7-44c1-9ed6-6aca54210719\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds"
Sep 16 19:51:49.256667 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:51:49.256645    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvd7b\" (UniqueName: \"kubernetes.io/projected/90f20be3-aae7-44c1-9ed6-6aca54210719-kube-api-access-gvd7b\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds\" (UID: \"90f20be3-aae7-44c1-9ed6-6aca54210719\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds"
Sep 16 19:51:49.451083 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:51:49.451025    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds"
Sep 16 19:51:49.724200 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:51:49.724170    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds"]
Sep 16 19:51:49.727250 ip-10-0-142-80 kubenswrapper[2578]: W0916 19:51:49.727223    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f20be3_aae7_44c1_9ed6_6aca54210719.slice/crio-abe1d7453b7be30d350d574da19f214acee544046d2055554e92688f20a1938f WatchSource:0}: Error finding container abe1d7453b7be30d350d574da19f214acee544046d2055554e92688f20a1938f: Status 404 returned error can't find the container with id abe1d7453b7be30d350d574da19f214acee544046d2055554e92688f20a1938f
Sep 16 19:51:50.042096 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:51:50.041978    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds" event={"ID":"90f20be3-aae7-44c1-9ed6-6aca54210719","Type":"ContainerStarted","Data":"25aa3f45d2f5050f25fa3261449cacb4d3af59916b9af816b35442c1ecbfdee3"}
Sep 16 19:51:50.042096 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:51:50.042077    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds"
Sep 16 19:51:50.042334 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:51:50.042100    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds" event={"ID":"90f20be3-aae7-44c1-9ed6-6aca54210719","Type":"ContainerStarted","Data":"abe1d7453b7be30d350d574da19f214acee544046d2055554e92688f20a1938f"}
Sep 16 19:51:50.060313 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:51:50.060259    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds" podStartSLOduration=1.060242185 podStartE2EDuration="1.060242185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:51:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:51:50.059450688 +0000 UTC m=+4278.294384132" watchObservedRunningTime="2026-09-16 19:51:50.060242185 +0000 UTC m=+4278.295175628"
Sep 16 19:51:54.860202 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:51:54.860155    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd" podUID="a91be1cf-9dbc-4810-ad79-36ff90efe774" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8001/health\": dial tcp 10.133.0.74:8001: connect: connection refused"
Sep 16 19:52:04.861019 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:04.860974    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd" podUID="a91be1cf-9dbc-4810-ad79-36ff90efe774" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8001/health\": dial tcp 10.133.0.74:8001: connect: connection refused"
Sep 16 19:52:14.875074 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:14.875023    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:52:14.890184 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:14.890155    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:52:21.051009 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:21.050976    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds"
Sep 16 19:52:33.336902 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:33.336817    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"]
Sep 16 19:52:48.360286 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:48.360229    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd" podUID="a91be1cf-9dbc-4810-ad79-36ff90efe774" containerName="main" containerID="cri-o://0a302967b4f5f80b547c375100ba17c208e0c10d0f8eb3259917cc6f3c9ba565" gracePeriod=45
Sep 16 19:52:48.977629 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:48.977596    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd" podUID="a91be1cf-9dbc-4810-ad79-36ff90efe774" containerName="llm-d-routing-sidecar" containerID="cri-o://2ed60a6e53ad672175d4480e559120b97db4f3aaf3f421ed4263784fbc90067e" gracePeriod=45
Sep 16 19:52:49.213604 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.213579    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:52:49.332218 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.332182    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spv4r\" (UniqueName: \"kubernetes.io/projected/a91be1cf-9dbc-4810-ad79-36ff90efe774-kube-api-access-spv4r\") pod \"a91be1cf-9dbc-4810-ad79-36ff90efe774\" (UID: \"a91be1cf-9dbc-4810-ad79-36ff90efe774\") "
Sep 16 19:52:49.332390 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.332242    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a91be1cf-9dbc-4810-ad79-36ff90efe774-model-cache\") pod \"a91be1cf-9dbc-4810-ad79-36ff90efe774\" (UID: \"a91be1cf-9dbc-4810-ad79-36ff90efe774\") "
Sep 16 19:52:49.332390 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.332263    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a91be1cf-9dbc-4810-ad79-36ff90efe774-dshm\") pod \"a91be1cf-9dbc-4810-ad79-36ff90efe774\" (UID: \"a91be1cf-9dbc-4810-ad79-36ff90efe774\") "
Sep 16 19:52:49.332390 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.332305    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a91be1cf-9dbc-4810-ad79-36ff90efe774-kserve-provision-location\") pod \"a91be1cf-9dbc-4810-ad79-36ff90efe774\" (UID: \"a91be1cf-9dbc-4810-ad79-36ff90efe774\") "
Sep 16 19:52:49.332390 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.332322    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a91be1cf-9dbc-4810-ad79-36ff90efe774-tls-certs\") pod \"a91be1cf-9dbc-4810-ad79-36ff90efe774\" (UID: \"a91be1cf-9dbc-4810-ad79-36ff90efe774\") "
Sep 16 19:52:49.332390 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.332344    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a91be1cf-9dbc-4810-ad79-36ff90efe774-home\") pod \"a91be1cf-9dbc-4810-ad79-36ff90efe774\" (UID: \"a91be1cf-9dbc-4810-ad79-36ff90efe774\") "
Sep 16 19:52:49.332390 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.332373    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a91be1cf-9dbc-4810-ad79-36ff90efe774-tmp-dir\") pod \"a91be1cf-9dbc-4810-ad79-36ff90efe774\" (UID: \"a91be1cf-9dbc-4810-ad79-36ff90efe774\") "
Sep 16 19:52:49.332707 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.332428    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a91be1cf-9dbc-4810-ad79-36ff90efe774-cabundle-cert\") pod \"a91be1cf-9dbc-4810-ad79-36ff90efe774\" (UID: \"a91be1cf-9dbc-4810-ad79-36ff90efe774\") "
Sep 16 19:52:49.332707 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.332497    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91be1cf-9dbc-4810-ad79-36ff90efe774-model-cache" (OuterVolumeSpecName: "model-cache") pod "a91be1cf-9dbc-4810-ad79-36ff90efe774" (UID: "a91be1cf-9dbc-4810-ad79-36ff90efe774"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:52:49.332815 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.332738    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a91be1cf-9dbc-4810-ad79-36ff90efe774-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:52:49.332815 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.332792    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a91be1cf-9dbc-4810-ad79-36ff90efe774-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a91be1cf-9dbc-4810-ad79-36ff90efe774" (UID: "a91be1cf-9dbc-4810-ad79-36ff90efe774"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:52:49.333138 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.333106    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91be1cf-9dbc-4810-ad79-36ff90efe774-home" (OuterVolumeSpecName: "home") pod "a91be1cf-9dbc-4810-ad79-36ff90efe774" (UID: "a91be1cf-9dbc-4810-ad79-36ff90efe774"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:52:49.334721 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.334592    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91be1cf-9dbc-4810-ad79-36ff90efe774-dshm" (OuterVolumeSpecName: "dshm") pod "a91be1cf-9dbc-4810-ad79-36ff90efe774" (UID: "a91be1cf-9dbc-4810-ad79-36ff90efe774"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:52:49.334721 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.334680    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a91be1cf-9dbc-4810-ad79-36ff90efe774-kube-api-access-spv4r" (OuterVolumeSpecName: "kube-api-access-spv4r") pod "a91be1cf-9dbc-4810-ad79-36ff90efe774" (UID: "a91be1cf-9dbc-4810-ad79-36ff90efe774"). InnerVolumeSpecName "kube-api-access-spv4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:52:49.334847 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.334794    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a91be1cf-9dbc-4810-ad79-36ff90efe774-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a91be1cf-9dbc-4810-ad79-36ff90efe774" (UID: "a91be1cf-9dbc-4810-ad79-36ff90efe774"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:52:49.344734 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.344711    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91be1cf-9dbc-4810-ad79-36ff90efe774-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a91be1cf-9dbc-4810-ad79-36ff90efe774" (UID: "a91be1cf-9dbc-4810-ad79-36ff90efe774"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:52:49.346616 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.346594    2578 generic.go:356] "Generic (PLEG): container finished" podID="a91be1cf-9dbc-4810-ad79-36ff90efe774" containerID="0a302967b4f5f80b547c375100ba17c208e0c10d0f8eb3259917cc6f3c9ba565" exitCode=0
Sep 16 19:52:49.346616 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.346613    2578 generic.go:356] "Generic (PLEG): container finished" podID="a91be1cf-9dbc-4810-ad79-36ff90efe774" containerID="2ed60a6e53ad672175d4480e559120b97db4f3aaf3f421ed4263784fbc90067e" exitCode=0
Sep 16 19:52:49.346732 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.346651    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"
Sep 16 19:52:49.346732 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.346678    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd" event={"ID":"a91be1cf-9dbc-4810-ad79-36ff90efe774","Type":"ContainerDied","Data":"0a302967b4f5f80b547c375100ba17c208e0c10d0f8eb3259917cc6f3c9ba565"}
Sep 16 19:52:49.346798 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.346744    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd" event={"ID":"a91be1cf-9dbc-4810-ad79-36ff90efe774","Type":"ContainerDied","Data":"2ed60a6e53ad672175d4480e559120b97db4f3aaf3f421ed4263784fbc90067e"}
Sep 16 19:52:49.346798 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.346773    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd" event={"ID":"a91be1cf-9dbc-4810-ad79-36ff90efe774","Type":"ContainerDied","Data":"2011f25772ddb479bc34fcbcb8eb6ef9b4c9582c4c3e95df01105d6aece80635"}
Sep 16 19:52:49.346865 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.346804    2578 scope.go:117] "RemoveContainer" containerID="0a302967b4f5f80b547c375100ba17c208e0c10d0f8eb3259917cc6f3c9ba565"
Sep 16 19:52:49.356514 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.356496    2578 scope.go:117] "RemoveContainer" containerID="58b4405acaaf22cc5e2eaef72168a1637b610a11f9019b7100407ec47e785885"
Sep 16 19:52:49.357634 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.357614    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91be1cf-9dbc-4810-ad79-36ff90efe774-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a91be1cf-9dbc-4810-ad79-36ff90efe774" (UID: "a91be1cf-9dbc-4810-ad79-36ff90efe774"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:52:49.364172 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.363979    2578 scope.go:117] "RemoveContainer" containerID="2ed60a6e53ad672175d4480e559120b97db4f3aaf3f421ed4263784fbc90067e"
Sep 16 19:52:49.373224 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.373130    2578 scope.go:117] "RemoveContainer" containerID="0a302967b4f5f80b547c375100ba17c208e0c10d0f8eb3259917cc6f3c9ba565"
Sep 16 19:52:49.373641 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:52:49.373607    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a302967b4f5f80b547c375100ba17c208e0c10d0f8eb3259917cc6f3c9ba565\": container with ID starting with 0a302967b4f5f80b547c375100ba17c208e0c10d0f8eb3259917cc6f3c9ba565 not found: ID does not exist" containerID="0a302967b4f5f80b547c375100ba17c208e0c10d0f8eb3259917cc6f3c9ba565"
Sep 16 19:52:49.373709 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.373649    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a302967b4f5f80b547c375100ba17c208e0c10d0f8eb3259917cc6f3c9ba565"} err="failed to get container status \"0a302967b4f5f80b547c375100ba17c208e0c10d0f8eb3259917cc6f3c9ba565\": rpc error: code = NotFound desc = could not find container \"0a302967b4f5f80b547c375100ba17c208e0c10d0f8eb3259917cc6f3c9ba565\": container with ID starting with 0a302967b4f5f80b547c375100ba17c208e0c10d0f8eb3259917cc6f3c9ba565 not found: ID does not exist"
Sep 16 19:52:49.373709 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.373670    2578 scope.go:117] "RemoveContainer" containerID="58b4405acaaf22cc5e2eaef72168a1637b610a11f9019b7100407ec47e785885"
Sep 16 19:52:49.373952 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:52:49.373933    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58b4405acaaf22cc5e2eaef72168a1637b610a11f9019b7100407ec47e785885\": container with ID starting with 58b4405acaaf22cc5e2eaef72168a1637b610a11f9019b7100407ec47e785885 not found: ID does not exist" containerID="58b4405acaaf22cc5e2eaef72168a1637b610a11f9019b7100407ec47e785885"
Sep 16 19:52:49.374013 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.373956    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58b4405acaaf22cc5e2eaef72168a1637b610a11f9019b7100407ec47e785885"} err="failed to get container status \"58b4405acaaf22cc5e2eaef72168a1637b610a11f9019b7100407ec47e785885\": rpc error: code = NotFound desc = could not find container \"58b4405acaaf22cc5e2eaef72168a1637b610a11f9019b7100407ec47e785885\": container with ID starting with 58b4405acaaf22cc5e2eaef72168a1637b610a11f9019b7100407ec47e785885 not found: ID does not exist"
Sep 16 19:52:49.374013 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.373974    2578 scope.go:117] "RemoveContainer" containerID="2ed60a6e53ad672175d4480e559120b97db4f3aaf3f421ed4263784fbc90067e"
Sep 16 19:52:49.374264 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:52:49.374246    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ed60a6e53ad672175d4480e559120b97db4f3aaf3f421ed4263784fbc90067e\": container with ID starting with 2ed60a6e53ad672175d4480e559120b97db4f3aaf3f421ed4263784fbc90067e not found: ID does not exist" containerID="2ed60a6e53ad672175d4480e559120b97db4f3aaf3f421ed4263784fbc90067e"
Sep 16 19:52:49.374331 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.374271    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ed60a6e53ad672175d4480e559120b97db4f3aaf3f421ed4263784fbc90067e"} err="failed to get container status \"2ed60a6e53ad672175d4480e559120b97db4f3aaf3f421ed4263784fbc90067e\": rpc error: code = NotFound desc = could not find container \"2ed60a6e53ad672175d4480e559120b97db4f3aaf3f421ed4263784fbc90067e\": container with ID starting with 2ed60a6e53ad672175d4480e559120b97db4f3aaf3f421ed4263784fbc90067e not found: ID does not exist"
Sep 16 19:52:49.374331 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.374291    2578 scope.go:117] "RemoveContainer" containerID="0a302967b4f5f80b547c375100ba17c208e0c10d0f8eb3259917cc6f3c9ba565"
Sep 16 19:52:49.374558 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.374540    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a302967b4f5f80b547c375100ba17c208e0c10d0f8eb3259917cc6f3c9ba565"} err="failed to get container status \"0a302967b4f5f80b547c375100ba17c208e0c10d0f8eb3259917cc6f3c9ba565\": rpc error: code = NotFound desc = could not find container \"0a302967b4f5f80b547c375100ba17c208e0c10d0f8eb3259917cc6f3c9ba565\": container with ID starting with 0a302967b4f5f80b547c375100ba17c208e0c10d0f8eb3259917cc6f3c9ba565 not found: ID does not exist"
Sep 16 19:52:49.374630 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.374558    2578 scope.go:117] "RemoveContainer" containerID="58b4405acaaf22cc5e2eaef72168a1637b610a11f9019b7100407ec47e785885"
Sep 16 19:52:49.374804 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.374787    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58b4405acaaf22cc5e2eaef72168a1637b610a11f9019b7100407ec47e785885"} err="failed to get container status \"58b4405acaaf22cc5e2eaef72168a1637b610a11f9019b7100407ec47e785885\": rpc error: code = NotFound desc = could not find container \"58b4405acaaf22cc5e2eaef72168a1637b610a11f9019b7100407ec47e785885\": container with ID starting with 58b4405acaaf22cc5e2eaef72168a1637b610a11f9019b7100407ec47e785885 not found: ID does not exist"
Sep 16 19:52:49.374870 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.374807    2578 scope.go:117] "RemoveContainer" containerID="2ed60a6e53ad672175d4480e559120b97db4f3aaf3f421ed4263784fbc90067e"
Sep 16 19:52:49.375065 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.375033    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ed60a6e53ad672175d4480e559120b97db4f3aaf3f421ed4263784fbc90067e"} err="failed to get container status \"2ed60a6e53ad672175d4480e559120b97db4f3aaf3f421ed4263784fbc90067e\": rpc error: code = NotFound desc = could not find container \"2ed60a6e53ad672175d4480e559120b97db4f3aaf3f421ed4263784fbc90067e\": container with ID starting with 2ed60a6e53ad672175d4480e559120b97db4f3aaf3f421ed4263784fbc90067e not found: ID does not exist"
Sep 16 19:52:49.433466 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.433444    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a91be1cf-9dbc-4810-ad79-36ff90efe774-kserve-provision-location\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:52:49.433548 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.433466    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a91be1cf-9dbc-4810-ad79-36ff90efe774-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:52:49.433548 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.433476    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a91be1cf-9dbc-4810-ad79-36ff90efe774-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:52:49.433548 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.433484    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a91be1cf-9dbc-4810-ad79-36ff90efe774-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:52:49.433548 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.433491    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a91be1cf-9dbc-4810-ad79-36ff90efe774-cabundle-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:52:49.433548 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.433500    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spv4r\" (UniqueName: \"kubernetes.io/projected/a91be1cf-9dbc-4810-ad79-36ff90efe774-kube-api-access-spv4r\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:52:49.433548 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.433507    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a91be1cf-9dbc-4810-ad79-36ff90efe774-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:52:49.676815 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.676785    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"]
Sep 16 19:52:49.681927 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:49.681900    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7b6cfd55d5-25gmd"]
Sep 16 19:52:50.217652 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:52:50.217615    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a91be1cf-9dbc-4810-ad79-36ff90efe774" path="/var/lib/kubelet/pods/a91be1cf-9dbc-4810-ad79-36ff90efe774/volumes"
Sep 16 19:53:28.389563 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:28.389524    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"]
Sep 16 19:53:28.390152 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:28.389706    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2" podUID="e1920012-14fb-4875-87fb-7ac2da56702e" containerName="istio-proxy" containerID="cri-o://49fa20e8c84262c2d1c82ff4231bbe1d082aad549de40838eaf81bb8434ca5a7" gracePeriod=30
Sep 16 19:53:33.585937 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:33.585908    2578 generic.go:356] "Generic (PLEG): container finished" podID="e1920012-14fb-4875-87fb-7ac2da56702e" containerID="49fa20e8c84262c2d1c82ff4231bbe1d082aad549de40838eaf81bb8434ca5a7" exitCode=0
Sep 16 19:53:33.586295 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:33.585952    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2" event={"ID":"e1920012-14fb-4875-87fb-7ac2da56702e","Type":"ContainerDied","Data":"49fa20e8c84262c2d1c82ff4231bbe1d082aad549de40838eaf81bb8434ca5a7"}
Sep 16 19:53:33.586295 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:33.586016    2578 scope.go:117] "RemoveContainer" containerID="fb413967c7562858589268daeeb4c0be9fd6647f083fba64b5ede4cd01716e9b"
Sep 16 19:53:33.640535 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:33.640514    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:53:33.758071 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:33.758028    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1920012-14fb-4875-87fb-7ac2da56702e-istio-token\") pod \"e1920012-14fb-4875-87fb-7ac2da56702e\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") "
Sep 16 19:53:33.758203 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:33.758076    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1920012-14fb-4875-87fb-7ac2da56702e-credential-socket\") pod \"e1920012-14fb-4875-87fb-7ac2da56702e\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") "
Sep 16 19:53:33.758203 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:33.758100    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1920012-14fb-4875-87fb-7ac2da56702e-istio-envoy\") pod \"e1920012-14fb-4875-87fb-7ac2da56702e\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") "
Sep 16 19:53:33.758203 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:33.758132    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1920012-14fb-4875-87fb-7ac2da56702e-workload-certs\") pod \"e1920012-14fb-4875-87fb-7ac2da56702e\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") "
Sep 16 19:53:33.758203 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:33.758150    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxzqb\" (UniqueName: \"kubernetes.io/projected/e1920012-14fb-4875-87fb-7ac2da56702e-kube-api-access-xxzqb\") pod \"e1920012-14fb-4875-87fb-7ac2da56702e\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") "
Sep 16 19:53:33.758203 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:33.758170    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1920012-14fb-4875-87fb-7ac2da56702e-istio-podinfo\") pod \"e1920012-14fb-4875-87fb-7ac2da56702e\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") "
Sep 16 19:53:33.758203 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:33.758187    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1920012-14fb-4875-87fb-7ac2da56702e-workload-socket\") pod \"e1920012-14fb-4875-87fb-7ac2da56702e\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") "
Sep 16 19:53:33.758504 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:33.758467    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1920012-14fb-4875-87fb-7ac2da56702e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e1920012-14fb-4875-87fb-7ac2da56702e" (UID: "e1920012-14fb-4875-87fb-7ac2da56702e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:53:33.758665 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:33.758512    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1920012-14fb-4875-87fb-7ac2da56702e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e1920012-14fb-4875-87fb-7ac2da56702e" (UID: "e1920012-14fb-4875-87fb-7ac2da56702e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:53:33.758665 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:33.758533    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1920012-14fb-4875-87fb-7ac2da56702e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e1920012-14fb-4875-87fb-7ac2da56702e" (UID: "e1920012-14fb-4875-87fb-7ac2da56702e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:53:33.758665 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:33.758221    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1920012-14fb-4875-87fb-7ac2da56702e-istiod-ca-cert\") pod \"e1920012-14fb-4875-87fb-7ac2da56702e\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") "
Sep 16 19:53:33.758665 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:33.758613    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1920012-14fb-4875-87fb-7ac2da56702e-istio-data\") pod \"e1920012-14fb-4875-87fb-7ac2da56702e\" (UID: \"e1920012-14fb-4875-87fb-7ac2da56702e\") "
Sep 16 19:53:33.758928 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:33.758893    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1920012-14fb-4875-87fb-7ac2da56702e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e1920012-14fb-4875-87fb-7ac2da56702e" (UID: "e1920012-14fb-4875-87fb-7ac2da56702e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:53:33.758928 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:33.758901    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1920012-14fb-4875-87fb-7ac2da56702e-istio-data" (OuterVolumeSpecName: "istio-data") pod "e1920012-14fb-4875-87fb-7ac2da56702e" (UID: "e1920012-14fb-4875-87fb-7ac2da56702e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:53:33.759059 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:33.758971    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1920012-14fb-4875-87fb-7ac2da56702e-istio-data\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:53:33.759059 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:33.758989    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1920012-14fb-4875-87fb-7ac2da56702e-credential-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:53:33.759059 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:33.759004    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1920012-14fb-4875-87fb-7ac2da56702e-workload-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:53:33.759059 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:33.759017    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1920012-14fb-4875-87fb-7ac2da56702e-workload-socket\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:53:33.759059 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:33.759032    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1920012-14fb-4875-87fb-7ac2da56702e-istiod-ca-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:53:33.760376 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:33.760348    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1920012-14fb-4875-87fb-7ac2da56702e-kube-api-access-xxzqb" (OuterVolumeSpecName: "kube-api-access-xxzqb") pod "e1920012-14fb-4875-87fb-7ac2da56702e" (UID: "e1920012-14fb-4875-87fb-7ac2da56702e"). InnerVolumeSpecName "kube-api-access-xxzqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:53:33.760507 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:33.760479    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1920012-14fb-4875-87fb-7ac2da56702e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e1920012-14fb-4875-87fb-7ac2da56702e" (UID: "e1920012-14fb-4875-87fb-7ac2da56702e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 19:53:33.760613 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:33.760595    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1920012-14fb-4875-87fb-7ac2da56702e-istio-token" (OuterVolumeSpecName: "istio-token") pod "e1920012-14fb-4875-87fb-7ac2da56702e" (UID: "e1920012-14fb-4875-87fb-7ac2da56702e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:53:33.760673 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:33.760613    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1920012-14fb-4875-87fb-7ac2da56702e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e1920012-14fb-4875-87fb-7ac2da56702e" (UID: "e1920012-14fb-4875-87fb-7ac2da56702e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:53:33.860449 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:33.860422    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1920012-14fb-4875-87fb-7ac2da56702e-istio-podinfo\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:53:33.860449 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:33.860449    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1920012-14fb-4875-87fb-7ac2da56702e-istio-token\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:53:33.860619 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:33.860457    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1920012-14fb-4875-87fb-7ac2da56702e-istio-envoy\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:53:33.860619 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:33.860465    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxzqb\" (UniqueName: \"kubernetes.io/projected/e1920012-14fb-4875-87fb-7ac2da56702e-kube-api-access-xxzqb\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:53:34.592800 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:34.592725    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"
Sep 16 19:53:34.592800 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:34.592765    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2" event={"ID":"e1920012-14fb-4875-87fb-7ac2da56702e","Type":"ContainerDied","Data":"f4793db5b295511e8d2d47f7a169abeee5ebf1528d81e635890ac04818141cdf"}
Sep 16 19:53:34.593253 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:34.592825    2578 scope.go:117] "RemoveContainer" containerID="49fa20e8c84262c2d1c82ff4231bbe1d082aad549de40838eaf81bb8434ca5a7"
Sep 16 19:53:34.613979 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:34.613951    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"]
Sep 16 19:53:34.620829 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:34.620808    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7f645c8cbd-qgdw2"]
Sep 16 19:53:36.217145 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:36.217113    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1920012-14fb-4875-87fb-7ac2da56702e" path="/var/lib/kubelet/pods/e1920012-14fb-4875-87fb-7ac2da56702e/volumes"
Sep 16 19:53:49.950234 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:49.950194    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"]
Sep 16 19:53:49.950605 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:49.950484    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a91be1cf-9dbc-4810-ad79-36ff90efe774" containerName="storage-initializer"
Sep 16 19:53:49.950605 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:49.950495    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a91be1cf-9dbc-4810-ad79-36ff90efe774" containerName="storage-initializer"
Sep 16 19:53:49.950605 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:49.950511    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a91be1cf-9dbc-4810-ad79-36ff90efe774" containerName="main"
Sep 16 19:53:49.950605 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:49.950516    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a91be1cf-9dbc-4810-ad79-36ff90efe774" containerName="main"
Sep 16 19:53:49.950605 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:49.950523    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1920012-14fb-4875-87fb-7ac2da56702e" containerName="istio-proxy"
Sep 16 19:53:49.950605 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:49.950529    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1920012-14fb-4875-87fb-7ac2da56702e" containerName="istio-proxy"
Sep 16 19:53:49.950605 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:49.950537    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a91be1cf-9dbc-4810-ad79-36ff90efe774" containerName="llm-d-routing-sidecar"
Sep 16 19:53:49.950605 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:49.950542    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a91be1cf-9dbc-4810-ad79-36ff90efe774" containerName="llm-d-routing-sidecar"
Sep 16 19:53:49.950605 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:49.950551    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1920012-14fb-4875-87fb-7ac2da56702e" containerName="istio-proxy"
Sep 16 19:53:49.950605 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:49.950556    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1920012-14fb-4875-87fb-7ac2da56702e" containerName="istio-proxy"
Sep 16 19:53:49.950605 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:49.950606    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1920012-14fb-4875-87fb-7ac2da56702e" containerName="istio-proxy"
Sep 16 19:53:49.950952 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:49.950615    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1920012-14fb-4875-87fb-7ac2da56702e" containerName="istio-proxy"
Sep 16 19:53:49.950952 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:49.950621    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a91be1cf-9dbc-4810-ad79-36ff90efe774" containerName="llm-d-routing-sidecar"
Sep 16 19:53:49.950952 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:49.950627    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a91be1cf-9dbc-4810-ad79-36ff90efe774" containerName="main"
Sep 16 19:53:49.954848 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:49.954822    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"
Sep 16 19:53:49.957483 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:49.957458    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:53:49.957710 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:49.957458    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 16 19:53:49.965227 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:49.965203    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"]
Sep 16 19:53:50.089808 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:50.089779    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hm6t\" (UniqueName: \"kubernetes.io/projected/d128fa42-41f0-4af8-95da-cce98c379a0b-kube-api-access-7hm6t\") pod \"prestop-hook-test-kserve-87c8d84b4-mkpnj\" (UID: \"d128fa42-41f0-4af8-95da-cce98c379a0b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"
Sep 16 19:53:50.089957 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:50.089813    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d128fa42-41f0-4af8-95da-cce98c379a0b-home\") pod \"prestop-hook-test-kserve-87c8d84b4-mkpnj\" (UID: \"d128fa42-41f0-4af8-95da-cce98c379a0b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"
Sep 16 19:53:50.089957 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:50.089836    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d128fa42-41f0-4af8-95da-cce98c379a0b-tls-certs\") pod \"prestop-hook-test-kserve-87c8d84b4-mkpnj\" (UID: \"d128fa42-41f0-4af8-95da-cce98c379a0b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"
Sep 16 19:53:50.089957 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:50.089910    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d128fa42-41f0-4af8-95da-cce98c379a0b-dshm\") pod \"prestop-hook-test-kserve-87c8d84b4-mkpnj\" (UID: \"d128fa42-41f0-4af8-95da-cce98c379a0b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"
Sep 16 19:53:50.089957 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:50.089944    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d128fa42-41f0-4af8-95da-cce98c379a0b-model-cache\") pod \"prestop-hook-test-kserve-87c8d84b4-mkpnj\" (UID: \"d128fa42-41f0-4af8-95da-cce98c379a0b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"
Sep 16 19:53:50.090146 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:50.089983    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d128fa42-41f0-4af8-95da-cce98c379a0b-cabundle-cert\") pod \"prestop-hook-test-kserve-87c8d84b4-mkpnj\" (UID: \"d128fa42-41f0-4af8-95da-cce98c379a0b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"
Sep 16 19:53:50.090146 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:50.090089    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d128fa42-41f0-4af8-95da-cce98c379a0b-kserve-provision-location\") pod \"prestop-hook-test-kserve-87c8d84b4-mkpnj\" (UID: \"d128fa42-41f0-4af8-95da-cce98c379a0b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"
Sep 16 19:53:50.090146 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:50.090117    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d128fa42-41f0-4af8-95da-cce98c379a0b-tmp-dir\") pod \"prestop-hook-test-kserve-87c8d84b4-mkpnj\" (UID: \"d128fa42-41f0-4af8-95da-cce98c379a0b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"
Sep 16 19:53:50.190815 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:50.190789    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d128fa42-41f0-4af8-95da-cce98c379a0b-cabundle-cert\") pod \"prestop-hook-test-kserve-87c8d84b4-mkpnj\" (UID: \"d128fa42-41f0-4af8-95da-cce98c379a0b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"
Sep 16 19:53:50.190931 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:50.190830    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d128fa42-41f0-4af8-95da-cce98c379a0b-kserve-provision-location\") pod \"prestop-hook-test-kserve-87c8d84b4-mkpnj\" (UID: \"d128fa42-41f0-4af8-95da-cce98c379a0b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"
Sep 16 19:53:50.190931 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:50.190849    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d128fa42-41f0-4af8-95da-cce98c379a0b-tmp-dir\") pod \"prestop-hook-test-kserve-87c8d84b4-mkpnj\" (UID: \"d128fa42-41f0-4af8-95da-cce98c379a0b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"
Sep 16 19:53:50.190931 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:50.190869    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hm6t\" (UniqueName: \"kubernetes.io/projected/d128fa42-41f0-4af8-95da-cce98c379a0b-kube-api-access-7hm6t\") pod \"prestop-hook-test-kserve-87c8d84b4-mkpnj\" (UID: \"d128fa42-41f0-4af8-95da-cce98c379a0b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"
Sep 16 19:53:50.191076 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:50.190976    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d128fa42-41f0-4af8-95da-cce98c379a0b-home\") pod \"prestop-hook-test-kserve-87c8d84b4-mkpnj\" (UID: \"d128fa42-41f0-4af8-95da-cce98c379a0b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"
Sep 16 19:53:50.191076 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:50.191012    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d128fa42-41f0-4af8-95da-cce98c379a0b-tls-certs\") pod \"prestop-hook-test-kserve-87c8d84b4-mkpnj\" (UID: \"d128fa42-41f0-4af8-95da-cce98c379a0b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"
Sep 16 19:53:50.191076 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:50.191063    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d128fa42-41f0-4af8-95da-cce98c379a0b-dshm\") pod \"prestop-hook-test-kserve-87c8d84b4-mkpnj\" (UID: \"d128fa42-41f0-4af8-95da-cce98c379a0b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"
Sep 16 19:53:50.191237 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:50.191090    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d128fa42-41f0-4af8-95da-cce98c379a0b-model-cache\") pod \"prestop-hook-test-kserve-87c8d84b4-mkpnj\" (UID: \"d128fa42-41f0-4af8-95da-cce98c379a0b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"
Sep 16 19:53:50.191237 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:50.191182    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d128fa42-41f0-4af8-95da-cce98c379a0b-tmp-dir\") pod \"prestop-hook-test-kserve-87c8d84b4-mkpnj\" (UID: \"d128fa42-41f0-4af8-95da-cce98c379a0b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"
Sep 16 19:53:50.191485 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:50.191462    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d128fa42-41f0-4af8-95da-cce98c379a0b-model-cache\") pod \"prestop-hook-test-kserve-87c8d84b4-mkpnj\" (UID: \"d128fa42-41f0-4af8-95da-cce98c379a0b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"
Sep 16 19:53:50.191572 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:50.191510    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d128fa42-41f0-4af8-95da-cce98c379a0b-home\") pod \"prestop-hook-test-kserve-87c8d84b4-mkpnj\" (UID: \"d128fa42-41f0-4af8-95da-cce98c379a0b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"
Sep 16 19:53:50.191617 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:50.191567    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d128fa42-41f0-4af8-95da-cce98c379a0b-kserve-provision-location\") pod \"prestop-hook-test-kserve-87c8d84b4-mkpnj\" (UID: \"d128fa42-41f0-4af8-95da-cce98c379a0b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"
Sep 16 19:53:50.191617 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:50.191607    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d128fa42-41f0-4af8-95da-cce98c379a0b-cabundle-cert\") pod \"prestop-hook-test-kserve-87c8d84b4-mkpnj\" (UID: \"d128fa42-41f0-4af8-95da-cce98c379a0b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"
Sep 16 19:53:50.193280 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:50.193262    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d128fa42-41f0-4af8-95da-cce98c379a0b-dshm\") pod \"prestop-hook-test-kserve-87c8d84b4-mkpnj\" (UID: \"d128fa42-41f0-4af8-95da-cce98c379a0b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"
Sep 16 19:53:50.193505 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:50.193487    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d128fa42-41f0-4af8-95da-cce98c379a0b-tls-certs\") pod \"prestop-hook-test-kserve-87c8d84b4-mkpnj\" (UID: \"d128fa42-41f0-4af8-95da-cce98c379a0b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"
Sep 16 19:53:50.199133 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:50.199114    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hm6t\" (UniqueName: \"kubernetes.io/projected/d128fa42-41f0-4af8-95da-cce98c379a0b-kube-api-access-7hm6t\") pod \"prestop-hook-test-kserve-87c8d84b4-mkpnj\" (UID: \"d128fa42-41f0-4af8-95da-cce98c379a0b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"
Sep 16 19:53:50.265521 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:50.265463    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"
Sep 16 19:53:50.544857 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:50.544832    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"]
Sep 16 19:53:50.547419 ip-10-0-142-80 kubenswrapper[2578]: W0916 19:53:50.547392    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd128fa42_41f0_4af8_95da_cce98c379a0b.slice/crio-02c353965caa59a5742f4e46255bd794ea36e7d4fa060322e9203b9c2203d2ff WatchSource:0}: Error finding container 02c353965caa59a5742f4e46255bd794ea36e7d4fa060322e9203b9c2203d2ff: Status 404 returned error can't find the container with id 02c353965caa59a5742f4e46255bd794ea36e7d4fa060322e9203b9c2203d2ff
Sep 16 19:53:50.673142 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:50.673108    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj" event={"ID":"d128fa42-41f0-4af8-95da-cce98c379a0b","Type":"ContainerStarted","Data":"686700c56f31c3145daf6d5b4b671cc840c2099b36ec795202675617a8732d82"}
Sep 16 19:53:50.673239 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:50.673163    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj" event={"ID":"d128fa42-41f0-4af8-95da-cce98c379a0b","Type":"ContainerStarted","Data":"02c353965caa59a5742f4e46255bd794ea36e7d4fa060322e9203b9c2203d2ff"}
Sep 16 19:53:52.685370 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:52.685337    2578 generic.go:356] "Generic (PLEG): container finished" podID="d128fa42-41f0-4af8-95da-cce98c379a0b" containerID="686700c56f31c3145daf6d5b4b671cc840c2099b36ec795202675617a8732d82" exitCode=0
Sep 16 19:53:52.685687 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:52.685394    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj" event={"ID":"d128fa42-41f0-4af8-95da-cce98c379a0b","Type":"ContainerDied","Data":"686700c56f31c3145daf6d5b4b671cc840c2099b36ec795202675617a8732d82"}
Sep 16 19:53:53.692741 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:53.692707    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj" event={"ID":"d128fa42-41f0-4af8-95da-cce98c379a0b","Type":"ContainerStarted","Data":"8697312dab774eb696b9a7a4d397830406a2cb191381347d0de9542f5a93de2a"}
Sep 16 19:53:53.716490 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:53.716443    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj" podStartSLOduration=2.689694611 podStartE2EDuration="4.716427733s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.026733122s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:53:53.714823831 +0000 UTC m=+4401.949757285" watchObservedRunningTime="2026-09-16 19:53:53.716427733 +0000 UTC m=+4401.951361175"
Sep 16 19:53:54.266548 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:53:54.266515    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"
Sep 16 19:54:00.265688 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:00.265607    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"
Sep 16 19:54:00.267093 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:00.267065    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj" podUID="d128fa42-41f0-4af8-95da-cce98c379a0b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Sep 16 19:54:10.266093 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:10.266024    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj" podUID="d128fa42-41f0-4af8-95da-cce98c379a0b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Sep 16 19:54:14.858716 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:14.858668    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds"]
Sep 16 19:54:14.867978 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:14.867946    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"]
Sep 16 19:54:21.051513 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:21.051475    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds" podUID="90f20be3-aae7-44c1-9ed6-6aca54210719" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:54:29.880735 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:29.880690    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds" podUID="90f20be3-aae7-44c1-9ed6-6aca54210719" containerName="main" containerID="cri-o://25aa3f45d2f5050f25fa3261449cacb4d3af59916b9af816b35442c1ecbfdee3" gracePeriod=45
Sep 16 19:54:29.897341 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:29.897303    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj" podUID="d128fa42-41f0-4af8-95da-cce98c379a0b" containerName="main" containerID="cri-o://8697312dab774eb696b9a7a4d397830406a2cb191381347d0de9542f5a93de2a" gracePeriod=45
Sep 16 19:54:31.051687 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.051638    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds" podUID="90f20be3-aae7-44c1-9ed6-6aca54210719" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:54:31.547512 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.547490    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-87c8d84b4-mkpnj_d128fa42-41f0-4af8-95da-cce98c379a0b/main/0.log"
Sep 16 19:54:31.550197 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.550179    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"
Sep 16 19:54:31.622086 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.622062    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d128fa42-41f0-4af8-95da-cce98c379a0b-tmp-dir\") pod \"d128fa42-41f0-4af8-95da-cce98c379a0b\" (UID: \"d128fa42-41f0-4af8-95da-cce98c379a0b\") "
Sep 16 19:54:31.622206 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.622094    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d128fa42-41f0-4af8-95da-cce98c379a0b-dshm\") pod \"d128fa42-41f0-4af8-95da-cce98c379a0b\" (UID: \"d128fa42-41f0-4af8-95da-cce98c379a0b\") "
Sep 16 19:54:31.622206 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.622121    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hm6t\" (UniqueName: \"kubernetes.io/projected/d128fa42-41f0-4af8-95da-cce98c379a0b-kube-api-access-7hm6t\") pod \"d128fa42-41f0-4af8-95da-cce98c379a0b\" (UID: \"d128fa42-41f0-4af8-95da-cce98c379a0b\") "
Sep 16 19:54:31.622206 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.622145    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d128fa42-41f0-4af8-95da-cce98c379a0b-kserve-provision-location\") pod \"d128fa42-41f0-4af8-95da-cce98c379a0b\" (UID: \"d128fa42-41f0-4af8-95da-cce98c379a0b\") "
Sep 16 19:54:31.622206 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.622174    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d128fa42-41f0-4af8-95da-cce98c379a0b-cabundle-cert\") pod \"d128fa42-41f0-4af8-95da-cce98c379a0b\" (UID: \"d128fa42-41f0-4af8-95da-cce98c379a0b\") "
Sep 16 19:54:31.622206 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.622203    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d128fa42-41f0-4af8-95da-cce98c379a0b-home\") pod \"d128fa42-41f0-4af8-95da-cce98c379a0b\" (UID: \"d128fa42-41f0-4af8-95da-cce98c379a0b\") "
Sep 16 19:54:31.622500 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.622251    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d128fa42-41f0-4af8-95da-cce98c379a0b-tls-certs\") pod \"d128fa42-41f0-4af8-95da-cce98c379a0b\" (UID: \"d128fa42-41f0-4af8-95da-cce98c379a0b\") "
Sep 16 19:54:31.622500 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.622297    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d128fa42-41f0-4af8-95da-cce98c379a0b-model-cache\") pod \"d128fa42-41f0-4af8-95da-cce98c379a0b\" (UID: \"d128fa42-41f0-4af8-95da-cce98c379a0b\") "
Sep 16 19:54:31.622500 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.622435    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d128fa42-41f0-4af8-95da-cce98c379a0b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d128fa42-41f0-4af8-95da-cce98c379a0b" (UID: "d128fa42-41f0-4af8-95da-cce98c379a0b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:54:31.622657 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.622549    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d128fa42-41f0-4af8-95da-cce98c379a0b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d128fa42-41f0-4af8-95da-cce98c379a0b" (UID: "d128fa42-41f0-4af8-95da-cce98c379a0b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:54:31.622657 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.622568    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d128fa42-41f0-4af8-95da-cce98c379a0b-tmp-dir\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:54:31.622775 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.622675    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d128fa42-41f0-4af8-95da-cce98c379a0b-model-cache" (OuterVolumeSpecName: "model-cache") pod "d128fa42-41f0-4af8-95da-cce98c379a0b" (UID: "d128fa42-41f0-4af8-95da-cce98c379a0b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:54:31.622775 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.622694    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d128fa42-41f0-4af8-95da-cce98c379a0b-home" (OuterVolumeSpecName: "home") pod "d128fa42-41f0-4af8-95da-cce98c379a0b" (UID: "d128fa42-41f0-4af8-95da-cce98c379a0b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:54:31.624353 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.624331    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d128fa42-41f0-4af8-95da-cce98c379a0b-dshm" (OuterVolumeSpecName: "dshm") pod "d128fa42-41f0-4af8-95da-cce98c379a0b" (UID: "d128fa42-41f0-4af8-95da-cce98c379a0b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:54:31.624485 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.624467    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d128fa42-41f0-4af8-95da-cce98c379a0b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d128fa42-41f0-4af8-95da-cce98c379a0b" (UID: "d128fa42-41f0-4af8-95da-cce98c379a0b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:54:31.624532 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.624486    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d128fa42-41f0-4af8-95da-cce98c379a0b-kube-api-access-7hm6t" (OuterVolumeSpecName: "kube-api-access-7hm6t") pod "d128fa42-41f0-4af8-95da-cce98c379a0b" (UID: "d128fa42-41f0-4af8-95da-cce98c379a0b"). InnerVolumeSpecName "kube-api-access-7hm6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:54:31.644495 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.644437    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d128fa42-41f0-4af8-95da-cce98c379a0b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d128fa42-41f0-4af8-95da-cce98c379a0b" (UID: "d128fa42-41f0-4af8-95da-cce98c379a0b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:54:31.723182 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.723159    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d128fa42-41f0-4af8-95da-cce98c379a0b-dshm\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:54:31.723182 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.723182    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hm6t\" (UniqueName: \"kubernetes.io/projected/d128fa42-41f0-4af8-95da-cce98c379a0b-kube-api-access-7hm6t\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:54:31.723309 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.723193    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d128fa42-41f0-4af8-95da-cce98c379a0b-kserve-provision-location\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:54:31.723309 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.723202    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d128fa42-41f0-4af8-95da-cce98c379a0b-cabundle-cert\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:54:31.723309 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.723210    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d128fa42-41f0-4af8-95da-cce98c379a0b-home\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:54:31.723309 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.723218    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d128fa42-41f0-4af8-95da-cce98c379a0b-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:54:31.723309 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.723225    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d128fa42-41f0-4af8-95da-cce98c379a0b-model-cache\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:54:31.896840 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.896779    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-87c8d84b4-mkpnj_d128fa42-41f0-4af8-95da-cce98c379a0b/main/0.log"
Sep 16 19:54:31.897193 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.897170    2578 generic.go:356] "Generic (PLEG): container finished" podID="d128fa42-41f0-4af8-95da-cce98c379a0b" containerID="8697312dab774eb696b9a7a4d397830406a2cb191381347d0de9542f5a93de2a" exitCode=130
Sep 16 19:54:31.897250 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.897221    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"
Sep 16 19:54:31.897293 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.897256    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj" event={"ID":"d128fa42-41f0-4af8-95da-cce98c379a0b","Type":"ContainerDied","Data":"8697312dab774eb696b9a7a4d397830406a2cb191381347d0de9542f5a93de2a"}
Sep 16 19:54:31.897329 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.897314    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj" event={"ID":"d128fa42-41f0-4af8-95da-cce98c379a0b","Type":"ContainerDied","Data":"02c353965caa59a5742f4e46255bd794ea36e7d4fa060322e9203b9c2203d2ff"}
Sep 16 19:54:31.897363 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.897338    2578 scope.go:117] "RemoveContainer" containerID="8697312dab774eb696b9a7a4d397830406a2cb191381347d0de9542f5a93de2a"
Sep 16 19:54:31.906936 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.906917    2578 scope.go:117] "RemoveContainer" containerID="686700c56f31c3145daf6d5b4b671cc840c2099b36ec795202675617a8732d82"
Sep 16 19:54:31.914801 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.914783    2578 scope.go:117] "RemoveContainer" containerID="8697312dab774eb696b9a7a4d397830406a2cb191381347d0de9542f5a93de2a"
Sep 16 19:54:31.915015 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:54:31.914994    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8697312dab774eb696b9a7a4d397830406a2cb191381347d0de9542f5a93de2a\": container with ID starting with 8697312dab774eb696b9a7a4d397830406a2cb191381347d0de9542f5a93de2a not found: ID does not exist" containerID="8697312dab774eb696b9a7a4d397830406a2cb191381347d0de9542f5a93de2a"
Sep 16 19:54:31.915080 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.915022    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8697312dab774eb696b9a7a4d397830406a2cb191381347d0de9542f5a93de2a"} err="failed to get container status \"8697312dab774eb696b9a7a4d397830406a2cb191381347d0de9542f5a93de2a\": rpc error: code = NotFound desc = could not find container \"8697312dab774eb696b9a7a4d397830406a2cb191381347d0de9542f5a93de2a\": container with ID starting with 8697312dab774eb696b9a7a4d397830406a2cb191381347d0de9542f5a93de2a not found: ID does not exist"
Sep 16 19:54:31.915080 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.915066    2578 scope.go:117] "RemoveContainer" containerID="686700c56f31c3145daf6d5b4b671cc840c2099b36ec795202675617a8732d82"
Sep 16 19:54:31.915322 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:54:31.915305    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"686700c56f31c3145daf6d5b4b671cc840c2099b36ec795202675617a8732d82\": container with ID starting with 686700c56f31c3145daf6d5b4b671cc840c2099b36ec795202675617a8732d82 not found: ID does not exist" containerID="686700c56f31c3145daf6d5b4b671cc840c2099b36ec795202675617a8732d82"
Sep 16 19:54:31.915374 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.915328    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"686700c56f31c3145daf6d5b4b671cc840c2099b36ec795202675617a8732d82"} err="failed to get container status \"686700c56f31c3145daf6d5b4b671cc840c2099b36ec795202675617a8732d82\": rpc error: code = NotFound desc = could not find container \"686700c56f31c3145daf6d5b4b671cc840c2099b36ec795202675617a8732d82\": container with ID starting with 686700c56f31c3145daf6d5b4b671cc840c2099b36ec795202675617a8732d82 not found: ID does not exist"
Sep 16 19:54:31.921936 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.921916    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"]
Sep 16 19:54:31.930114 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:31.930093    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-87c8d84b4-mkpnj"]
Sep 16 19:54:32.219993 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:32.219970    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d128fa42-41f0-4af8-95da-cce98c379a0b" path="/var/lib/kubelet/pods/d128fa42-41f0-4af8-95da-cce98c379a0b/volumes"
Sep 16 19:54:41.050942 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:41.050889    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds" podUID="90f20be3-aae7-44c1-9ed6-6aca54210719" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:54:41.051416 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:41.050997    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds"
Sep 16 19:54:51.051550 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:54:51.051507    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds" podUID="90f20be3-aae7-44c1-9ed6-6aca54210719" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:55:00.055420 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:00.055390    2578 generic.go:356] "Generic (PLEG): container finished" podID="90f20be3-aae7-44c1-9ed6-6aca54210719" containerID="25aa3f45d2f5050f25fa3261449cacb4d3af59916b9af816b35442c1ecbfdee3" exitCode=0
Sep 16 19:55:00.055746 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:00.055435    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds" event={"ID":"90f20be3-aae7-44c1-9ed6-6aca54210719","Type":"ContainerDied","Data":"25aa3f45d2f5050f25fa3261449cacb4d3af59916b9af816b35442c1ecbfdee3"}
Sep 16 19:55:00.127844 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:00.127821    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds"
Sep 16 19:55:00.225872 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:00.225835    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90f20be3-aae7-44c1-9ed6-6aca54210719-tls-certs\") pod \"90f20be3-aae7-44c1-9ed6-6aca54210719\" (UID: \"90f20be3-aae7-44c1-9ed6-6aca54210719\") "
Sep 16 19:55:00.226020 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:00.225909    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvd7b\" (UniqueName: \"kubernetes.io/projected/90f20be3-aae7-44c1-9ed6-6aca54210719-kube-api-access-gvd7b\") pod \"90f20be3-aae7-44c1-9ed6-6aca54210719\" (UID: \"90f20be3-aae7-44c1-9ed6-6aca54210719\") "
Sep 16 19:55:00.227802 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:00.227775    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90f20be3-aae7-44c1-9ed6-6aca54210719-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "90f20be3-aae7-44c1-9ed6-6aca54210719" (UID: "90f20be3-aae7-44c1-9ed6-6aca54210719"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:55:00.227903 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:00.227801    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f20be3-aae7-44c1-9ed6-6aca54210719-kube-api-access-gvd7b" (OuterVolumeSpecName: "kube-api-access-gvd7b") pod "90f20be3-aae7-44c1-9ed6-6aca54210719" (UID: "90f20be3-aae7-44c1-9ed6-6aca54210719"). InnerVolumeSpecName "kube-api-access-gvd7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:55:00.326952 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:00.326926    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90f20be3-aae7-44c1-9ed6-6aca54210719-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:55:00.326952 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:00.326951    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvd7b\" (UniqueName: \"kubernetes.io/projected/90f20be3-aae7-44c1-9ed6-6aca54210719-kube-api-access-gvd7b\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:55:01.061151 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:01.061122    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds"
Sep 16 19:55:01.061671 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:01.061122    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds" event={"ID":"90f20be3-aae7-44c1-9ed6-6aca54210719","Type":"ContainerDied","Data":"abe1d7453b7be30d350d574da19f214acee544046d2055554e92688f20a1938f"}
Sep 16 19:55:01.061671 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:01.061266    2578 scope.go:117] "RemoveContainer" containerID="25aa3f45d2f5050f25fa3261449cacb4d3af59916b9af816b35442c1ecbfdee3"
Sep 16 19:55:01.084958 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:01.084931    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds"]
Sep 16 19:55:01.089353 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:01.089329    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-xb2ds"]
Sep 16 19:55:02.217692 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:02.217655    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90f20be3-aae7-44c1-9ed6-6aca54210719" path="/var/lib/kubelet/pods/90f20be3-aae7-44c1-9ed6-6aca54210719/volumes"
Sep 16 19:55:23.428846 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:23.428806    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m"]
Sep 16 19:55:23.431241 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:23.429154    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d128fa42-41f0-4af8-95da-cce98c379a0b" containerName="storage-initializer"
Sep 16 19:55:23.431241 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:23.429170    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d128fa42-41f0-4af8-95da-cce98c379a0b" containerName="storage-initializer"
Sep 16 19:55:23.431241 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:23.429187    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d128fa42-41f0-4af8-95da-cce98c379a0b" containerName="main"
Sep 16 19:55:23.431241 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:23.429193    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d128fa42-41f0-4af8-95da-cce98c379a0b" containerName="main"
Sep 16 19:55:23.431241 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:23.429210    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90f20be3-aae7-44c1-9ed6-6aca54210719" containerName="main"
Sep 16 19:55:23.431241 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:23.429216    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="90f20be3-aae7-44c1-9ed6-6aca54210719" containerName="main"
Sep 16 19:55:23.431241 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:23.429264    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="90f20be3-aae7-44c1-9ed6-6aca54210719" containerName="main"
Sep 16 19:55:23.431241 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:23.429278    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d128fa42-41f0-4af8-95da-cce98c379a0b" containerName="main"
Sep 16 19:55:23.432133 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:23.432113    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m"
Sep 16 19:55:23.436192 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:23.436160    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 16 19:55:23.436337 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:23.436167    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-b6wch\""
Sep 16 19:55:23.436337 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:23.436192    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-krlxx\""
Sep 16 19:55:23.436337 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:23.436192    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 16 19:55:23.436337 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:23.436266    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 16 19:55:23.441611 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:23.441587    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m"]
Sep 16 19:55:23.492243 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:23.492220    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3474f71d-47b6-4376-b963-728fe096fba0-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m\" (UID: \"3474f71d-47b6-4376-b963-728fe096fba0\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m"
Sep 16 19:55:23.492360 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:23.492305    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw2xp\" (UniqueName: \"kubernetes.io/projected/3474f71d-47b6-4376-b963-728fe096fba0-kube-api-access-xw2xp\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m\" (UID: \"3474f71d-47b6-4376-b963-728fe096fba0\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m"
Sep 16 19:55:23.593201 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:23.593169    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw2xp\" (UniqueName: \"kubernetes.io/projected/3474f71d-47b6-4376-b963-728fe096fba0-kube-api-access-xw2xp\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m\" (UID: \"3474f71d-47b6-4376-b963-728fe096fba0\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m"
Sep 16 19:55:23.593333 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:23.593221    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3474f71d-47b6-4376-b963-728fe096fba0-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m\" (UID: \"3474f71d-47b6-4376-b963-728fe096fba0\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m"
Sep 16 19:55:23.595625 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:23.595606    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3474f71d-47b6-4376-b963-728fe096fba0-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m\" (UID: \"3474f71d-47b6-4376-b963-728fe096fba0\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m"
Sep 16 19:55:23.602793 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:23.602768    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw2xp\" (UniqueName: \"kubernetes.io/projected/3474f71d-47b6-4376-b963-728fe096fba0-kube-api-access-xw2xp\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m\" (UID: \"3474f71d-47b6-4376-b963-728fe096fba0\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m"
Sep 16 19:55:23.744574 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:23.744497    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m"
Sep 16 19:55:24.018026 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:24.017997    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m"]
Sep 16 19:55:24.020734 ip-10-0-142-80 kubenswrapper[2578]: W0916 19:55:24.020698    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3474f71d_47b6_4376_b963_728fe096fba0.slice/crio-6c914a7040f2898bc3245102042f64289941ef01dcd9db550ce200ed2b6d0dbd WatchSource:0}: Error finding container 6c914a7040f2898bc3245102042f64289941ef01dcd9db550ce200ed2b6d0dbd: Status 404 returned error can't find the container with id 6c914a7040f2898bc3245102042f64289941ef01dcd9db550ce200ed2b6d0dbd
Sep 16 19:55:24.181175 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:24.181134    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m" event={"ID":"3474f71d-47b6-4376-b963-728fe096fba0","Type":"ContainerStarted","Data":"33406303670ecbddd7f7dd3d90492ac01c4f2f7f46d5a48bc90267b28276251d"}
Sep 16 19:55:24.181374 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:24.181214    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m" event={"ID":"3474f71d-47b6-4376-b963-728fe096fba0","Type":"ContainerStarted","Data":"6c914a7040f2898bc3245102042f64289941ef01dcd9db550ce200ed2b6d0dbd"}
Sep 16 19:55:24.181473 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:24.181372    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m"
Sep 16 19:55:24.212142 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:24.212075    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m" podStartSLOduration=1.212053902 podStartE2EDuration="1.212053902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:55:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:55:24.20864283 +0000 UTC m=+4492.443576273" watchObservedRunningTime="2026-09-16 19:55:24.212053902 +0000 UTC m=+4492.446987339"
Sep 16 19:55:55.189377 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:55:55.189345    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m"
Sep 16 19:56:03.670911 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:56:03.670876    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m"]
Sep 16 19:56:05.189681 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:56:05.189641    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m" podUID="3474f71d-47b6-4376-b963-728fe096fba0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:56:15.189296 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:56:15.189250    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m" podUID="3474f71d-47b6-4376-b963-728fe096fba0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:56:18.690692 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:56:18.690634    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m" podUID="3474f71d-47b6-4376-b963-728fe096fba0" containerName="main" containerID="cri-o://33406303670ecbddd7f7dd3d90492ac01c4f2f7f46d5a48bc90267b28276251d" gracePeriod=45
Sep 16 19:56:25.189645 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:56:25.189601    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m" podUID="3474f71d-47b6-4376-b963-728fe096fba0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:56:25.190032 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:56:25.189717    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m"
Sep 16 19:56:35.188821 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:56:35.188780    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m" podUID="3474f71d-47b6-4376-b963-728fe096fba0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:56:45.189286 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:56:45.189244    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m" podUID="3474f71d-47b6-4376-b963-728fe096fba0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:56:48.926381 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:56:48.926357    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m"
Sep 16 19:56:49.001195 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:56:49.001129    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3474f71d-47b6-4376-b963-728fe096fba0-tls-certs\") pod \"3474f71d-47b6-4376-b963-728fe096fba0\" (UID: \"3474f71d-47b6-4376-b963-728fe096fba0\") "
Sep 16 19:56:49.001195 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:56:49.001167    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xw2xp\" (UniqueName: \"kubernetes.io/projected/3474f71d-47b6-4376-b963-728fe096fba0-kube-api-access-xw2xp\") pod \"3474f71d-47b6-4376-b963-728fe096fba0\" (UID: \"3474f71d-47b6-4376-b963-728fe096fba0\") "
Sep 16 19:56:49.003203 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:56:49.003177    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3474f71d-47b6-4376-b963-728fe096fba0-kube-api-access-xw2xp" (OuterVolumeSpecName: "kube-api-access-xw2xp") pod "3474f71d-47b6-4376-b963-728fe096fba0" (UID: "3474f71d-47b6-4376-b963-728fe096fba0"). InnerVolumeSpecName "kube-api-access-xw2xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:56:49.003203 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:56:49.003190    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3474f71d-47b6-4376-b963-728fe096fba0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3474f71d-47b6-4376-b963-728fe096fba0" (UID: "3474f71d-47b6-4376-b963-728fe096fba0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:56:49.102522 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:56:49.102496    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3474f71d-47b6-4376-b963-728fe096fba0-tls-certs\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:56:49.102522 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:56:49.102520    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw2xp\" (UniqueName: \"kubernetes.io/projected/3474f71d-47b6-4376-b963-728fe096fba0-kube-api-access-xw2xp\") on node \"ip-10-0-142-80.ec2.internal\" DevicePath \"\""
Sep 16 19:56:49.624839 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:56:49.624800    2578 generic.go:356] "Generic (PLEG): container finished" podID="3474f71d-47b6-4376-b963-728fe096fba0" containerID="33406303670ecbddd7f7dd3d90492ac01c4f2f7f46d5a48bc90267b28276251d" exitCode=0
Sep 16 19:56:49.625059 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:56:49.624865    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m"
Sep 16 19:56:49.625059 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:56:49.624900    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m" event={"ID":"3474f71d-47b6-4376-b963-728fe096fba0","Type":"ContainerDied","Data":"33406303670ecbddd7f7dd3d90492ac01c4f2f7f46d5a48bc90267b28276251d"}
Sep 16 19:56:49.625059 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:56:49.624968    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m" event={"ID":"3474f71d-47b6-4376-b963-728fe096fba0","Type":"ContainerDied","Data":"6c914a7040f2898bc3245102042f64289941ef01dcd9db550ce200ed2b6d0dbd"}
Sep 16 19:56:49.625059 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:56:49.624995    2578 scope.go:117] "RemoveContainer" containerID="33406303670ecbddd7f7dd3d90492ac01c4f2f7f46d5a48bc90267b28276251d"
Sep 16 19:56:49.635114 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:56:49.635094    2578 scope.go:117] "RemoveContainer" containerID="33406303670ecbddd7f7dd3d90492ac01c4f2f7f46d5a48bc90267b28276251d"
Sep 16 19:56:49.635416 ip-10-0-142-80 kubenswrapper[2578]: E0916 19:56:49.635395    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33406303670ecbddd7f7dd3d90492ac01c4f2f7f46d5a48bc90267b28276251d\": container with ID starting with 33406303670ecbddd7f7dd3d90492ac01c4f2f7f46d5a48bc90267b28276251d not found: ID does not exist" containerID="33406303670ecbddd7f7dd3d90492ac01c4f2f7f46d5a48bc90267b28276251d"
Sep 16 19:56:49.635517 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:56:49.635423    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33406303670ecbddd7f7dd3d90492ac01c4f2f7f46d5a48bc90267b28276251d"} err="failed to get container status \"33406303670ecbddd7f7dd3d90492ac01c4f2f7f46d5a48bc90267b28276251d\": rpc error: code = NotFound desc = could not find container \"33406303670ecbddd7f7dd3d90492ac01c4f2f7f46d5a48bc90267b28276251d\": container with ID starting with 33406303670ecbddd7f7dd3d90492ac01c4f2f7f46d5a48bc90267b28276251d not found: ID does not exist"
Sep 16 19:56:49.649285 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:56:49.649259    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m"]
Sep 16 19:56:49.654190 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:56:49.654164    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-kdz2m"]
Sep 16 19:56:50.217194 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:56:50.217163    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3474f71d-47b6-4376-b963-728fe096fba0" path="/var/lib/kubelet/pods/3474f71d-47b6-4376-b963-728fe096fba0/volumes"
Sep 16 19:57:21.552370 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:21.552333    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-4bmk9_f68303c7-a114-497f-be72-c30f89d7dd1b/discovery/0.log"
Sep 16 19:57:21.567782 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:21.567737    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7_79e2c88c-45e8-4432-b974-05ea559e2be6/istio-proxy/0.log"
Sep 16 19:57:21.598949 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:21.598922    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7_79e2c88c-45e8-4432-b974-05ea559e2be6/istio-proxy/1.log"
Sep 16 19:57:22.449382 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:22.449346    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-4bmk9_f68303c7-a114-497f-be72-c30f89d7dd1b/discovery/0.log"
Sep 16 19:57:22.466436 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:22.466406    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7_79e2c88c-45e8-4432-b974-05ea559e2be6/istio-proxy/0.log"
Sep 16 19:57:22.496063 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:22.496028    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7_79e2c88c-45e8-4432-b974-05ea559e2be6/istio-proxy/1.log"
Sep 16 19:57:23.856562 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:23.856529    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-jlvf7_5f868101-ea7a-4685-8c91-d67140498fd3/authorino/0.log"
Sep 16 19:57:23.879371 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:23.879332    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-24fl9_4af900d0-1ca8-4ee6-9d51-2774904b4ac1/manager/0.log"
Sep 16 19:57:24.131460 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:24.131380    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-lnwj8_eed3ce9f-0c0b-4559-8eef-71ac47313c8d/manager/0.log"
Sep 16 19:57:33.164807 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:33.164726    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qsdt4_6b56f6c6-d7a4-4ca4-a9f1-d7670c1ee872/global-pull-secret-syncer/0.log"
Sep 16 19:57:33.279772 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:33.279746    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gw4zp_fe4ed2ab-567f-4482-8c96-7293f1c82bb9/konnectivity-agent/0.log"
Sep 16 19:57:33.409939 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:33.409910    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-80.ec2.internal_61224f73c725c86faa0f55f841c5d69e/haproxy/0.log"
Sep 16 19:57:38.170788 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:38.170757    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-jlvf7_5f868101-ea7a-4685-8c91-d67140498fd3/authorino/0.log"
Sep 16 19:57:38.303226 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:38.303199    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-24fl9_4af900d0-1ca8-4ee6-9d51-2774904b4ac1/manager/0.log"
Sep 16 19:57:38.630843 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:38.630809    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-lnwj8_eed3ce9f-0c0b-4559-8eef-71ac47313c8d/manager/0.log"
Sep 16 19:57:40.172440 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:40.172403    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-cjpnz_aeaf5a49-9e3f-46c9-992f-fa4e24a2c06b/cluster-monitoring-operator/0.log"
Sep 16 19:57:40.354663 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:40.354621    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sz8rh_75757666-809b-4bd6-85f5-34118dd9955f/node-exporter/0.log"
Sep 16 19:57:40.380474 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:40.380428    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sz8rh_75757666-809b-4bd6-85f5-34118dd9955f/kube-rbac-proxy/0.log"
Sep 16 19:57:40.409888 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:40.409870    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sz8rh_75757666-809b-4bd6-85f5-34118dd9955f/init-textfile/0.log"
Sep 16 19:57:42.492246 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:42.492209    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-8zxkm_7c12c8f8-59c4-4d2c-ab39-b2199dcfb1d4/networking-console-plugin/0.log"
Sep 16 19:57:43.181962 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:43.181921    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mkvvc/perf-node-gather-daemonset-sj9bd"]
Sep 16 19:57:43.182296 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:43.182279    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3474f71d-47b6-4376-b963-728fe096fba0" containerName="main"
Sep 16 19:57:43.182358 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:43.182297    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3474f71d-47b6-4376-b963-728fe096fba0" containerName="main"
Sep 16 19:57:43.182358 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:43.182352    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3474f71d-47b6-4376-b963-728fe096fba0" containerName="main"
Sep 16 19:57:43.186582 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:43.186558    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-sj9bd"
Sep 16 19:57:43.188903 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:43.188874    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mkvvc\"/\"kube-root-ca.crt\""
Sep 16 19:57:43.189002 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:43.188966    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mkvvc\"/\"default-dockercfg-h7jhd\""
Sep 16 19:57:43.190072 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:43.190035    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mkvvc\"/\"openshift-service-ca.crt\""
Sep 16 19:57:43.197079 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:43.197053    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mkvvc/perf-node-gather-daemonset-sj9bd"]
Sep 16 19:57:43.277718 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:43.277694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5b11837f-317f-453c-b740-010cc030b261-podres\") pod \"perf-node-gather-daemonset-sj9bd\" (UID: \"5b11837f-317f-453c-b740-010cc030b261\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-sj9bd"
Sep 16 19:57:43.277827 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:43.277732    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phjxr\" (UniqueName: \"kubernetes.io/projected/5b11837f-317f-453c-b740-010cc030b261-kube-api-access-phjxr\") pod \"perf-node-gather-daemonset-sj9bd\" (UID: \"5b11837f-317f-453c-b740-010cc030b261\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-sj9bd"
Sep 16 19:57:43.277827 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:43.277758    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b11837f-317f-453c-b740-010cc030b261-sys\") pod \"perf-node-gather-daemonset-sj9bd\" (UID: \"5b11837f-317f-453c-b740-010cc030b261\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-sj9bd"
Sep 16 19:57:43.277827 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:43.277789    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5b11837f-317f-453c-b740-010cc030b261-proc\") pod \"perf-node-gather-daemonset-sj9bd\" (UID: \"5b11837f-317f-453c-b740-010cc030b261\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-sj9bd"
Sep 16 19:57:43.277827 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:43.277815    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5b11837f-317f-453c-b740-010cc030b261-lib-modules\") pod \"perf-node-gather-daemonset-sj9bd\" (UID: \"5b11837f-317f-453c-b740-010cc030b261\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-sj9bd"
Sep 16 19:57:43.378338 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:43.378310    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5b11837f-317f-453c-b740-010cc030b261-lib-modules\") pod \"perf-node-gather-daemonset-sj9bd\" (UID: \"5b11837f-317f-453c-b740-010cc030b261\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-sj9bd"
Sep 16 19:57:43.378446 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:43.378367    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5b11837f-317f-453c-b740-010cc030b261-podres\") pod \"perf-node-gather-daemonset-sj9bd\" (UID: \"5b11837f-317f-453c-b740-010cc030b261\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-sj9bd"
Sep 16 19:57:43.378446 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:43.378422    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phjxr\" (UniqueName: \"kubernetes.io/projected/5b11837f-317f-453c-b740-010cc030b261-kube-api-access-phjxr\") pod \"perf-node-gather-daemonset-sj9bd\" (UID: \"5b11837f-317f-453c-b740-010cc030b261\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-sj9bd"
Sep 16 19:57:43.378547 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:43.378456    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b11837f-317f-453c-b740-010cc030b261-sys\") pod \"perf-node-gather-daemonset-sj9bd\" (UID: \"5b11837f-317f-453c-b740-010cc030b261\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-sj9bd"
Sep 16 19:57:43.378547 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:43.378485    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5b11837f-317f-453c-b740-010cc030b261-proc\") pod \"perf-node-gather-daemonset-sj9bd\" (UID: \"5b11837f-317f-453c-b740-010cc030b261\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-sj9bd"
Sep 16 19:57:43.378547 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:43.378495    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5b11837f-317f-453c-b740-010cc030b261-lib-modules\") pod \"perf-node-gather-daemonset-sj9bd\" (UID: \"5b11837f-317f-453c-b740-010cc030b261\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-sj9bd"
Sep 16 19:57:43.378681 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:43.378557    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5b11837f-317f-453c-b740-010cc030b261-podres\") pod \"perf-node-gather-daemonset-sj9bd\" (UID: \"5b11837f-317f-453c-b740-010cc030b261\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-sj9bd"
Sep 16 19:57:43.378681 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:43.378563    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5b11837f-317f-453c-b740-010cc030b261-proc\") pod \"perf-node-gather-daemonset-sj9bd\" (UID: \"5b11837f-317f-453c-b740-010cc030b261\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-sj9bd"
Sep 16 19:57:43.378681 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:43.378577    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b11837f-317f-453c-b740-010cc030b261-sys\") pod \"perf-node-gather-daemonset-sj9bd\" (UID: \"5b11837f-317f-453c-b740-010cc030b261\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-sj9bd"
Sep 16 19:57:43.388749 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:43.388721    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phjxr\" (UniqueName: \"kubernetes.io/projected/5b11837f-317f-453c-b740-010cc030b261-kube-api-access-phjxr\") pod \"perf-node-gather-daemonset-sj9bd\" (UID: \"5b11837f-317f-453c-b740-010cc030b261\") " pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-sj9bd"
Sep 16 19:57:43.497763 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:43.497695    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-sj9bd"
Sep 16 19:57:43.537366 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:43.537329    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5597478dbc-8t5zp_35cda191-1b89-423c-8366-a88a210a6ffb/console/0.log"
Sep 16 19:57:43.598795 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:43.598768    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-5rf9x_f5ea7094-a8a5-487d-be9d-195bd5c0fa11/download-server/0.log"
Sep 16 19:57:43.773625 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:43.773561    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mkvvc/perf-node-gather-daemonset-sj9bd"]
Sep 16 19:57:43.775090 ip-10-0-142-80 kubenswrapper[2578]: W0916 19:57:43.775060    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5b11837f_317f_453c_b740_010cc030b261.slice/crio-bede241cdb36ae6ed40d34afb575411c26fcaf500fcb5c441f7279bfe1de09bb WatchSource:0}: Error finding container bede241cdb36ae6ed40d34afb575411c26fcaf500fcb5c441f7279bfe1de09bb: Status 404 returned error can't find the container with id bede241cdb36ae6ed40d34afb575411c26fcaf500fcb5c441f7279bfe1de09bb
Sep 16 19:57:43.905448 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:43.905418    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-sj9bd" event={"ID":"5b11837f-317f-453c-b740-010cc030b261","Type":"ContainerStarted","Data":"e21c0210661d3726c87f797f90de1bc57ef396d9229337a0702738b6620655bc"}
Sep 16 19:57:43.905558 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:43.905465    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-sj9bd" event={"ID":"5b11837f-317f-453c-b740-010cc030b261","Type":"ContainerStarted","Data":"bede241cdb36ae6ed40d34afb575411c26fcaf500fcb5c441f7279bfe1de09bb"}
Sep 16 19:57:43.905558 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:43.905536    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-sj9bd"
Sep 16 19:57:43.931152 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:43.931106    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-sj9bd" podStartSLOduration=0.931093379 podStartE2EDuration="931.093379ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:57:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:57:43.928953094 +0000 UTC m=+4632.163886535" watchObservedRunningTime="2026-09-16 19:57:43.931093379 +0000 UTC m=+4632.166026822"
Sep 16 19:57:45.262629 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:45.262594    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-86bz5_309dbcdf-d468-4001-805b-e71bb5bdd775/dns/0.log"
Sep 16 19:57:45.287708 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:45.287675    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-86bz5_309dbcdf-d468-4001-805b-e71bb5bdd775/kube-rbac-proxy/0.log"
Sep 16 19:57:45.474112 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:45.474085    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xbtsz_c3e865df-22be-4129-bbf3-300285c29e69/dns-node-resolver/0.log"
Sep 16 19:57:46.103111 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:46.103074    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-95lgg_6ab1b46d-5a59-42a7-87c4-13c16da817ad/node-ca/0.log"
Sep 16 19:57:47.362259 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:47.362226    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-4bmk9_f68303c7-a114-497f-be72-c30f89d7dd1b/discovery/0.log"
Sep 16 19:57:47.401029 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:47.400996    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7_79e2c88c-45e8-4432-b974-05ea559e2be6/istio-proxy/0.log"
Sep 16 19:57:47.430018 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:47.429996    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7_79e2c88c-45e8-4432-b974-05ea559e2be6/istio-proxy/1.log"
Sep 16 19:57:48.114150 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:48.114118    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-scmml_5dc0ed37-07b5-439b-94cf-1cf3104ad233/serve-healthcheck-canary/0.log"
Sep 16 19:57:48.730177 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:48.730149    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sj4ck_f1156b17-5b7a-48e4-aac5-d0a527142061/kube-rbac-proxy/0.log"
Sep 16 19:57:48.756784 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:48.756755    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sj4ck_f1156b17-5b7a-48e4-aac5-d0a527142061/exporter/0.log"
Sep 16 19:57:48.781917 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:48.781868    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sj4ck_f1156b17-5b7a-48e4-aac5-d0a527142061/extractor/0.log"
Sep 16 19:57:49.921740 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:49.921702    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mkvvc/perf-node-gather-daemonset-sj9bd"
Sep 16 19:57:52.841163 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:52.841083    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-599b79d756-47w5x_cfec1bc7-76c3-4d11-90ea-268fa89169bc/manager/0.log"
Sep 16 19:57:52.969374 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:52.969348    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-4bmk9_f68303c7-a114-497f-be72-c30f89d7dd1b/discovery/0.log"
Sep 16 19:57:53.014643 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:53.014616    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7_79e2c88c-45e8-4432-b974-05ea559e2be6/istio-proxy/0.log"
Sep 16 19:57:53.037412 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:53.037385    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7_79e2c88c-45e8-4432-b974-05ea559e2be6/istio-proxy/1.log"
Sep 16 19:57:53.364244 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:53.364213    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-jv92k_f3ffdc74-be37-40a0-9f72-be40cfe0896b/server/0.log"
Sep 16 19:57:54.513510 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:54.513479    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-4bmk9_f68303c7-a114-497f-be72-c30f89d7dd1b/discovery/0.log"
Sep 16 19:57:54.540468 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:54.540436    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7_79e2c88c-45e8-4432-b974-05ea559e2be6/istio-proxy/0.log"
Sep 16 19:57:54.573500 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:54.573476    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-bj8p7_79e2c88c-45e8-4432-b974-05ea559e2be6/istio-proxy/1.log"
Sep 16 19:57:59.944775 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:59.944709    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-xfz8z_2260af00-cdea-4591-9ef1-cf8631deb0fc/kube-storage-version-migrator-operator/1.log"
Sep 16 19:57:59.946358 ip-10-0-142-80 kubenswrapper[2578]: I0916 19:57:59.946330    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-xfz8z_2260af00-cdea-4591-9ef1-cf8631deb0fc/kube-storage-version-migrator-operator/0.log"