Sep 18 19:36:07.254844 ip-10-0-143-134 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 18 19:36:07.254852 ip-10-0-143-134 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 18 19:36:07.254859 ip-10-0-143-134 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 18 19:36:07.255070 ip-10-0-143-134 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 18 19:36:17.360685 ip-10-0-143-134 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 18 19:36:17.360706 ip-10-0-143-134 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 9f4c5397768f414b90eda8548d414c73 --
Sep 18 19:37:56.784134 ip-10-0-143-134 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 19:37:57.213707 ip-10-0-143-134 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:37:57.213707 ip-10-0-143-134 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 19:37:57.213707 ip-10-0-143-134 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:37:57.213707 ip-10-0-143-134 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 19:37:57.213707 ip-10-0-143-134 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:37:57.215830 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.215744    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 19:37:57.221669 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221653    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:37:57.221669 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221669    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:37:57.221731 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221673    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:37:57.221731 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221676    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:37:57.221731 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221679    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:37:57.221731 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221683    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:37:57.221731 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221686    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:37:57.221731 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221689    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:37:57.221731 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221692    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:37:57.221731 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221696    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:37:57.221731 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221700    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:37:57.221731 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221702    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:37:57.221731 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221705    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:37:57.221731 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221707    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:37:57.221731 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221710    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:37:57.221731 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221713    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:37:57.221731 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221716    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:37:57.221731 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221718    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:37:57.221731 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221721    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:37:57.221731 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221724    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:37:57.221731 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221726    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:37:57.222202 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221729    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:37:57.222202 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221732    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:37:57.222202 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221734    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:37:57.222202 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221737    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:37:57.222202 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221740    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:37:57.222202 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221743    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:37:57.222202 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221745    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:37:57.222202 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221748    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:37:57.222202 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221750    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:37:57.222202 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221752    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:37:57.222202 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221755    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:37:57.222202 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221757    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:37:57.222202 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221760    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:37:57.222202 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221762    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:37:57.222202 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221764    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:37:57.222202 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221767    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:37:57.222202 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221769    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:37:57.222202 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221772    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:37:57.222202 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221774    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:37:57.222202 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221777    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:37:57.222676 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221779    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:37:57.222676 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221789    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:37:57.222676 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221793    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:37:57.222676 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221796    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:37:57.222676 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221799    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:37:57.222676 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221801    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:37:57.222676 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221804    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:37:57.222676 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221806    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:37:57.222676 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221809    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:37:57.222676 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221811    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:37:57.222676 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221813    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:37:57.222676 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221816    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:37:57.222676 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221820    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:37:57.222676 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221824    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:37:57.222676 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221827    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:37:57.222676 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221841    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:37:57.222676 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221847    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:37:57.222676 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221850    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:37:57.222676 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221853    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:37:57.223181 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221856    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:37:57.223181 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221858    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:37:57.223181 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221862    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:37:57.223181 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221865    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:37:57.223181 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221868    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:37:57.223181 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221870    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:37:57.223181 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221873    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:37:57.223181 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221876    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:37:57.223181 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221879    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:37:57.223181 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221881    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:37:57.223181 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221885    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:37:57.223181 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221887    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:37:57.223181 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221891    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:37:57.223181 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221893    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:37:57.223181 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221896    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:37:57.223181 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221898    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:37:57.223181 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221901    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:37:57.223181 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221903    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:37:57.223181 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221906    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:37:57.223181 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221909    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:37:57.223655 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221911    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:37:57.223655 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221914    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:37:57.223655 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221917    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:37:57.223655 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221919    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:37:57.223655 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221922    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:37:57.223655 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221924    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:37:57.223655 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221928    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:37:57.223655 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221931    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:37:57.223655 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221933    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:37:57.223655 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221936    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:37:57.223655 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221938    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:37:57.223655 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221941    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:37:57.223655 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221943    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:37:57.223655 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221946    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:37:57.223655 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221948    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:37:57.223655 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221951    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:37:57.223655 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221953    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:37:57.223655 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221955    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:37:57.223655 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.221958    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:37:57.224460 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224447    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:37:57.224460 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224460    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:37:57.224514 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224464    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:37:57.224514 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224467    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:37:57.224514 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224469    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:37:57.224514 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224472    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:37:57.224514 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224475    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:37:57.224514 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224478    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:37:57.224514 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224480    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:37:57.224514 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224483    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:37:57.224514 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224489    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:37:57.224514 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224492    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:37:57.224514 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224495    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:37:57.224514 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224498    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:37:57.224514 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224500    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:37:57.224514 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224503    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:37:57.224514 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224505    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:37:57.224514 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224508    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:37:57.224514 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224510    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:37:57.224514 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224513    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:37:57.224514 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224516    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:37:57.224514 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224518    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:37:57.225130 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224521    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:37:57.225130 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224524    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:37:57.225130 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224526    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:37:57.225130 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224529    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:37:57.225130 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224531    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:37:57.225130 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224534    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:37:57.225130 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224536    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:37:57.225130 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224539    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:37:57.225130 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224541    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:37:57.225130 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224543    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:37:57.225130 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224546    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:37:57.225130 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224549    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:37:57.225130 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224551    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:37:57.225130 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224554    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:37:57.225130 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224556    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:37:57.225130 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224559    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:37:57.225130 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224562    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:37:57.225130 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224566    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:37:57.225130 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224569    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:37:57.225130 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224571    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:37:57.225945 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224574    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:37:57.225945 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224577    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:37:57.225945 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224580    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:37:57.225945 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224582    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:37:57.225945 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224584    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:37:57.225945 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224587    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:37:57.225945 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224589    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:37:57.225945 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224592    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:37:57.225945 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224594    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:37:57.225945 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224596    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:37:57.225945 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224599    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:37:57.225945 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224617    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:37:57.225945 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224621    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:37:57.225945 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224625    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:37:57.225945 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224628    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:37:57.225945 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224631    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:37:57.225945 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224634    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:37:57.225945 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224637    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:37:57.225945 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224640    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:37:57.225945 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224642    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:37:57.226785 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224645    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:37:57.226785 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224647    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:37:57.226785 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224650    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:37:57.226785 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224652    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:37:57.226785 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224655    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:37:57.226785 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224657    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:37:57.226785 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224659    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:37:57.226785 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224662    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:37:57.226785 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224665    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:37:57.226785 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224667    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:37:57.226785 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224669    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:37:57.226785 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224672    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:37:57.226785 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224675    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:37:57.226785 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224677    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:37:57.226785 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224681    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:37:57.226785 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224685    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:37:57.226785 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224688    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:37:57.226785 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224690    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:37:57.226785 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224693    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:37:57.227614 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224696    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:37:57.227614 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224699    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:37:57.227614 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224702    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:37:57.227614 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224704    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:37:57.227614 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224707    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:37:57.227614 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224709    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:37:57.227614 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224713    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:37:57.227614 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224715    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:37:57.227614 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224718    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:37:57.227614 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224722    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:37:57.227614 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224725    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:37:57.227614 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224728    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:37:57.227614 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224731    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:37:57.227614 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224733    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:37:57.227614 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224736    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:37:57.227614 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224739    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:37:57.227614 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224741    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:37:57.227614 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.224744    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:37:57.227614 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226345    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:37:57.228444 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226358    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:37:57.228444 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226364    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:37:57.228444 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226370    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:37:57.228444 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226375    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:37:57.228444 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226380    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:37:57.228444 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226386    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:37:57.228444 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226390    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:37:57.228444 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226394    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:37:57.228444 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226399    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:37:57.228444 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226403    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:37:57.228444 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226408    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:37:57.228444 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226412    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:37:57.228444 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226416    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:37:57.228444 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226420    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:37:57.228444 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226424    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:37:57.228444 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226429    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:37:57.228444 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226433    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:37:57.228444 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226437    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:37:57.228444 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226441    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:37:57.228444 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226445    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:37:57.228946 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226449    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:37:57.228946 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226453    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:37:57.228946 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226458    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:37:57.228946 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226462    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:37:57.228946 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226466    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:37:57.228946 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226471    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:37:57.228946 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226475    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:37:57.228946 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226488    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:37:57.228946 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226496    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:37:57.228946 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226503    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:37:57.228946 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226509    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:37:57.228946 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226518    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:37:57.228946 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226523    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:37:57.228946 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226528    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:37:57.228946 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226532    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:37:57.228946 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226536    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:37:57.228946 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226540    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:37:57.228946 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226545    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:37:57.228946 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226549    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:37:57.229530 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226553    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:37:57.229530 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226558    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:37:57.229530 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226562    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:37:57.229530 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226568    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:37:57.229530 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226574    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:37:57.229530 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226578    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:37:57.229530 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226582    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:37:57.229530 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226587    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:37:57.229530 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226591    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:37:57.229530 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226595    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:37:57.229530 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226599    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:37:57.229530 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226603    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:37:57.229530 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226607    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:37:57.229530 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226611    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:37:57.229530 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226615    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:37:57.229530 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226619    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:37:57.229530 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226624    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:37:57.229530 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226628    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:37:57.229530 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226633    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:37:57.230009 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226637    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:37:57.230009 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226641    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:37:57.230009 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226645    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:37:57.230009 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226649    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:37:57.230009 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226653    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:37:57.230009 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226657    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:37:57.230009 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226662    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:37:57.230009 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226669    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:37:57.230009 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226673    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:37:57.230009 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226677    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:37:57.230009 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226681    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:37:57.230009 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226686    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:37:57.230009 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226690    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:37:57.230009 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226694    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:37:57.230009 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226698    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:37:57.230009 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226702    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:37:57.230009 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226706    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:37:57.230009 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226710    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:37:57.230009 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226714    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:37:57.230009 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226719    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:37:57.230515 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226723    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:37:57.230515 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226727    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:37:57.230515 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226731    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:37:57.230515 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226737    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:37:57.230515 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226741    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:37:57.230515 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226745    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:37:57.230515 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226750    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:37:57.230515 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226753    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:37:57.230515 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226757    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:37:57.230515 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226762    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:37:57.230515 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226766    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:37:57.230515 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226770    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:37:57.230515 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226774    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:37:57.230515 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226778    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:37:57.230515 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226782    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:37:57.230515 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226786    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:37:57.230515 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226790    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:37:57.230515 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226795    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:37:57.230515 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226799    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:37:57.230515 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.226803    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:37:57.231096 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.226927    2581 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 19:37:57.231096 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.226981    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 19:37:57.231096 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.226996    2581 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 19:37:57.231096 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227003    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 19:37:57.231096 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227009    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 19:37:57.231096 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227014    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 19:37:57.231096 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227025    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 19:37:57.231096 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227032    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 19:37:57.231096 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227037    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 19:37:57.231096 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227042    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 19:37:57.231096 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227047    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 19:37:57.231096 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227052    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 19:37:57.231096 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227057    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 19:37:57.231096 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227062    2581 flags.go:64] FLAG: --cgroup-root=""
Sep 18 19:37:57.231096 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227067    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 19:37:57.231096 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227071    2581 flags.go:64] FLAG: --client-ca-file=""
Sep 18 19:37:57.231096 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227078    2581 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 19:37:57.231096 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227082    2581 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 19:37:57.231096 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227089    2581 flags.go:64] FLAG: --cluster-domain=""
Sep 18 19:37:57.231096 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227093    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 19:37:57.231096 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227098    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 19:37:57.231096 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227103    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 19:37:57.231096 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227108    2581 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 19:37:57.231096 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227114    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 19:37:57.231782 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227119    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 19:37:57.231782 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227124    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 19:37:57.231782 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227129    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 19:37:57.231782 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227134    2581 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 19:37:57.231782 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227139    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 19:37:57.231782 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227144    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 19:37:57.231782 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227149    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 19:37:57.231782 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227154    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 19:37:57.231782 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227160    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 19:37:57.231782 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227166    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 19:37:57.231782 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227170    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 19:37:57.231782 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227175    2581 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 19:37:57.231782 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227179    2581 flags.go:64] FLAG: --enable-server="true"
Sep 18 19:37:57.231782 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227184    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 19:37:57.231782 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227191    2581 flags.go:64] FLAG: --event-burst="100"
Sep 18 19:37:57.231782 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227195    2581 flags.go:64] FLAG: --event-qps="50"
Sep 18 19:37:57.231782 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227200    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 19:37:57.231782 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227205    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 19:37:57.231782 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227209    2581 flags.go:64] FLAG: --eviction-hard=""
Sep 18 19:37:57.231782 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227215    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 19:37:57.231782 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227220    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 19:37:57.231782 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227225    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 19:37:57.231782 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227229    2581 flags.go:64] FLAG: --eviction-soft=""
Sep 18 19:37:57.231782 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227235    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 19:37:57.231782 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227240    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 19:37:57.232417 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227246    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 19:37:57.232417 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227251    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 19:37:57.232417 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227255    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 19:37:57.232417 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227260    2581 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 19:37:57.232417 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227264    2581 flags.go:64] FLAG: --feature-gates=""
Sep 18 19:37:57.232417 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227276    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 19:37:57.232417 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227281    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 19:37:57.232417 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227285    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 19:37:57.232417 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227290    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 19:37:57.232417 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227295    2581 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 19:37:57.232417 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227300    2581 flags.go:64] FLAG: --help="false"
Sep 18 19:37:57.232417 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227304    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-143-134.ec2.internal"
Sep 18 19:37:57.232417 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227310    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 19:37:57.232417 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227316    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 19:37:57.232417 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227321    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 19:37:57.232417 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227326    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 19:37:57.232417 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227332    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 19:37:57.232417 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227337    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 19:37:57.232417 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227341    2581 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 19:37:57.232417 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227346    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 19:37:57.232417 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227350    2581 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 19:37:57.232417 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227355    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 19:37:57.232417 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227360    2581 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 19:37:57.232417 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227364    2581 flags.go:64] FLAG: --kube-reserved=""
Sep 18 19:37:57.233243 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227370    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 19:37:57.233243 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227374    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 19:37:57.233243 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227379    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 19:37:57.233243 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227384    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 19:37:57.233243 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227388    2581 flags.go:64] FLAG: --lock-file=""
Sep 18 19:37:57.233243 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227393    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 19:37:57.233243 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227397    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 19:37:57.233243 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227402    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 19:37:57.233243 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227417    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 19:37:57.233243 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227422    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 19:37:57.233243 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227427    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 19:37:57.233243 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227431    2581 flags.go:64] FLAG: --logging-format="text"
Sep 18 19:37:57.233243 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227436    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 19:37:57.233243 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227442    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 19:37:57.233243 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227447    2581 flags.go:64] FLAG: --manifest-url=""
Sep 18 19:37:57.233243 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227452    2581 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 19:37:57.233243 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227458    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 19:37:57.233243 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227464    2581 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 19:37:57.233243 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227470    2581 flags.go:64] FLAG: --max-pods="110"
Sep 18 19:37:57.233243 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227474    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 19:37:57.233243 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227479    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 19:37:57.233243 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227483    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 19:37:57.233243 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227488    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 19:37:57.233243 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227492    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 19:37:57.233243 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227497    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 19:37:57.233882 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227501    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 19:37:57.233882 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227512    2581 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 19:37:57.233882 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227517    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 19:37:57.233882 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227522    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 19:37:57.233882 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227526    2581 flags.go:64] FLAG: --pod-cidr=""
Sep 18 19:37:57.233882 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227531    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 19:37:57.233882 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227540    2581 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 19:37:57.233882 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227545    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 19:37:57.233882 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227550    2581 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 19:37:57.233882 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227554    2581 flags.go:64] FLAG: --port="10250"
Sep 18 19:37:57.233882 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227559    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 19:37:57.233882 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227566    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f22c2b0b8c773700"
Sep 18 19:37:57.233882 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227571    2581 flags.go:64] FLAG: --qos-reserved=""
Sep 18 19:37:57.233882 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227576    2581 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 19:37:57.233882 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227580    2581 flags.go:64] FLAG: --register-node="true"
Sep 18 19:37:57.233882 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227585    2581 flags.go:64] FLAG: --register-with-taints=""
Sep 18 19:37:57.233882 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227590    2581 flags.go:64] FLAG: --registry-burst="10"
Sep 18 19:37:57.233882 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227596    2581 flags.go:64] FLAG: --registry-qps="5"
Sep 18 19:37:57.233882 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227601    2581 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 19:37:57.233882 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227605    2581 flags.go:64] FLAG: --reserved-memory=""
Sep 18 19:37:57.233882 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227611    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 19:37:57.233882 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227618    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 19:37:57.233882 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227622    2581 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 19:37:57.233882 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227627    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 19:37:57.233882 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227631    2581 flags.go:64] FLAG: --runonce="false"
Sep 18 19:37:57.234478 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227635    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 19:37:57.234478 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227640    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 19:37:57.234478 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227645    2581 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 19:37:57.234478 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227650    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 19:37:57.234478 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227654    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 19:37:57.234478 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227659    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 19:37:57.234478 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227664    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 19:37:57.234478 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227668    2581 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 19:37:57.234478 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227673    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 19:37:57.234478 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227677    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 19:37:57.234478 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227682    2581 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 19:37:57.234478 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227686    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 19:37:57.234478 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227691    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 19:37:57.234478 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227696    2581 flags.go:64] FLAG: --system-cgroups=""
Sep 18 19:37:57.234478 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227700    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 19:37:57.234478 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227709    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 19:37:57.234478 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227713    2581 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 19:37:57.234478 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227718    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 19:37:57.234478 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227724    2581 flags.go:64] FLAG: --tls-min-version=""
Sep 18 19:37:57.234478 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227728    2581 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 19:37:57.234478 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227733    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 19:37:57.234478 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227737    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 19:37:57.234478 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227742    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 19:37:57.234478 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227746    2581 flags.go:64] FLAG: --v="2"
Sep 18 19:37:57.234478 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227752    2581 flags.go:64] FLAG: --version="false"
Sep 18 19:37:57.235097 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227759    2581 flags.go:64] FLAG: --vmodule=""
Sep 18 19:37:57.235097 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227765    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 19:37:57.235097 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.227770    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 19:37:57.235097 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.227965    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:37:57.235097 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.227973    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:37:57.235097 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.227979    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:37:57.235097 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.227984    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:37:57.235097 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.227990    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:37:57.235097 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.227996    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:37:57.235097 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228001    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:37:57.235097 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228005    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:37:57.235097 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228010    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:37:57.235097 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228014    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:37:57.235097 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228018    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:37:57.235097 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228023    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:37:57.235097 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228027    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:37:57.235097 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228031    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:37:57.235097 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228035    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:37:57.235097 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228039    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:37:57.235097 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228043    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:37:57.235575 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228047    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:37:57.235575 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228052    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:37:57.235575 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228056    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:37:57.235575 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228060    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:37:57.235575 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228064    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:37:57.235575 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228069    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:37:57.235575 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228073    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:37:57.235575 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228077    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:37:57.235575 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228082    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:37:57.235575 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228086    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:37:57.235575 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228090    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:37:57.235575 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228094    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:37:57.235575 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228098    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:37:57.235575 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228103    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:37:57.235575 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228107    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:37:57.235575 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228113    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:37:57.235575 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228117    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:37:57.235575 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228121    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:37:57.235575 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228127    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:37:57.236132 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228131    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:37:57.236132 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228135    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:37:57.236132 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228139    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:37:57.236132 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228143    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:37:57.236132 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228147    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:37:57.236132 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228151    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:37:57.236132 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228155    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:37:57.236132 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228159    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:37:57.236132 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228162    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:37:57.236132 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228167    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:37:57.236132 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228171    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:37:57.236132 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228175    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:37:57.236132 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228179    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:37:57.236132 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228183    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:37:57.236132 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228187    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:37:57.236132 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228191    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:37:57.236132 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228195    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:37:57.236132 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228199    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:37:57.236132 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228203    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:37:57.236132 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228208    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:37:57.236647 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228212    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:37:57.236647 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228217    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:37:57.236647 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228221    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:37:57.236647 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228225    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:37:57.236647 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228229    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:37:57.236647 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228233    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:37:57.236647 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228237    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:37:57.236647 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228242    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:37:57.236647 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228249    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:37:57.236647 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228253    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:37:57.236647 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228257    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:37:57.236647 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228261    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:37:57.236647 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228265    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:37:57.236647 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228269    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:37:57.236647 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228273    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:37:57.236647 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228278    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:37:57.236647 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228282    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:37:57.236647 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228286    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:37:57.236647 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228290    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:37:57.236647 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228295    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:37:57.237237 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228299    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:37:57.237237 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228303    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:37:57.237237 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228308    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:37:57.237237 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228312    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:37:57.237237 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228316    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:37:57.237237 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228320    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:37:57.237237 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228324    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:37:57.237237 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228328    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:37:57.237237 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228332    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:37:57.237237 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228336    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:37:57.237237 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228340    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:37:57.237237 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228344    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:37:57.237237 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228348    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:37:57.237237 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228353    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:37:57.237237 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228357    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:37:57.237237 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228363    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:37:57.237237 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228368    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:37:57.237237 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228373    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:37:57.237237 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228378    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:37:57.237697 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228383    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:37:57.237697 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228389    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:37:57.237697 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228393    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:37:57.237697 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.228399    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:37:57.237697 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.229187    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:37:57.237697 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.237454    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 19:37:57.237697 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.237582    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 19:37:57.238508 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.238498    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 19:37:57.238537 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.238509    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 19:37:57.238582 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238574    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:37:57.238582 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238582    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:37:57.238633 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238587    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:37:57.238633 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238592    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:37:57.238633 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238596    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:37:57.238633 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238600    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:37:57.238633 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238603    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:37:57.238633 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238606    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:37:57.238633 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238609    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:37:57.238633 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238612    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:37:57.238633 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238615    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:37:57.238633 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238617    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:37:57.238633 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238620    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:37:57.238633 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238622    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:37:57.238633 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238625    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:37:57.238633 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238627    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:37:57.238633 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238630    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:37:57.238633 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238632    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:37:57.238633 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238635    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:37:57.238633 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238638    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:37:57.238633 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238641    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:37:57.239114 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238644    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:37:57.239114 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238647    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:37:57.239114 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238649    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:37:57.239114 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238652    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:37:57.239114 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238654    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:37:57.239114 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238656    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:37:57.239114 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238659    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:37:57.239114 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238661    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:37:57.239114 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238664    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:37:57.239114 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238667    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:37:57.239114 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238670    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:37:57.239114 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238672    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:37:57.239114 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238675    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:37:57.239114 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238677    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:37:57.239114 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238680    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:37:57.239114 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238682    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:37:57.239114 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238685    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:37:57.239114 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238687    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:37:57.239114 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238689    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:37:57.239114 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238692    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:37:57.239593 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238694    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:37:57.239593 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238697    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:37:57.239593 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238699    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:37:57.239593 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238701    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:37:57.239593 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238704    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:37:57.239593 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238706    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:37:57.239593 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238709    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:37:57.239593 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238711    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:37:57.239593 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238714    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:37:57.239593 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238716    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:37:57.239593 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238719    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:37:57.239593 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238722    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:37:57.239593 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238725    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:37:57.239593 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238728    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:37:57.239593 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238730    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:37:57.239593 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238733    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:37:57.239593 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238735    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:37:57.239593 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238738    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:37:57.239593 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238741    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:37:57.239593 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238743    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:37:57.240120 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238746    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:37:57.240120 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238749    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:37:57.240120 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238752    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:37:57.240120 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238754    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:37:57.240120 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238758    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:37:57.240120 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238761    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:37:57.240120 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238764    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:37:57.240120 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238767    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:37:57.240120 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238770    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:37:57.240120 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238772    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:37:57.240120 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238775    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:37:57.240120 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238777    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:37:57.240120 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238780    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:37:57.240120 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238783    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:37:57.240120 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238785    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:37:57.240120 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238788    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:37:57.240120 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238790    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:37:57.240120 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238793    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:37:57.240120 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238796    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:37:57.240568 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238798    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:37:57.240568 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238801    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:37:57.240568 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238804    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:37:57.240568 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238806    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:37:57.240568 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238809    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:37:57.240568 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238812    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:37:57.240568 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238814    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:37:57.240568 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238817    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:37:57.240568 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238819    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:37:57.240568 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238822    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:37:57.240568 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238824    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:37:57.240568 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238827    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:37:57.240568 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238829    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:37:57.240568 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238847    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:37:57.240568 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238850    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:37:57.240568 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238854    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:37:57.240568 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238856    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:37:57.240568 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238859    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:37:57.240568 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238862    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:37:57.241052 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.238867    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:37:57.241052 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238976    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:37:57.241052 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238982    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:37:57.241052 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238985    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:37:57.241052 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238989    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:37:57.241052 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238993    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:37:57.241052 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238996    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:37:57.241052 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.238999    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:37:57.241052 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239002    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:37:57.241052 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239005    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:37:57.241052 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239008    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:37:57.241052 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239010    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:37:57.241052 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239013    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:37:57.241052 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239015    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:37:57.241052 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239019    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:37:57.241417 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239021    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:37:57.241417 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239024    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:37:57.241417 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239027    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:37:57.241417 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239030    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:37:57.241417 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239032    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:37:57.241417 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239035    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:37:57.241417 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239038    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:37:57.241417 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239041    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:37:57.241417 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239043    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:37:57.241417 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239046    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:37:57.241417 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239048    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:37:57.241417 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239051    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:37:57.241417 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239053    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:37:57.241417 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239056    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:37:57.241417 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239059    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:37:57.241417 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239061    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:37:57.241417 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239065    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:37:57.241417 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239068    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:37:57.241417 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239071    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:37:57.241895 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239074    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:37:57.241895 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239077    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:37:57.241895 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239080    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:37:57.241895 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239083    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:37:57.241895 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239086    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:37:57.241895 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239088    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:37:57.241895 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239091    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:37:57.241895 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239093    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:37:57.241895 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239096    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:37:57.241895 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239098    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:37:57.241895 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239101    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:37:57.241895 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239103    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:37:57.241895 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239106    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:37:57.241895 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239108    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:37:57.241895 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239111    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:37:57.241895 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239114    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:37:57.241895 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239118    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:37:57.241895 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239120    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:37:57.241895 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239123    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:37:57.242344 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239125    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:37:57.242344 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239128    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:37:57.242344 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239131    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:37:57.242344 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239133    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:37:57.242344 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239136    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:37:57.242344 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239138    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:37:57.242344 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239142    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:37:57.242344 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239145    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:37:57.242344 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239147    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:37:57.242344 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239151    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:37:57.242344 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239153    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:37:57.242344 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239156    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:37:57.242344 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239158    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:37:57.242344 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239161    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:37:57.242344 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239163    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:37:57.242344 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239166    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:37:57.242344 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239168    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:37:57.242344 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239171    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:37:57.242344 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239173    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:37:57.242787 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239175    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:37:57.242787 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239178    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:37:57.242787 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239181    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:37:57.242787 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239183    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:37:57.242787 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239185    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:37:57.242787 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239188    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:37:57.242787 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239190    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:37:57.242787 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239193    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:37:57.242787 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239195    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:37:57.242787 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239198    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:37:57.242787 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239200    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:37:57.242787 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239203    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:37:57.242787 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239205    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:37:57.242787 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239208    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:37:57.242787 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239210    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:37:57.242787 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239213    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:37:57.242787 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239215    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:37:57.242787 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239218    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:37:57.242787 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239220    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:37:57.242787 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239223    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:37:57.243312 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239225    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:37:57.243312 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239228    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:37:57.243312 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239230    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:37:57.243312 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239233    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:37:57.243312 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239236    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:37:57.243312 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239238    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:37:57.243312 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239241    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:37:57.243312 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:57.239243    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:37:57.243312 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.239248    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:37:57.243312 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.239355    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 19:37:57.243312 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.241865    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 19:37:57.243312 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.242993    2581 server.go:1019] "Starting client certificate rotation"
Sep 18 19:37:57.243312 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.243114    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:37:57.243312 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.243151    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:37:57.246154 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.246141    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:37:57.247292 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.247275    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:37:57.260939 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.260914    2581 log.go:25] "Validated CRI v1 runtime API"
Sep 18 19:37:57.262759 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.262741    2581 log.go:25] "Validated CRI v1 image API"
Sep 18 19:37:57.263647 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.263633    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 19:37:57.268518 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.268500    2581 fs.go:135] Filesystem UUIDs: map[1f10a683-65d8-4e87-b84b-89ff8b76e732:/dev/nvme0n1p3 70a804ba-5d55-433e-86de-7580b5c92667:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 18 19:37:57.268574 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.268518    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 19:37:57.273378 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.273273    2581 manager.go:216] Machine: {Timestamp:2026-09-18 19:37:57.27200353 +0000 UTC m=+0.374291849 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100550 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2831e783ceb062f774d2ad6585af79 SystemUUID:ec2831e7-83ce-b062-f774-d2ad6585af79 BootID:9f4c5397-768f-414b-90ed-a8548d414c73 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e1:36:0e:dd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e1:36:0e:dd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:52:06:66:e5:4d:61 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 19:37:57.273378 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.273377    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 19:37:57.273489 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.273478    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 19:37:57.275908 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.275889    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:37:57.276713 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.276689    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 19:37:57.276858 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.276717    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-134.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 19:37:57.277466 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.277456    2581 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 19:37:57.277502 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.277469    2581 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 19:37:57.277502 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.277476    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:37:57.277502 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.277490    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:37:57.279272 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.279262    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 19:37:57.284303 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.284292    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:37:57.284412 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.284403    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 19:37:57.284450 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.284433    2581 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 19:37:57.284450 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.284441    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 19:37:57.284510 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.284455    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 19:37:57.284510 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.284464    2581 kubelet.go:398] "Adding apiserver pod source"
Sep 18 19:37:57.284510 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.284473    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 19:37:57.284920 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.284908    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:37:57.284974 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.284931    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:37:57.287719 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.287705    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 19:37:57.290290 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.290272    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 19:37:57.290362 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.290303    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 19:37:57.291750 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.291738    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 19:37:57.291792 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.291755    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 19:37:57.291792 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.291761    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 19:37:57.291792 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.291766    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 19:37:57.291792 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.291774    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 19:37:57.291792 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.291780    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 19:37:57.291792 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.291793    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 19:37:57.291977 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.291798    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 19:37:57.291977 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.291804    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 19:37:57.291977 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.291809    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 19:37:57.291977 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.291822    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 19:37:57.291977 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.291827    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 19:37:57.292741 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.292730    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 19:37:57.292741 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.292741    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 19:37:57.295950 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.295918    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-134.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 19:37:57.295950 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:57.295937    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-134.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 19:37:57.296058 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:57.295966    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 19:37:57.296912 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.296899    2581 server.go:1268] "Started kubelet"
Sep 18 19:37:57.297004 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.296977    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 19:37:57.297004 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.296984    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 19:37:57.297106 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.297032    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 19:37:57.297643 ip-10-0-143-134 systemd[1]: Started Kubernetes Kubelet.
Sep 18 19:37:57.301385 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.301359    2581 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 19:37:57.302226 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.302207    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 19:37:57.303281 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:57.303262    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 19:37:57.305299 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:57.304159    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-134.ec2.internal.18d6817d596276d4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-134.ec2.internal,UID:ip-10-0-143-134.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-134.ec2.internal,},FirstTimestamp:2026-09-18 19:37:57.296871124 +0000 UTC m=+0.399159442,LastTimestamp:2026-09-18 19:37:57.296871124 +0000 UTC m=+0.399159442,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-134.ec2.internal,}"
Sep 18 19:37:57.312093 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.312074    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 19:37:57.312779 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.312763    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 19:37:57.313004 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:57.312986    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-134.ec2.internal\" not found"
Sep 18 19:37:57.313601 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.313583    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 19:37:57.313601 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.313589    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 19:37:57.313721 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.313609    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 19:37:57.313721 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.313609    2581 factory.go:55] Registering systemd factory
Sep 18 19:37:57.313721 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.313627    2581 factory.go:223] Registration of the systemd container factory successfully
Sep 18 19:37:57.313721 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.313687    2581 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 19:37:57.313721 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.313693    2581 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 19:37:57.313987 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.313893    2581 factory.go:153] Registering CRI-O factory
Sep 18 19:37:57.313987 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.313904    2581 factory.go:223] Registration of the crio container factory successfully
Sep 18 19:37:57.313987 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.313947    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 19:37:57.313987 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.313959    2581 factory.go:103] Registering Raw factory
Sep 18 19:37:57.313987 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.313970    2581 manager.go:1196] Started watching for new ooms in manager
Sep 18 19:37:57.314264 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.314252    2581 manager.go:349] Starting recovery of all containers
Sep 18 19:37:57.322318 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.322301    2581 manager.go:354] Recovery completed
Sep 18 19:37:57.322777 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:57.322748    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-134.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 19:37:57.322899 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:57.322876    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 19:37:57.325354 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.325334    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s46nc"
Sep 18 19:37:57.326322 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.326309    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:37:57.328706 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.328693    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-134.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:37:57.328764 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.328721    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-134.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:37:57.328764 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.328731    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-134.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:37:57.329349 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.329337    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:37:57.329683 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.329669    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 19:37:57.329683 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.329679    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 19:37:57.329802 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.329695    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:37:57.330025 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.330011    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-134.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:37:57.330069 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.330038    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-134.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:37:57.330069 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.330050    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-134.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:37:57.330652 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:57.330591    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-134.ec2.internal.18d6817d5b483f40  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-134.ec2.internal,UID:ip-10-0-143-134.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-134.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-134.ec2.internal,},FirstTimestamp:2026-09-18 19:37:57.328707392 +0000 UTC m=+0.430995710,LastTimestamp:2026-09-18 19:37:57.328707392 +0000 UTC m=+0.430995710,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-134.ec2.internal,}"
Sep 18 19:37:57.331694 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.331682    2581 policy_none.go:49] "None policy: Start"
Sep 18 19:37:57.331740 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.331698    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 19:37:57.331740 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.331708    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 19:37:57.333143 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.333126    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s46nc"
Sep 18 19:37:57.335171 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.335159    2581 policy_none.go:47] "Start"
Sep 18 19:37:57.340937 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:57.340866    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-134.ec2.internal.18d6817d5b4888b1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-134.ec2.internal,UID:ip-10-0-143-134.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-134.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-134.ec2.internal,},FirstTimestamp:2026-09-18 19:37:57.328726193 +0000 UTC m=+0.431014511,LastTimestamp:2026-09-18 19:37:57.328726193 +0000 UTC m=+0.431014511,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-134.ec2.internal,}"
Sep 18 19:37:57.342981 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:57.342910    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-134.ec2.internal.18d6817d5b48ab45  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-134.ec2.internal,UID:ip-10-0-143-134.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-134.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-134.ec2.internal,},FirstTimestamp:2026-09-18 19:37:57.328735045 +0000 UTC m=+0.431023364,LastTimestamp:2026-09-18 19:37:57.328735045 +0000 UTC m=+0.431023364,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-134.ec2.internal,}"
Sep 18 19:37:57.384365 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.384350    2581 manager.go:337] "Starting Device Plugin manager"
Sep 18 19:37:57.400988 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:57.384393    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 19:37:57.400988 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.384402    2581 server.go:87] "Starting device plugin registration server"
Sep 18 19:37:57.400988 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.384653    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 19:37:57.400988 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.384668    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 19:37:57.400988 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.384758    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 19:37:57.400988 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.384879    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 19:37:57.400988 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.384888    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 19:37:57.400988 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:57.385434    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 19:37:57.400988 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:57.385481    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-134.ec2.internal\" not found"
Sep 18 19:37:57.425245 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.425168    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 19:37:57.426493 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.426479    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 19:37:57.426562 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.426531    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 19:37:57.426562 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.426554    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 19:37:57.426658 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:57.426597    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 19:37:57.429093 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.429062    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:37:57.485732 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.485711    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:37:57.486556 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.486542    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-134.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:37:57.486616 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.486569    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-134.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:37:57.486616 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.486581    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-134.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:37:57.486616 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.486609    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-134.ec2.internal"
Sep 18 19:37:57.501335 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.501317    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-134.ec2.internal"
Sep 18 19:37:57.501396 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:57.501337    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-134.ec2.internal\": node \"ip-10-0-143-134.ec2.internal\" not found"
Sep 18 19:37:57.526942 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.526922    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-134.ec2.internal"]
Sep 18 19:37:57.527018 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.526977    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:37:57.528306 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.528294    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-134.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:37:57.528360 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.528319    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-134.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:37:57.528360 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.528328    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-134.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:37:57.530510 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.530499    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:37:57.530642 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.530627    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.ec2.internal"
Sep 18 19:37:57.530726 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.530661    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:37:57.531666 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.531650    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-134.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:37:57.531758 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.531676    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-134.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:37:57.531758 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.531692    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-134.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:37:57.531758 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.531712    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-134.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:37:57.531758 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.531735    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-134.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:37:57.531758 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.531750    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-134.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:37:57.533826 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.533810    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.ec2.internal"
Sep 18 19:37:57.533913 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.533854    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:37:57.534538 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.534518    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-134.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:37:57.534609 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.534543    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-134.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:37:57.534609 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.534552    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-134.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:37:57.545276 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:57.545259    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-134.ec2.internal\" not found"
Sep 18 19:37:57.558437 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:57.558422    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-134.ec2.internal\" not found" node="ip-10-0-143-134.ec2.internal"
Sep 18 19:37:57.562929 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:57.562913    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-134.ec2.internal\" not found" node="ip-10-0-143-134.ec2.internal"
Sep 18 19:37:57.615753 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.615729    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/18134cd66d519f96be48d8125235fba9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-134.ec2.internal\" (UID: \"18134cd66d519f96be48d8125235fba9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.ec2.internal"
Sep 18 19:37:57.615818 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.615787    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18134cd66d519f96be48d8125235fba9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-134.ec2.internal\" (UID: \"18134cd66d519f96be48d8125235fba9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.ec2.internal"
Sep 18 19:37:57.615818 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.615804    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dff242b51bfc62e6a56c5d2723611cd5-config\") pod \"kube-apiserver-proxy-ip-10-0-143-134.ec2.internal\" (UID: \"dff242b51bfc62e6a56c5d2723611cd5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.ec2.internal"
Sep 18 19:37:57.645665 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:57.645645    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-134.ec2.internal\" not found"
Sep 18 19:37:57.716441 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.716378    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/18134cd66d519f96be48d8125235fba9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-134.ec2.internal\" (UID: \"18134cd66d519f96be48d8125235fba9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.ec2.internal"
Sep 18 19:37:57.716441 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.716331    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/18134cd66d519f96be48d8125235fba9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-134.ec2.internal\" (UID: \"18134cd66d519f96be48d8125235fba9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.ec2.internal"
Sep 18 19:37:57.716566 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.716454    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18134cd66d519f96be48d8125235fba9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-134.ec2.internal\" (UID: \"18134cd66d519f96be48d8125235fba9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.ec2.internal"
Sep 18 19:37:57.716566 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.716424    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18134cd66d519f96be48d8125235fba9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-134.ec2.internal\" (UID: \"18134cd66d519f96be48d8125235fba9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.ec2.internal"
Sep 18 19:37:57.716566 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.716479    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dff242b51bfc62e6a56c5d2723611cd5-config\") pod \"kube-apiserver-proxy-ip-10-0-143-134.ec2.internal\" (UID: \"dff242b51bfc62e6a56c5d2723611cd5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.ec2.internal"
Sep 18 19:37:57.716566 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.716501    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dff242b51bfc62e6a56c5d2723611cd5-config\") pod \"kube-apiserver-proxy-ip-10-0-143-134.ec2.internal\" (UID: \"dff242b51bfc62e6a56c5d2723611cd5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.ec2.internal"
Sep 18 19:37:57.746468 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:57.746450    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-134.ec2.internal\" not found"
Sep 18 19:37:57.847138 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:57.847107    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-134.ec2.internal\" not found"
Sep 18 19:37:57.862262 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.862244    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.ec2.internal"
Sep 18 19:37:57.866107 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:57.866090    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.ec2.internal"
Sep 18 19:37:57.947959 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:57.947933    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-134.ec2.internal\" not found"
Sep 18 19:37:58.048497 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:58.048439    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-134.ec2.internal\" not found"
Sep 18 19:37:58.148986 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:58.148966    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-134.ec2.internal\" not found"
Sep 18 19:37:58.163081 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:58.163057    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:37:58.242986 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:58.242961    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 19:37:58.243377 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:58.243115    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 19:37:58.243377 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:58.243121    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 19:37:58.249248 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:58.249225    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-134.ec2.internal\" not found"
Sep 18 19:37:58.341267 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:58.341202    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 19:32:58 +0000 UTC" deadline="2028-06-15 07:26:15.981361522 +0000 UTC"
Sep 18 19:37:58.341267 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:58.341231    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15251h48m17.640133484s"
Sep 18 19:37:58.349306 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:58.349279    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-134.ec2.internal\" not found"
Sep 18 19:37:58.407062 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:58.406882    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18134cd66d519f96be48d8125235fba9.slice/crio-0a63803f40e35049745a5613024dbc4cdd5644c51ba180061273c3e00ee7b1a7 WatchSource:0}: Error finding container 0a63803f40e35049745a5613024dbc4cdd5644c51ba180061273c3e00ee7b1a7: Status 404 returned error can't find the container with id 0a63803f40e35049745a5613024dbc4cdd5644c51ba180061273c3e00ee7b1a7
Sep 18 19:37:58.407301 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:37:58.407282    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff242b51bfc62e6a56c5d2723611cd5.slice/crio-1153fb2868f49b982896f82fe47ffe86d56fbf063d3db010f0520551ff6a9ca9 WatchSource:0}: Error finding container 1153fb2868f49b982896f82fe47ffe86d56fbf063d3db010f0520551ff6a9ca9: Status 404 returned error can't find the container with id 1153fb2868f49b982896f82fe47ffe86d56fbf063d3db010f0520551ff6a9ca9
Sep 18 19:37:58.408361 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:58.408349    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:37:58.409497 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:58.409473    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-134.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:37:58.409559 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:58.409503    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-134.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:37:58.409559 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:58.409514    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-134.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:37:58.413204 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:58.413083    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:37:58.430478 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:58.430437    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.ec2.internal" event={"ID":"18134cd66d519f96be48d8125235fba9","Type":"ContainerStarted","Data":"0a63803f40e35049745a5613024dbc4cdd5644c51ba180061273c3e00ee7b1a7"}
Sep 18 19:37:58.431491 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:58.431469    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.ec2.internal" event={"ID":"dff242b51bfc62e6a56c5d2723611cd5","Type":"ContainerStarted","Data":"1153fb2868f49b982896f82fe47ffe86d56fbf063d3db010f0520551ff6a9ca9"}
Sep 18 19:37:58.449632 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:58.449613    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-134.ec2.internal\" not found"
Sep 18 19:37:58.550086 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:58.550055    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-134.ec2.internal\" not found"
Sep 18 19:37:58.557368 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:58.557347    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:37:58.650693 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:58.650627    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-134.ec2.internal\" not found"
Sep 18 19:37:58.751245 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:58.751217    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-134.ec2.internal\" not found"
Sep 18 19:37:58.812292 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:58.812272    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:37:58.813716 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:58.813700    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.ec2.internal"
Sep 18 19:37:58.824981 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:58.824956    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:37:58.826108 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:58.826094    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.ec2.internal"
Sep 18 19:37:58.831681 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:58.831667    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:37:59.286211 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.285989    2581 apiserver.go:52] "Watching apiserver"
Sep 18 19:37:59.294672 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.294637    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 19:37:59.295212 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.295188    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-7nrfp","openshift-multus/multus-additional-cni-plugins-mzsk4","openshift-multus/network-metrics-daemon-s9j6n","openshift-network-diagnostics/network-check-target-67bgb","kube-system/konnectivity-agent-cg6mk","openshift-cluster-node-tuning-operator/tuned-lld5l","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.ec2.internal","openshift-multus/multus-4j6th","openshift-network-operator/iptables-alerter-s2vsh","openshift-ovn-kubernetes/ovnkube-node-2wj49","kube-system/global-pull-secret-syncer-2p2bp","kube-system/kube-apiserver-proxy-ip-10-0-143-134.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jb5","openshift-dns/node-resolver-w9dlm"]
Sep 18 19:37:59.298511 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.298483    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.300560 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.300534    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jb5"
Sep 18 19:37:59.301314 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.301292    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 19:37:59.301565 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.301542    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 19:37:59.302872 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.302497    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 19:37:59.302872 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.302526    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 19:37:59.302872 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.302547    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 19:37:59.302872 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.302675    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hmlj7\""
Sep 18 19:37:59.302872 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.302690    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 19:37:59.302872 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.302739    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 19:37:59.302872 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.302764    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 19:37:59.302872 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.302844    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 19:37:59.303979 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.303436    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:37:59.303979 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:59.303525    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9j6n" podUID="1173b48c-041e-4ec6-9708-98391b2bf0fe"
Sep 18 19:37:59.303979 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.303802    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zh2j5\""
Sep 18 19:37:59.305699 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.305679    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67bgb"
Sep 18 19:37:59.305821 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:59.305797    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67bgb" podUID="0e5d82f6-4e09-448b-b9b7-8644ea3803a3"
Sep 18 19:37:59.308550 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.308279    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cg6mk"
Sep 18 19:37:59.310503 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.310482    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 19:37:59.311137 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.310832    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-j4xl2\""
Sep 18 19:37:59.311137 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.311043    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 19:37:59.311137 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.311054    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.312654 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.312633    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 19:37:59.313391 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.313369    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s2vsh"
Sep 18 19:37:59.313669 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.313651    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vvjv9\""
Sep 18 19:37:59.313800 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.313779    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:37:59.314083 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.314031    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 19:37:59.315645 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.315627    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:37:59.316142 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.315935    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6cfz6\""
Sep 18 19:37:59.316142 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.315974    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 19:37:59.316287 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.316236    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 19:37:59.316900 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.316882    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w9dlm"
Sep 18 19:37:59.318812 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.318783    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mzsk4"
Sep 18 19:37:59.321420 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.321400    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 19:37:59.321678 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.321660    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 19:37:59.321890 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.321873    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8mlf2\""
Sep 18 19:37:59.323187 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.322363    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 19:37:59.323187 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.322902    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 19:37:59.323338 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.323207    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 19:37:59.323695 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.323458    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jnq68\""
Sep 18 19:37:59.324922 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.324688    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-run\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.324922 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.324746    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58fa049b-20bb-4466-bb38-8b46448db93b-tmp-dir\") pod \"node-resolver-w9dlm\" (UID: \"58fa049b-20bb-4466-bb38-8b46448db93b\") " pod="openshift-dns/node-resolver-w9dlm"
Sep 18 19:37:59.324922 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.324774    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-host-slash\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.324922 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.324806    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-host-run-netns\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.324922 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.324849    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a40538a7-7d46-4051-9292-9827fd5cb8af-ovnkube-config\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.324922 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.324882    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-842cm\" (UniqueName: \"kubernetes.io/projected/0e5d82f6-4e09-448b-b9b7-8644ea3803a3-kube-api-access-842cm\") pod \"network-check-target-67bgb\" (UID: \"0e5d82f6-4e09-448b-b9b7-8644ea3803a3\") " pod="openshift-network-diagnostics/network-check-target-67bgb"
Sep 18 19:37:59.325321 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.324935    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t2jb5\" (UID: \"466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jb5"
Sep 18 19:37:59.325321 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.324963    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4krsg\" (UniqueName: \"kubernetes.io/projected/466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f-kube-api-access-4krsg\") pod \"aws-ebs-csi-driver-node-t2jb5\" (UID: \"466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jb5"
Sep 18 19:37:59.325321 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.324998    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-sys\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.325321 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.325029    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-etc-tuned\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.325321 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.325060    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-run-systemd\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.325321 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.325090    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-var-lib-openvswitch\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.325321 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.325115    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-host-cni-bin\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.325321 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.325144    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-host-cni-netd\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.325321 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.325171    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a40538a7-7d46-4051-9292-9827fd5cb8af-ovn-node-metrics-cert\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.325321 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.325211    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d58g5\" (UniqueName: \"kubernetes.io/projected/a40538a7-7d46-4051-9292-9827fd5cb8af-kube-api-access-d58g5\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.325321 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.325286    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-etc-openvswitch\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.325321 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.325318    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a40538a7-7d46-4051-9292-9827fd5cb8af-ovnkube-script-lib\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.326090 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.325359    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f-etc-selinux\") pod \"aws-ebs-csi-driver-node-t2jb5\" (UID: \"466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jb5"
Sep 18 19:37:59.326090 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.325391    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6ff14ef8-6eac-448b-b297-b1fb15d59eac-agent-certs\") pod \"konnectivity-agent-cg6mk\" (UID: \"6ff14ef8-6eac-448b-b297-b1fb15d59eac\") " pod="kube-system/konnectivity-agent-cg6mk"
Sep 18 19:37:59.326090 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.325448    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-etc-modprobe-d\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.326090 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.325486    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-etc-sysctl-conf\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.326090 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.325524    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-host\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.326090 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.325677    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 19:37:59.326392 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.326309    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-run-ovn\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.326392 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.326382    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1173b48c-041e-4ec6-9708-98391b2bf0fe-metrics-certs\") pod \"network-metrics-daemon-s9j6n\" (UID: \"1173b48c-041e-4ec6-9708-98391b2bf0fe\") " pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:37:59.326523 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.326452    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqdgq\" (UniqueName: \"kubernetes.io/projected/1173b48c-041e-4ec6-9708-98391b2bf0fe-kube-api-access-jqdgq\") pod \"network-metrics-daemon-s9j6n\" (UID: \"1173b48c-041e-4ec6-9708-98391b2bf0fe\") " pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:37:59.326523 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.326487    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-etc-systemd\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.326523 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.326516    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txdzh\" (UniqueName: \"kubernetes.io/projected/ee926faa-2422-4be9-82b5-ad1f0ec42ecd-kube-api-access-txdzh\") pod \"iptables-alerter-s2vsh\" (UID: \"ee926faa-2422-4be9-82b5-ad1f0ec42ecd\") " pod="openshift-network-operator/iptables-alerter-s2vsh"
Sep 18 19:37:59.326688 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.326558    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4c59\" (UniqueName: \"kubernetes.io/projected/58fa049b-20bb-4466-bb38-8b46448db93b-kube-api-access-v4c59\") pod \"node-resolver-w9dlm\" (UID: \"58fa049b-20bb-4466-bb38-8b46448db93b\") " pod="openshift-dns/node-resolver-w9dlm"
Sep 18 19:37:59.326688 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.326595    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-systemd-units\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.326688 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.326625    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-node-log\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.326688 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.326649    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-host-run-ovn-kubernetes\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.326688 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.326681    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.326998 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.326713    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f-sys-fs\") pod \"aws-ebs-csi-driver-node-t2jb5\" (UID: \"466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jb5"
Sep 18 19:37:59.326998 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.326738    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:37:59.326998 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.326742    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-lib-modules\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.326998 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.326846    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-var-lib-kubelet\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.326998 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.326908    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-host-kubelet\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.326998 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.326964    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-run-openvswitch\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.327311 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.327005    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f-registration-dir\") pod \"aws-ebs-csi-driver-node-t2jb5\" (UID: \"466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jb5"
Sep 18 19:37:59.327311 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.327034    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6ff14ef8-6eac-448b-b297-b1fb15d59eac-konnectivity-ca\") pod \"konnectivity-agent-cg6mk\" (UID: \"6ff14ef8-6eac-448b-b297-b1fb15d59eac\") " pod="kube-system/konnectivity-agent-cg6mk"
Sep 18 19:37:59.327311 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.327066    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-tmp\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.327311 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.327100    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kl7v\" (UniqueName: \"kubernetes.io/projected/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-kube-api-access-7kl7v\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.327311 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.327132    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ee926faa-2422-4be9-82b5-ad1f0ec42ecd-iptables-alerter-script\") pod \"iptables-alerter-s2vsh\" (UID: \"ee926faa-2422-4be9-82b5-ad1f0ec42ecd\") " pod="openshift-network-operator/iptables-alerter-s2vsh"
Sep 18 19:37:59.327311 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.327159    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f-socket-dir\") pod \"aws-ebs-csi-driver-node-t2jb5\" (UID: \"466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jb5"
Sep 18 19:37:59.327311 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.327190    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-etc-kubernetes\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.327311 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.327220    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-etc-sysctl-d\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.327311 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.327252    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ee926faa-2422-4be9-82b5-ad1f0ec42ecd-host-slash\") pod \"iptables-alerter-s2vsh\" (UID: \"ee926faa-2422-4be9-82b5-ad1f0ec42ecd\") " pod="openshift-network-operator/iptables-alerter-s2vsh"
Sep 18 19:37:59.327311 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.327297    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/58fa049b-20bb-4466-bb38-8b46448db93b-hosts-file\") pod \"node-resolver-w9dlm\" (UID: \"58fa049b-20bb-4466-bb38-8b46448db93b\") " pod="openshift-dns/node-resolver-w9dlm"
Sep 18 19:37:59.327971 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.327355    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-log-socket\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.327971 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.327400    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a40538a7-7d46-4051-9292-9827fd5cb8af-env-overrides\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.327971 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.327427    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f-device-dir\") pod \"aws-ebs-csi-driver-node-t2jb5\" (UID: \"466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jb5"
Sep 18 19:37:59.327971 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.327467    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-etc-sysconfig\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.329098 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.329077    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 19:37:59.330132 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.330085    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7nrfp"
Sep 18 19:37:59.332208 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.332180    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:37:59.332303 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:59.332272    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2p2bp" podUID="5d3541d3-fb52-42a5-b977-9cc44ac396bc"
Sep 18 19:37:59.334356 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.334326    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 19:37:59.335189 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.334605    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 19:37:59.335189 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.334726    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.335189 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.334794    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 19:37:59.335189 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.334993    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-27rzb\""
Sep 18 19:37:59.338849 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.338830    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 19:37:59.339050 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.339035    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4bh2j\""
Sep 18 19:37:59.350624 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.350601    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vvngq"
Sep 18 19:37:59.358514 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.358493    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vvngq"
Sep 18 19:37:59.414362 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.414338    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 19:37:59.428150 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.427883    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-log-socket\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.428150 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.427917    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f-device-dir\") pod \"aws-ebs-csi-driver-node-t2jb5\" (UID: \"466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jb5"
Sep 18 19:37:59.428150 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.427943    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-etc-sysconfig\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.428150 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.427975    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-host-var-lib-kubelet\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.428150 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.427994    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-log-socket\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.428150 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428018    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-host-slash\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.428150 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428037    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f-device-dir\") pod \"aws-ebs-csi-driver-node-t2jb5\" (UID: \"466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jb5"
Sep 18 19:37:59.428150 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428042    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-etc-sysconfig\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.428150 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428075    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-host-slash\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.428150 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428046    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a40538a7-7d46-4051-9292-9827fd5cb8af-ovnkube-config\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.428150 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428115    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-842cm\" (UniqueName: \"kubernetes.io/projected/0e5d82f6-4e09-448b-b9b7-8644ea3803a3-kube-api-access-842cm\") pod \"network-check-target-67bgb\" (UID: \"0e5d82f6-4e09-448b-b9b7-8644ea3803a3\") " pod="openshift-network-diagnostics/network-check-target-67bgb"
Sep 18 19:37:59.428150 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428157    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cff0cd8b-6053-42d8-9b1e-20930e49616b-cnibin\") pod \"multus-additional-cni-plugins-mzsk4\" (UID: \"cff0cd8b-6053-42d8-9b1e-20930e49616b\") " pod="openshift-multus/multus-additional-cni-plugins-mzsk4"
Sep 18 19:37:59.428991 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428185    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5d3541d3-fb52-42a5-b977-9cc44ac396bc-kubelet-config\") pod \"global-pull-secret-syncer-2p2bp\" (UID: \"5d3541d3-fb52-42a5-b977-9cc44ac396bc\") " pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:37:59.428991 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428212    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-run-systemd\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.428991 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428239    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-run-systemd\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.428991 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428249    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-var-lib-openvswitch\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.428991 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428274    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-var-lib-openvswitch\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.428991 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428291    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cff0cd8b-6053-42d8-9b1e-20930e49616b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mzsk4\" (UID: \"cff0cd8b-6053-42d8-9b1e-20930e49616b\") " pod="openshift-multus/multus-additional-cni-plugins-mzsk4"
Sep 18 19:37:59.428991 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428317    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-host\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.428991 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428340    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4c59\" (UniqueName: \"kubernetes.io/projected/58fa049b-20bb-4466-bb38-8b46448db93b-kube-api-access-v4c59\") pod \"node-resolver-w9dlm\" (UID: \"58fa049b-20bb-4466-bb38-8b46448db93b\") " pod="openshift-dns/node-resolver-w9dlm"
Sep 18 19:37:59.428991 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428366    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn45z\" (UniqueName: \"kubernetes.io/projected/68f35ba9-52cf-4a53-bc81-b2863eeb0930-kube-api-access-tn45z\") pod \"node-ca-7nrfp\" (UID: \"68f35ba9-52cf-4a53-bc81-b2863eeb0930\") " pod="openshift-image-registry/node-ca-7nrfp"
Sep 18 19:37:59.428991 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428397    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-host\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.428991 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428391    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cff0cd8b-6053-42d8-9b1e-20930e49616b-system-cni-dir\") pod \"multus-additional-cni-plugins-mzsk4\" (UID: \"cff0cd8b-6053-42d8-9b1e-20930e49616b\") " pod="openshift-multus/multus-additional-cni-plugins-mzsk4"
Sep 18 19:37:59.428991 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428436    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d3541d3-fb52-42a5-b977-9cc44ac396bc-original-pull-secret\") pod \"global-pull-secret-syncer-2p2bp\" (UID: \"5d3541d3-fb52-42a5-b977-9cc44ac396bc\") " pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:37:59.428991 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428469    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-system-cni-dir\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.428991 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428505    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-host-run-netns\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.428991 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428534    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-run-ovn\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.428991 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428584    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-etc-systemd\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.428991 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428593    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-run-ovn\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.430045 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428627    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/697bb62f-6332-4a66-a1eb-84aa42f6a620-cni-binary-copy\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.430045 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428654    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-multus-socket-dir-parent\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.430045 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428668    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-etc-systemd\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.430045 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428683    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/697bb62f-6332-4a66-a1eb-84aa42f6a620-multus-daemon-config\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.430045 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428714    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-node-log\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.430045 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428740    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-lib-modules\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.430045 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428766    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-run-openvswitch\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.430045 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428791    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f-registration-dir\") pod \"aws-ebs-csi-driver-node-t2jb5\" (UID: \"466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jb5"
Sep 18 19:37:59.430045 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428816    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6ff14ef8-6eac-448b-b297-b1fb15d59eac-konnectivity-ca\") pod \"konnectivity-agent-cg6mk\" (UID: \"6ff14ef8-6eac-448b-b297-b1fb15d59eac\") " pod="kube-system/konnectivity-agent-cg6mk"
Sep 18 19:37:59.430045 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428839    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-multus-cni-dir\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.430045 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428766    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-node-log\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.430045 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428902    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-run-openvswitch\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.430045 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428936    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f-socket-dir\") pod \"aws-ebs-csi-driver-node-t2jb5\" (UID: \"466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jb5"
Sep 18 19:37:59.430045 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428936    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f-registration-dir\") pod \"aws-ebs-csi-driver-node-t2jb5\" (UID: \"466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jb5"
Sep 18 19:37:59.430045 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428967    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-etc-kubernetes\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.430045 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428994    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-host-var-lib-cni-bin\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.430045 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a40538a7-7d46-4051-9292-9827fd5cb8af-env-overrides\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.430957 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429043    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-run\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.430957 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429066    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58fa049b-20bb-4466-bb38-8b46448db93b-tmp-dir\") pod \"node-resolver-w9dlm\" (UID: \"58fa049b-20bb-4466-bb38-8b46448db93b\") " pod="openshift-dns/node-resolver-w9dlm"
Sep 18 19:37:59.430957 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429084    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f-socket-dir\") pod \"aws-ebs-csi-driver-node-t2jb5\" (UID: \"466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jb5"
Sep 18 19:37:59.430957 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.428968    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-lib-modules\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.430957 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429102    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-etc-kubernetes\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.430957 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429129    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-host-run-netns\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.430957 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429153    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t2jb5\" (UID: \"466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jb5"
Sep 18 19:37:59.430957 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429180    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4krsg\" (UniqueName: \"kubernetes.io/projected/466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f-kube-api-access-4krsg\") pod \"aws-ebs-csi-driver-node-t2jb5\" (UID: \"466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jb5"
Sep 18 19:37:59.430957 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429208    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-sys\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.430957 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429218    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-host-run-netns\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.430957 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429233    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-etc-tuned\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.430957 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429237    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t2jb5\" (UID: \"466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jb5"
Sep 18 19:37:59.430957 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429180    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-run\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.430957 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429260    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fq2t\" (UniqueName: \"kubernetes.io/projected/cff0cd8b-6053-42d8-9b1e-20930e49616b-kube-api-access-7fq2t\") pod \"multus-additional-cni-plugins-mzsk4\" (UID: \"cff0cd8b-6053-42d8-9b1e-20930e49616b\") " pod="openshift-multus/multus-additional-cni-plugins-mzsk4"
Sep 18 19:37:59.430957 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429289    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-cnibin\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.430957 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429343    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-host-run-k8s-cni-cncf-io\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.430957 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429370    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-host-cni-bin\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.431759 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429394    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-host-cni-netd\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.431759 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429395    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-etc-kubernetes\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.431759 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429433    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a40538a7-7d46-4051-9292-9827fd5cb8af-ovn-node-metrics-cert\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.431759 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429467    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d58g5\" (UniqueName: \"kubernetes.io/projected/a40538a7-7d46-4051-9292-9827fd5cb8af-kube-api-access-d58g5\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.431759 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429494    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/68f35ba9-52cf-4a53-bc81-b2863eeb0930-host\") pod \"node-ca-7nrfp\" (UID: \"68f35ba9-52cf-4a53-bc81-b2863eeb0930\") " pod="openshift-image-registry/node-ca-7nrfp"
Sep 18 19:37:59.431759 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429495    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-host-cni-netd\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.431759 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429511    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-host-cni-bin\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.431759 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429534    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6ff14ef8-6eac-448b-b297-b1fb15d59eac-konnectivity-ca\") pod \"konnectivity-agent-cg6mk\" (UID: \"6ff14ef8-6eac-448b-b297-b1fb15d59eac\") " pod="kube-system/konnectivity-agent-cg6mk"
Sep 18 19:37:59.431759 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429556    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a40538a7-7d46-4051-9292-9827fd5cb8af-env-overrides\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.431759 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429618    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-sys\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.431759 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-etc-openvswitch\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.431759 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429680    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a40538a7-7d46-4051-9292-9827fd5cb8af-ovnkube-script-lib\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.431759 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429705    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f-etc-selinux\") pod \"aws-ebs-csi-driver-node-t2jb5\" (UID: \"466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jb5"
Sep 18 19:37:59.431759 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429764    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-etc-openvswitch\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.431759 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429797    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6ff14ef8-6eac-448b-b297-b1fb15d59eac-agent-certs\") pod \"konnectivity-agent-cg6mk\" (UID: \"6ff14ef8-6eac-448b-b297-b1fb15d59eac\") " pod="kube-system/konnectivity-agent-cg6mk"
Sep 18 19:37:59.431759 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429800    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 19:37:59.431759 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429826    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-etc-modprobe-d\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.431759 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429852    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-etc-sysctl-conf\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.432730 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429856    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58fa049b-20bb-4466-bb38-8b46448db93b-tmp-dir\") pod \"node-resolver-w9dlm\" (UID: \"58fa049b-20bb-4466-bb38-8b46448db93b\") " pod="openshift-dns/node-resolver-w9dlm"
Sep 18 19:37:59.432730 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429881    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txdzh\" (UniqueName: \"kubernetes.io/projected/ee926faa-2422-4be9-82b5-ad1f0ec42ecd-kube-api-access-txdzh\") pod \"iptables-alerter-s2vsh\" (UID: \"ee926faa-2422-4be9-82b5-ad1f0ec42ecd\") " pod="openshift-network-operator/iptables-alerter-s2vsh"
Sep 18 19:37:59.432730 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429914    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f-etc-selinux\") pod \"aws-ebs-csi-driver-node-t2jb5\" (UID: \"466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jb5"
Sep 18 19:37:59.432730 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429930    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1173b48c-041e-4ec6-9708-98391b2bf0fe-metrics-certs\") pod \"network-metrics-daemon-s9j6n\" (UID: \"1173b48c-041e-4ec6-9708-98391b2bf0fe\") " pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:37:59.432730 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429962    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqdgq\" (UniqueName: \"kubernetes.io/projected/1173b48c-041e-4ec6-9708-98391b2bf0fe-kube-api-access-jqdgq\") pod \"network-metrics-daemon-s9j6n\" (UID: \"1173b48c-041e-4ec6-9708-98391b2bf0fe\") " pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:37:59.432730 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.429992    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-host-var-lib-cni-multus\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.432730 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430006    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-etc-modprobe-d\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.432730 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430019    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-hostroot\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.432730 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430042    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-multus-conf-dir\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.432730 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:59.430077    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:37:59.432730 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430103    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-host-run-multus-certs\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.432730 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:59.430146    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1173b48c-041e-4ec6-9708-98391b2bf0fe-metrics-certs podName:1173b48c-041e-4ec6-9708-98391b2bf0fe nodeName:}" failed. No retries permitted until 2026-09-18 19:37:59.930124152 +0000 UTC m=+3.032412482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1173b48c-041e-4ec6-9708-98391b2bf0fe-metrics-certs") pod "network-metrics-daemon-s9j6n" (UID: "1173b48c-041e-4ec6-9708-98391b2bf0fe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:37:59.432730 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430170    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-systemd-units\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.432730 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430201    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-host-run-ovn-kubernetes\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.432730 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430233    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.432730 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430261    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f-sys-fs\") pod \"aws-ebs-csi-driver-node-t2jb5\" (UID: \"466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jb5"
Sep 18 19:37:59.432730 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430282    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-etc-sysctl-conf\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.433811 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430336    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-var-lib-kubelet\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.433811 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430288    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-var-lib-kubelet\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.433811 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430378    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-tmp\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.433811 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430408    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kl7v\" (UniqueName: \"kubernetes.io/projected/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-kube-api-access-7kl7v\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.433811 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430435    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-host-kubelet\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.433811 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430460    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-systemd-units\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.433811 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430408    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-host-run-ovn-kubernetes\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.433811 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430463    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ee926faa-2422-4be9-82b5-ad1f0ec42ecd-iptables-alerter-script\") pod \"iptables-alerter-s2vsh\" (UID: \"ee926faa-2422-4be9-82b5-ad1f0ec42ecd\") " pod="openshift-network-operator/iptables-alerter-s2vsh"
Sep 18 19:37:59.433811 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430516    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ee926faa-2422-4be9-82b5-ad1f0ec42ecd-host-slash\") pod \"iptables-alerter-s2vsh\" (UID: \"ee926faa-2422-4be9-82b5-ad1f0ec42ecd\") " pod="openshift-network-operator/iptables-alerter-s2vsh"
Sep 18 19:37:59.433811 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430543    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/58fa049b-20bb-4466-bb38-8b46448db93b-hosts-file\") pod \"node-resolver-w9dlm\" (UID: \"58fa049b-20bb-4466-bb38-8b46448db93b\") " pod="openshift-dns/node-resolver-w9dlm"
Sep 18 19:37:59.433811 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430557    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-host-kubelet\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.433811 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430570    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/68f35ba9-52cf-4a53-bc81-b2863eeb0930-serviceca\") pod \"node-ca-7nrfp\" (UID: \"68f35ba9-52cf-4a53-bc81-b2863eeb0930\") " pod="openshift-image-registry/node-ca-7nrfp"
Sep 18 19:37:59.433811 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430512    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a40538a7-7d46-4051-9292-9827fd5cb8af-ovnkube-script-lib\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.433811 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430599    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ee926faa-2422-4be9-82b5-ad1f0ec42ecd-host-slash\") pod \"iptables-alerter-s2vsh\" (UID: \"ee926faa-2422-4be9-82b5-ad1f0ec42ecd\") " pod="openshift-network-operator/iptables-alerter-s2vsh"
Sep 18 19:37:59.433811 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430606    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cff0cd8b-6053-42d8-9b1e-20930e49616b-cni-binary-copy\") pod \"multus-additional-cni-plugins-mzsk4\" (UID: \"cff0cd8b-6053-42d8-9b1e-20930e49616b\") " pod="openshift-multus/multus-additional-cni-plugins-mzsk4"
Sep 18 19:37:59.433811 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430563    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f-sys-fs\") pod \"aws-ebs-csi-driver-node-t2jb5\" (UID: \"466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jb5"
Sep 18 19:37:59.433811 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430632    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/58fa049b-20bb-4466-bb38-8b46448db93b-hosts-file\") pod \"node-resolver-w9dlm\" (UID: \"58fa049b-20bb-4466-bb38-8b46448db93b\") " pod="openshift-dns/node-resolver-w9dlm"
Sep 18 19:37:59.434808 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430660    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cff0cd8b-6053-42d8-9b1e-20930e49616b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mzsk4\" (UID: \"cff0cd8b-6053-42d8-9b1e-20930e49616b\") " pod="openshift-multus/multus-additional-cni-plugins-mzsk4"
Sep 18 19:37:59.434808 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430689    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a40538a7-7d46-4051-9292-9827fd5cb8af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.434808 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430696    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf7j6\" (UniqueName: \"kubernetes.io/projected/697bb62f-6332-4a66-a1eb-84aa42f6a620-kube-api-access-sf7j6\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.434808 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430732    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-etc-sysctl-d\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.434808 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430763    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cff0cd8b-6053-42d8-9b1e-20930e49616b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mzsk4\" (UID: \"cff0cd8b-6053-42d8-9b1e-20930e49616b\") " pod="openshift-multus/multus-additional-cni-plugins-mzsk4"
Sep 18 19:37:59.434808 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430788    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5d3541d3-fb52-42a5-b977-9cc44ac396bc-dbus\") pod \"global-pull-secret-syncer-2p2bp\" (UID: \"5d3541d3-fb52-42a5-b977-9cc44ac396bc\") " pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:37:59.434808 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.430869    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-etc-sysctl-d\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.434808 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.431383    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a40538a7-7d46-4051-9292-9827fd5cb8af-ovnkube-config\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.434808 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.431777    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ee926faa-2422-4be9-82b5-ad1f0ec42ecd-iptables-alerter-script\") pod \"iptables-alerter-s2vsh\" (UID: \"ee926faa-2422-4be9-82b5-ad1f0ec42ecd\") " pod="openshift-network-operator/iptables-alerter-s2vsh"
Sep 18 19:37:59.434808 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:59.434409    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:37:59.434808 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:59.434473    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:37:59.434808 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:59.434488    2581 projected.go:196] Error preparing data for projected volume kube-api-access-842cm for pod openshift-network-diagnostics/network-check-target-67bgb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:37:59.434808 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:59.434576    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e5d82f6-4e09-448b-b9b7-8644ea3803a3-kube-api-access-842cm podName:0e5d82f6-4e09-448b-b9b7-8644ea3803a3 nodeName:}" failed. No retries permitted until 2026-09-18 19:37:59.934557133 +0000 UTC m=+3.036845445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-842cm" (UniqueName: "kubernetes.io/projected/0e5d82f6-4e09-448b-b9b7-8644ea3803a3-kube-api-access-842cm") pod "network-check-target-67bgb" (UID: "0e5d82f6-4e09-448b-b9b7-8644ea3803a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:37:59.435968 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.435946    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a40538a7-7d46-4051-9292-9827fd5cb8af-ovn-node-metrics-cert\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.436112 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.436093    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-etc-tuned\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.436453 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.436435    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6ff14ef8-6eac-448b-b297-b1fb15d59eac-agent-certs\") pod \"konnectivity-agent-cg6mk\" (UID: \"6ff14ef8-6eac-448b-b297-b1fb15d59eac\") " pod="kube-system/konnectivity-agent-cg6mk"
Sep 18 19:37:59.436605 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.436585    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-tmp\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.438526 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.438498    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4c59\" (UniqueName: \"kubernetes.io/projected/58fa049b-20bb-4466-bb38-8b46448db93b-kube-api-access-v4c59\") pod \"node-resolver-w9dlm\" (UID: \"58fa049b-20bb-4466-bb38-8b46448db93b\") " pod="openshift-dns/node-resolver-w9dlm"
Sep 18 19:37:59.439601 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.439508    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txdzh\" (UniqueName: \"kubernetes.io/projected/ee926faa-2422-4be9-82b5-ad1f0ec42ecd-kube-api-access-txdzh\") pod \"iptables-alerter-s2vsh\" (UID: \"ee926faa-2422-4be9-82b5-ad1f0ec42ecd\") " pod="openshift-network-operator/iptables-alerter-s2vsh"
Sep 18 19:37:59.440840 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.440812    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d58g5\" (UniqueName: \"kubernetes.io/projected/a40538a7-7d46-4051-9292-9827fd5cb8af-kube-api-access-d58g5\") pod \"ovnkube-node-2wj49\" (UID: \"a40538a7-7d46-4051-9292-9827fd5cb8af\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.440999 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.440836    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4krsg\" (UniqueName: \"kubernetes.io/projected/466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f-kube-api-access-4krsg\") pod \"aws-ebs-csi-driver-node-t2jb5\" (UID: \"466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jb5"
Sep 18 19:37:59.444018 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.442889    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kl7v\" (UniqueName: \"kubernetes.io/projected/2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e-kube-api-access-7kl7v\") pod \"tuned-lld5l\" (UID: \"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e\") " pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.444018 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.443077    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqdgq\" (UniqueName: \"kubernetes.io/projected/1173b48c-041e-4ec6-9708-98391b2bf0fe-kube-api-access-jqdgq\") pod \"network-metrics-daemon-s9j6n\" (UID: \"1173b48c-041e-4ec6-9708-98391b2bf0fe\") " pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:37:59.531065 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531028    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cff0cd8b-6053-42d8-9b1e-20930e49616b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mzsk4\" (UID: \"cff0cd8b-6053-42d8-9b1e-20930e49616b\") " pod="openshift-multus/multus-additional-cni-plugins-mzsk4"
Sep 18 19:37:59.531065 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531061    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn45z\" (UniqueName: \"kubernetes.io/projected/68f35ba9-52cf-4a53-bc81-b2863eeb0930-kube-api-access-tn45z\") pod \"node-ca-7nrfp\" (UID: \"68f35ba9-52cf-4a53-bc81-b2863eeb0930\") " pod="openshift-image-registry/node-ca-7nrfp"
Sep 18 19:37:59.531270 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531082    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cff0cd8b-6053-42d8-9b1e-20930e49616b-system-cni-dir\") pod \"multus-additional-cni-plugins-mzsk4\" (UID: \"cff0cd8b-6053-42d8-9b1e-20930e49616b\") " pod="openshift-multus/multus-additional-cni-plugins-mzsk4"
Sep 18 19:37:59.531270 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531239    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d3541d3-fb52-42a5-b977-9cc44ac396bc-original-pull-secret\") pod \"global-pull-secret-syncer-2p2bp\" (UID: \"5d3541d3-fb52-42a5-b977-9cc44ac396bc\") " pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:37:59.531378 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531271    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cff0cd8b-6053-42d8-9b1e-20930e49616b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mzsk4\" (UID: \"cff0cd8b-6053-42d8-9b1e-20930e49616b\") " pod="openshift-multus/multus-additional-cni-plugins-mzsk4"
Sep 18 19:37:59.531378 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531294    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-system-cni-dir\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.531378 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531325    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-host-run-netns\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.531378 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531344    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-system-cni-dir\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.531378 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:59.531353    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:37:59.531378 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531271    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cff0cd8b-6053-42d8-9b1e-20930e49616b-system-cni-dir\") pod \"multus-additional-cni-plugins-mzsk4\" (UID: \"cff0cd8b-6053-42d8-9b1e-20930e49616b\") " pod="openshift-multus/multus-additional-cni-plugins-mzsk4"
Sep 18 19:37:59.531378 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531360    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/697bb62f-6332-4a66-a1eb-84aa42f6a620-cni-binary-copy\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.531725 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531405    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-multus-socket-dir-parent\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.531725 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531410    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-host-run-netns\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.531725 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:59.531420    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d3541d3-fb52-42a5-b977-9cc44ac396bc-original-pull-secret podName:5d3541d3-fb52-42a5-b977-9cc44ac396bc nodeName:}" failed. No retries permitted until 2026-09-18 19:38:00.031389326 +0000 UTC m=+3.133677637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d3541d3-fb52-42a5-b977-9cc44ac396bc-original-pull-secret") pod "global-pull-secret-syncer-2p2bp" (UID: "5d3541d3-fb52-42a5-b977-9cc44ac396bc") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:37:59.531725 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531449    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/697bb62f-6332-4a66-a1eb-84aa42f6a620-multus-daemon-config\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.531725 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531451    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-multus-socket-dir-parent\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.531725 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531570    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-multus-cni-dir\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.531725 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531599    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-host-var-lib-cni-bin\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.531725 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531625    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-etc-kubernetes\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.531725 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531656    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fq2t\" (UniqueName: \"kubernetes.io/projected/cff0cd8b-6053-42d8-9b1e-20930e49616b-kube-api-access-7fq2t\") pod \"multus-additional-cni-plugins-mzsk4\" (UID: \"cff0cd8b-6053-42d8-9b1e-20930e49616b\") " pod="openshift-multus/multus-additional-cni-plugins-mzsk4"
Sep 18 19:37:59.531725 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531681    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-cnibin\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.531725 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531701    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-host-run-k8s-cni-cncf-io\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.531725 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531721    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/68f35ba9-52cf-4a53-bc81-b2863eeb0930-host\") pod \"node-ca-7nrfp\" (UID: \"68f35ba9-52cf-4a53-bc81-b2863eeb0930\") " pod="openshift-image-registry/node-ca-7nrfp"
Sep 18 19:37:59.532355 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531757    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-host-var-lib-cni-multus\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.532355 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531773    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-hostroot\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.532355 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531782    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-multus-cni-dir\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.532355 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531823    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-host-var-lib-cni-bin\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.532355 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531836    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-multus-conf-dir\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.532355 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531786    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-multus-conf-dir\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.532355 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531884    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-etc-kubernetes\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.532355 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531897    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-host-run-multus-certs\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.532355 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531933    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/68f35ba9-52cf-4a53-bc81-b2863eeb0930-serviceca\") pod \"node-ca-7nrfp\" (UID: \"68f35ba9-52cf-4a53-bc81-b2863eeb0930\") " pod="openshift-image-registry/node-ca-7nrfp"
Sep 18 19:37:59.532355 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531950    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cff0cd8b-6053-42d8-9b1e-20930e49616b-cni-binary-copy\") pod \"multus-additional-cni-plugins-mzsk4\" (UID: \"cff0cd8b-6053-42d8-9b1e-20930e49616b\") " pod="openshift-multus/multus-additional-cni-plugins-mzsk4"
Sep 18 19:37:59.532355 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531953    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-cnibin\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.532355 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531968    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cff0cd8b-6053-42d8-9b1e-20930e49616b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mzsk4\" (UID: \"cff0cd8b-6053-42d8-9b1e-20930e49616b\") " pod="openshift-multus/multus-additional-cni-plugins-mzsk4"
Sep 18 19:37:59.532355 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531979    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-host-run-k8s-cni-cncf-io\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.532355 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531988    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf7j6\" (UniqueName: \"kubernetes.io/projected/697bb62f-6332-4a66-a1eb-84aa42f6a620-kube-api-access-sf7j6\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.532355 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.531989    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/697bb62f-6332-4a66-a1eb-84aa42f6a620-cni-binary-copy\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.532355 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.532015    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/68f35ba9-52cf-4a53-bc81-b2863eeb0930-host\") pod \"node-ca-7nrfp\" (UID: \"68f35ba9-52cf-4a53-bc81-b2863eeb0930\") " pod="openshift-image-registry/node-ca-7nrfp"
Sep 18 19:37:59.532355 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.532017    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cff0cd8b-6053-42d8-9b1e-20930e49616b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mzsk4\" (UID: \"cff0cd8b-6053-42d8-9b1e-20930e49616b\") " pod="openshift-multus/multus-additional-cni-plugins-mzsk4"
Sep 18 19:37:59.532355 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.532044    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5d3541d3-fb52-42a5-b977-9cc44ac396bc-dbus\") pod \"global-pull-secret-syncer-2p2bp\" (UID: \"5d3541d3-fb52-42a5-b977-9cc44ac396bc\") " pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:37:59.533171 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.532063    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-hostroot\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.533171 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.532072    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-host-var-lib-kubelet\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.533171 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.532119    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-host-var-lib-kubelet\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.533171 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.532433    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/68f35ba9-52cf-4a53-bc81-b2863eeb0930-serviceca\") pod \"node-ca-7nrfp\" (UID: \"68f35ba9-52cf-4a53-bc81-b2863eeb0930\") " pod="openshift-image-registry/node-ca-7nrfp"
Sep 18 19:37:59.533171 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.532491    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-host-var-lib-cni-multus\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.533171 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.532538    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/697bb62f-6332-4a66-a1eb-84aa42f6a620-host-run-multus-certs\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.533171 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.532635    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cff0cd8b-6053-42d8-9b1e-20930e49616b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mzsk4\" (UID: \"cff0cd8b-6053-42d8-9b1e-20930e49616b\") " pod="openshift-multus/multus-additional-cni-plugins-mzsk4"
Sep 18 19:37:59.533171 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.532695    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5d3541d3-fb52-42a5-b977-9cc44ac396bc-dbus\") pod \"global-pull-secret-syncer-2p2bp\" (UID: \"5d3541d3-fb52-42a5-b977-9cc44ac396bc\") " pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:37:59.533171 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.532701    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cff0cd8b-6053-42d8-9b1e-20930e49616b-cni-binary-copy\") pod \"multus-additional-cni-plugins-mzsk4\" (UID: \"cff0cd8b-6053-42d8-9b1e-20930e49616b\") " pod="openshift-multus/multus-additional-cni-plugins-mzsk4"
Sep 18 19:37:59.533171 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.532750    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cff0cd8b-6053-42d8-9b1e-20930e49616b-cnibin\") pod \"multus-additional-cni-plugins-mzsk4\" (UID: \"cff0cd8b-6053-42d8-9b1e-20930e49616b\") " pod="openshift-multus/multus-additional-cni-plugins-mzsk4"
Sep 18 19:37:59.533171 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.532776    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5d3541d3-fb52-42a5-b977-9cc44ac396bc-kubelet-config\") pod \"global-pull-secret-syncer-2p2bp\" (UID: \"5d3541d3-fb52-42a5-b977-9cc44ac396bc\") " pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:37:59.533171 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.532855    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5d3541d3-fb52-42a5-b977-9cc44ac396bc-kubelet-config\") pod \"global-pull-secret-syncer-2p2bp\" (UID: \"5d3541d3-fb52-42a5-b977-9cc44ac396bc\") " pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:37:59.533171 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.532886    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cff0cd8b-6053-42d8-9b1e-20930e49616b-cnibin\") pod \"multus-additional-cni-plugins-mzsk4\" (UID: \"cff0cd8b-6053-42d8-9b1e-20930e49616b\") " pod="openshift-multus/multus-additional-cni-plugins-mzsk4"
Sep 18 19:37:59.533171 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.533141    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cff0cd8b-6053-42d8-9b1e-20930e49616b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mzsk4\" (UID: \"cff0cd8b-6053-42d8-9b1e-20930e49616b\") " pod="openshift-multus/multus-additional-cni-plugins-mzsk4"
Sep 18 19:37:59.533668 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.533332    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/697bb62f-6332-4a66-a1eb-84aa42f6a620-multus-daemon-config\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.549824 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.549762    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn45z\" (UniqueName: \"kubernetes.io/projected/68f35ba9-52cf-4a53-bc81-b2863eeb0930-kube-api-access-tn45z\") pod \"node-ca-7nrfp\" (UID: \"68f35ba9-52cf-4a53-bc81-b2863eeb0930\") " pod="openshift-image-registry/node-ca-7nrfp"
Sep 18 19:37:59.549824 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.549773    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fq2t\" (UniqueName: \"kubernetes.io/projected/cff0cd8b-6053-42d8-9b1e-20930e49616b-kube-api-access-7fq2t\") pod \"multus-additional-cni-plugins-mzsk4\" (UID: \"cff0cd8b-6053-42d8-9b1e-20930e49616b\") " pod="openshift-multus/multus-additional-cni-plugins-mzsk4"
Sep 18 19:37:59.557088 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.557068    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf7j6\" (UniqueName: \"kubernetes.io/projected/697bb62f-6332-4a66-a1eb-84aa42f6a620-kube-api-access-sf7j6\") pod \"multus-4j6th\" (UID: \"697bb62f-6332-4a66-a1eb-84aa42f6a620\") " pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.612238 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.612210    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:37:59.624708 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.624686    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jb5"
Sep 18 19:37:59.632750 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.632730    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cg6mk"
Sep 18 19:37:59.644647 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.644625    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lld5l"
Sep 18 19:37:59.652529 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.652510    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s2vsh"
Sep 18 19:37:59.663409 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.663384    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w9dlm"
Sep 18 19:37:59.673305 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.673282    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mzsk4"
Sep 18 19:37:59.680181 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.680159    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7nrfp"
Sep 18 19:37:59.687300 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.687278    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4j6th"
Sep 18 19:37:59.711977 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.711955    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:37:59.936713 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.936622    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1173b48c-041e-4ec6-9708-98391b2bf0fe-metrics-certs\") pod \"network-metrics-daemon-s9j6n\" (UID: \"1173b48c-041e-4ec6-9708-98391b2bf0fe\") " pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:37:59.936713 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:37:59.936691    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-842cm\" (UniqueName: \"kubernetes.io/projected/0e5d82f6-4e09-448b-b9b7-8644ea3803a3-kube-api-access-842cm\") pod \"network-check-target-67bgb\" (UID: \"0e5d82f6-4e09-448b-b9b7-8644ea3803a3\") " pod="openshift-network-diagnostics/network-check-target-67bgb"
Sep 18 19:37:59.936954 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:59.936791    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:37:59.936954 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:59.936873    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:37:59.936954 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:59.936888    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1173b48c-041e-4ec6-9708-98391b2bf0fe-metrics-certs podName:1173b48c-041e-4ec6-9708-98391b2bf0fe nodeName:}" failed. No retries permitted until 2026-09-18 19:38:00.936871669 +0000 UTC m=+4.039159976 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1173b48c-041e-4ec6-9708-98391b2bf0fe-metrics-certs") pod "network-metrics-daemon-s9j6n" (UID: "1173b48c-041e-4ec6-9708-98391b2bf0fe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:37:59.936954 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:59.936895    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:37:59.936954 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:59.936905    2581 projected.go:196] Error preparing data for projected volume kube-api-access-842cm for pod openshift-network-diagnostics/network-check-target-67bgb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:37:59.936954 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:37:59.936952    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e5d82f6-4e09-448b-b9b7-8644ea3803a3-kube-api-access-842cm podName:0e5d82f6-4e09-448b-b9b7-8644ea3803a3 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:00.936940189 +0000 UTC m=+4.039228497 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-842cm" (UniqueName: "kubernetes.io/projected/0e5d82f6-4e09-448b-b9b7-8644ea3803a3-kube-api-access-842cm") pod "network-check-target-67bgb" (UID: "0e5d82f6-4e09-448b-b9b7-8644ea3803a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:00.037154 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:00.037089    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d3541d3-fb52-42a5-b977-9cc44ac396bc-original-pull-secret\") pod \"global-pull-secret-syncer-2p2bp\" (UID: \"5d3541d3-fb52-42a5-b977-9cc44ac396bc\") " pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:38:00.037353 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:00.037236    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:00.037353 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:00.037324    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d3541d3-fb52-42a5-b977-9cc44ac396bc-original-pull-secret podName:5d3541d3-fb52-42a5-b977-9cc44ac396bc nodeName:}" failed. No retries permitted until 2026-09-18 19:38:01.037301381 +0000 UTC m=+4.139589708 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d3541d3-fb52-42a5-b977-9cc44ac396bc-original-pull-secret") pod "global-pull-secret-syncer-2p2bp" (UID: "5d3541d3-fb52-42a5-b977-9cc44ac396bc") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:00.363297 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:00.363172    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:33:00 +0000 UTC" deadline="2028-07-05 00:55:53.858263236 +0000 UTC"
Sep 18 19:38:00.363297 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:00.363232    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15725h17m53.495036469s"
Sep 18 19:38:00.427216 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:00.427180    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:38:00.427361 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:00.427323    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2p2bp" podUID="5d3541d3-fb52-42a5-b977-9cc44ac396bc"
Sep 18 19:38:00.548191 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:38:00.548140    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2480d7a4_45ca_4ec7_ad05_9ef85b6fff0e.slice/crio-1ca75787b3ab223c566bbb143a0c4c0633709d76d08db4ca470a04c54704a4ee WatchSource:0}: Error finding container 1ca75787b3ab223c566bbb143a0c4c0633709d76d08db4ca470a04c54704a4ee: Status 404 returned error can't find the container with id 1ca75787b3ab223c566bbb143a0c4c0633709d76d08db4ca470a04c54704a4ee
Sep 18 19:38:00.549989 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:38:00.549964    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff14ef8_6eac_448b_b297_b1fb15d59eac.slice/crio-3666e64385bdcb655e3a2196b3c78e498b42ba50608363086bcf041246d61a9b WatchSource:0}: Error finding container 3666e64385bdcb655e3a2196b3c78e498b42ba50608363086bcf041246d61a9b: Status 404 returned error can't find the container with id 3666e64385bdcb655e3a2196b3c78e498b42ba50608363086bcf041246d61a9b
Sep 18 19:38:00.552187 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:38:00.552160    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58fa049b_20bb_4466_bb38_8b46448db93b.slice/crio-1735c5b07cc090e2dec1257f9dcae22738074cd0b09ecd8f28651d1afec332f4 WatchSource:0}: Error finding container 1735c5b07cc090e2dec1257f9dcae22738074cd0b09ecd8f28651d1afec332f4: Status 404 returned error can't find the container with id 1735c5b07cc090e2dec1257f9dcae22738074cd0b09ecd8f28651d1afec332f4
Sep 18 19:38:00.555984 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:38:00.555641    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff0cd8b_6053_42d8_9b1e_20930e49616b.slice/crio-25d9f297cbdc71c30114e7cebdf1aab5618ad12bcc398ab34c904a7f400dc8f0 WatchSource:0}: Error finding container 25d9f297cbdc71c30114e7cebdf1aab5618ad12bcc398ab34c904a7f400dc8f0: Status 404 returned error can't find the container with id 25d9f297cbdc71c30114e7cebdf1aab5618ad12bcc398ab34c904a7f400dc8f0
Sep 18 19:38:00.558971 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:38:00.556471    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda40538a7_7d46_4051_9292_9827fd5cb8af.slice/crio-6b4dbca06c385301e1a6286dada858431df9bd711dc4bf822bbed03f372e9697 WatchSource:0}: Error finding container 6b4dbca06c385301e1a6286dada858431df9bd711dc4bf822bbed03f372e9697: Status 404 returned error can't find the container with id 6b4dbca06c385301e1a6286dada858431df9bd711dc4bf822bbed03f372e9697
Sep 18 19:38:00.558971 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:38:00.557788    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f35ba9_52cf_4a53_bc81_b2863eeb0930.slice/crio-40caad49bb65bbbdafb8d2c56b1540c8f7dffeec157a4d4fabeb65041f488140 WatchSource:0}: Error finding container 40caad49bb65bbbdafb8d2c56b1540c8f7dffeec157a4d4fabeb65041f488140: Status 404 returned error can't find the container with id 40caad49bb65bbbdafb8d2c56b1540c8f7dffeec157a4d4fabeb65041f488140
Sep 18 19:38:00.945935 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:00.945901    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-842cm\" (UniqueName: \"kubernetes.io/projected/0e5d82f6-4e09-448b-b9b7-8644ea3803a3-kube-api-access-842cm\") pod \"network-check-target-67bgb\" (UID: \"0e5d82f6-4e09-448b-b9b7-8644ea3803a3\") " pod="openshift-network-diagnostics/network-check-target-67bgb"
Sep 18 19:38:00.946067 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:00.945964    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1173b48c-041e-4ec6-9708-98391b2bf0fe-metrics-certs\") pod \"network-metrics-daemon-s9j6n\" (UID: \"1173b48c-041e-4ec6-9708-98391b2bf0fe\") " pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:38:00.946121 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:00.946070    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:00.946121 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:00.946075    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:00.946121 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:00.946095    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:00.946121 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:00.946106    2581 projected.go:196] Error preparing data for projected volume kube-api-access-842cm for pod openshift-network-diagnostics/network-check-target-67bgb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:00.946245 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:00.946133    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1173b48c-041e-4ec6-9708-98391b2bf0fe-metrics-certs podName:1173b48c-041e-4ec6-9708-98391b2bf0fe nodeName:}" failed. No retries permitted until 2026-09-18 19:38:02.946111848 +0000 UTC m=+6.048400180 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1173b48c-041e-4ec6-9708-98391b2bf0fe-metrics-certs") pod "network-metrics-daemon-s9j6n" (UID: "1173b48c-041e-4ec6-9708-98391b2bf0fe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:00.946245 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:00.946163    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e5d82f6-4e09-448b-b9b7-8644ea3803a3-kube-api-access-842cm podName:0e5d82f6-4e09-448b-b9b7-8644ea3803a3 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:02.946145572 +0000 UTC m=+6.048433885 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-842cm" (UniqueName: "kubernetes.io/projected/0e5d82f6-4e09-448b-b9b7-8644ea3803a3-kube-api-access-842cm") pod "network-check-target-67bgb" (UID: "0e5d82f6-4e09-448b-b9b7-8644ea3803a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:01.047368 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:01.047233    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d3541d3-fb52-42a5-b977-9cc44ac396bc-original-pull-secret\") pod \"global-pull-secret-syncer-2p2bp\" (UID: \"5d3541d3-fb52-42a5-b977-9cc44ac396bc\") " pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:38:01.047549 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:01.047392    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:01.047549 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:01.047459    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d3541d3-fb52-42a5-b977-9cc44ac396bc-original-pull-secret podName:5d3541d3-fb52-42a5-b977-9cc44ac396bc nodeName:}" failed. No retries permitted until 2026-09-18 19:38:03.047440259 +0000 UTC m=+6.149728588 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d3541d3-fb52-42a5-b977-9cc44ac396bc-original-pull-secret") pod "global-pull-secret-syncer-2p2bp" (UID: "5d3541d3-fb52-42a5-b977-9cc44ac396bc") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:01.221985 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:01.221878    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:01.363479 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:01.363370    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:33:00 +0000 UTC" deadline="2028-05-29 10:14:03.218113124 +0000 UTC"
Sep 18 19:38:01.363479 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:01.363480    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14846h36m1.854640398s"
Sep 18 19:38:01.427994 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:01.427968    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:38:01.428149 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:01.427999    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67bgb"
Sep 18 19:38:01.428149 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:01.428113    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9j6n" podUID="1173b48c-041e-4ec6-9708-98391b2bf0fe"
Sep 18 19:38:01.428418 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:01.428386    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67bgb" podUID="0e5d82f6-4e09-448b-b9b7-8644ea3803a3"
Sep 18 19:38:01.447358 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:01.447327    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s2vsh" event={"ID":"ee926faa-2422-4be9-82b5-ad1f0ec42ecd","Type":"ContainerStarted","Data":"8805d93fa82626f9502a3dd656b85dcf8b5240221c1dff38ea21b9c5906768b3"}
Sep 18 19:38:01.469741 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:01.469710    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4j6th" event={"ID":"697bb62f-6332-4a66-a1eb-84aa42f6a620","Type":"ContainerStarted","Data":"38c6faf471bcbeaee00bb4f3666a6eb5cc153030a580622d0f9ca11a93c69868"}
Sep 18 19:38:01.471597 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:01.471534    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7nrfp" event={"ID":"68f35ba9-52cf-4a53-bc81-b2863eeb0930","Type":"ContainerStarted","Data":"40caad49bb65bbbdafb8d2c56b1540c8f7dffeec157a4d4fabeb65041f488140"}
Sep 18 19:38:01.487288 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:01.487209    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzsk4" event={"ID":"cff0cd8b-6053-42d8-9b1e-20930e49616b","Type":"ContainerStarted","Data":"25d9f297cbdc71c30114e7cebdf1aab5618ad12bcc398ab34c904a7f400dc8f0"}
Sep 18 19:38:01.492244 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:01.492210    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.ec2.internal" event={"ID":"dff242b51bfc62e6a56c5d2723611cd5","Type":"ContainerStarted","Data":"db4c6dd8cc7df60b5f344b04f170c3e849d791fbefd48fb32452f7481aed1087"}
Sep 18 19:38:01.504106 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:01.504078    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jb5" event={"ID":"466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f","Type":"ContainerStarted","Data":"6a58ffae4d0a9e75d2c7690aee9eaa5c8d1a5594a4c2692e9ccee8f36045e8db"}
Sep 18 19:38:01.514046 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:01.514018    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2wj49" event={"ID":"a40538a7-7d46-4051-9292-9827fd5cb8af","Type":"ContainerStarted","Data":"6b4dbca06c385301e1a6286dada858431df9bd711dc4bf822bbed03f372e9697"}
Sep 18 19:38:01.529939 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:01.529849    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w9dlm" event={"ID":"58fa049b-20bb-4466-bb38-8b46448db93b","Type":"ContainerStarted","Data":"1735c5b07cc090e2dec1257f9dcae22738074cd0b09ecd8f28651d1afec332f4"}
Sep 18 19:38:01.539817 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:01.539748    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cg6mk" event={"ID":"6ff14ef8-6eac-448b-b297-b1fb15d59eac","Type":"ContainerStarted","Data":"3666e64385bdcb655e3a2196b3c78e498b42ba50608363086bcf041246d61a9b"}
Sep 18 19:38:01.548996 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:01.548965    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lld5l" event={"ID":"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e","Type":"ContainerStarted","Data":"1ca75787b3ab223c566bbb143a0c4c0633709d76d08db4ca470a04c54704a4ee"}
Sep 18 19:38:01.567300 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:01.566766    2581 generic.go:356] "Generic (PLEG): container finished" podID="18134cd66d519f96be48d8125235fba9" containerID="f0ae17d4f9ebbb64cb0f96c53d08be9ec245bedfe1506d63e39c8ff06e760774" exitCode=0
Sep 18 19:38:01.567300 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:01.566802    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.ec2.internal" event={"ID":"18134cd66d519f96be48d8125235fba9","Type":"ContainerDied","Data":"f0ae17d4f9ebbb64cb0f96c53d08be9ec245bedfe1506d63e39c8ff06e760774"}
Sep 18 19:38:01.583274 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:01.583191    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-134.ec2.internal" podStartSLOduration=2.583041667 podStartE2EDuration="2.583041667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:37:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:01.50941684 +0000 UTC m=+4.611705173" watchObservedRunningTime="2026-09-18 19:38:01.583041667 +0000 UTC m=+4.685330006"
Sep 18 19:38:02.427866 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:02.427589    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:38:02.428388 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:02.428354    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2p2bp" podUID="5d3541d3-fb52-42a5-b977-9cc44ac396bc"
Sep 18 19:38:02.586042 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:02.585984    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.ec2.internal" event={"ID":"18134cd66d519f96be48d8125235fba9","Type":"ContainerStarted","Data":"a82c1119bb2a8ea7fedfe5d78b4ca02766e97f11df8a90b0cd1a8199328e2951"}
Sep 18 19:38:02.964757 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:02.964717    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-842cm\" (UniqueName: \"kubernetes.io/projected/0e5d82f6-4e09-448b-b9b7-8644ea3803a3-kube-api-access-842cm\") pod \"network-check-target-67bgb\" (UID: \"0e5d82f6-4e09-448b-b9b7-8644ea3803a3\") " pod="openshift-network-diagnostics/network-check-target-67bgb"
Sep 18 19:38:02.964982 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:02.964810    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1173b48c-041e-4ec6-9708-98391b2bf0fe-metrics-certs\") pod \"network-metrics-daemon-s9j6n\" (UID: \"1173b48c-041e-4ec6-9708-98391b2bf0fe\") " pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:38:02.964982 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:02.964955    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:02.965100 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:02.965023    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1173b48c-041e-4ec6-9708-98391b2bf0fe-metrics-certs podName:1173b48c-041e-4ec6-9708-98391b2bf0fe nodeName:}" failed. No retries permitted until 2026-09-18 19:38:06.965001568 +0000 UTC m=+10.067289935 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1173b48c-041e-4ec6-9708-98391b2bf0fe-metrics-certs") pod "network-metrics-daemon-s9j6n" (UID: "1173b48c-041e-4ec6-9708-98391b2bf0fe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:02.965468 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:02.965445    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:02.965565 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:02.965477    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:02.965565 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:02.965491    2581 projected.go:196] Error preparing data for projected volume kube-api-access-842cm for pod openshift-network-diagnostics/network-check-target-67bgb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:02.965565 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:02.965538    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e5d82f6-4e09-448b-b9b7-8644ea3803a3-kube-api-access-842cm podName:0e5d82f6-4e09-448b-b9b7-8644ea3803a3 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:06.9655235 +0000 UTC m=+10.067811810 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-842cm" (UniqueName: "kubernetes.io/projected/0e5d82f6-4e09-448b-b9b7-8644ea3803a3-kube-api-access-842cm") pod "network-check-target-67bgb" (UID: "0e5d82f6-4e09-448b-b9b7-8644ea3803a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:03.065476 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:03.065437    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d3541d3-fb52-42a5-b977-9cc44ac396bc-original-pull-secret\") pod \"global-pull-secret-syncer-2p2bp\" (UID: \"5d3541d3-fb52-42a5-b977-9cc44ac396bc\") " pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:38:03.065667 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:03.065561    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:03.065667 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:03.065626    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d3541d3-fb52-42a5-b977-9cc44ac396bc-original-pull-secret podName:5d3541d3-fb52-42a5-b977-9cc44ac396bc nodeName:}" failed. No retries permitted until 2026-09-18 19:38:07.065605798 +0000 UTC m=+10.167894130 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d3541d3-fb52-42a5-b977-9cc44ac396bc-original-pull-secret") pod "global-pull-secret-syncer-2p2bp" (UID: "5d3541d3-fb52-42a5-b977-9cc44ac396bc") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:03.430276 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:03.430238    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:38:03.430948 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:03.430399    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9j6n" podUID="1173b48c-041e-4ec6-9708-98391b2bf0fe"
Sep 18 19:38:03.430948 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:03.430851    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67bgb"
Sep 18 19:38:03.431062 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:03.430971    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67bgb" podUID="0e5d82f6-4e09-448b-b9b7-8644ea3803a3"
Sep 18 19:38:04.428152 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:04.427648    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:38:04.428152 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:04.427790    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2p2bp" podUID="5d3541d3-fb52-42a5-b977-9cc44ac396bc"
Sep 18 19:38:05.428005 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:05.427971    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:38:05.428472 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:05.428134    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9j6n" podUID="1173b48c-041e-4ec6-9708-98391b2bf0fe"
Sep 18 19:38:05.428472 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:05.428189    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67bgb"
Sep 18 19:38:05.428472 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:05.428365    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67bgb" podUID="0e5d82f6-4e09-448b-b9b7-8644ea3803a3"
Sep 18 19:38:06.427209 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:06.427170    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:38:06.427362 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:06.427341    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2p2bp" podUID="5d3541d3-fb52-42a5-b977-9cc44ac396bc"
Sep 18 19:38:06.997224 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:06.997178    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1173b48c-041e-4ec6-9708-98391b2bf0fe-metrics-certs\") pod \"network-metrics-daemon-s9j6n\" (UID: \"1173b48c-041e-4ec6-9708-98391b2bf0fe\") " pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:38:06.997709 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:06.997236    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-842cm\" (UniqueName: \"kubernetes.io/projected/0e5d82f6-4e09-448b-b9b7-8644ea3803a3-kube-api-access-842cm\") pod \"network-check-target-67bgb\" (UID: \"0e5d82f6-4e09-448b-b9b7-8644ea3803a3\") " pod="openshift-network-diagnostics/network-check-target-67bgb"
Sep 18 19:38:06.997709 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:06.997392    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:06.997709 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:06.997414    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:06.997709 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:06.997429    2581 projected.go:196] Error preparing data for projected volume kube-api-access-842cm for pod openshift-network-diagnostics/network-check-target-67bgb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:06.997709 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:06.997493    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e5d82f6-4e09-448b-b9b7-8644ea3803a3-kube-api-access-842cm podName:0e5d82f6-4e09-448b-b9b7-8644ea3803a3 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:14.997471471 +0000 UTC m=+18.099759800 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-842cm" (UniqueName: "kubernetes.io/projected/0e5d82f6-4e09-448b-b9b7-8644ea3803a3-kube-api-access-842cm") pod "network-check-target-67bgb" (UID: "0e5d82f6-4e09-448b-b9b7-8644ea3803a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:06.997952 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:06.997884    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:06.997998 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:06.997956    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1173b48c-041e-4ec6-9708-98391b2bf0fe-metrics-certs podName:1173b48c-041e-4ec6-9708-98391b2bf0fe nodeName:}" failed. No retries permitted until 2026-09-18 19:38:14.997938901 +0000 UTC m=+18.100227212 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1173b48c-041e-4ec6-9708-98391b2bf0fe-metrics-certs") pod "network-metrics-daemon-s9j6n" (UID: "1173b48c-041e-4ec6-9708-98391b2bf0fe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:07.099029 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:07.098402    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d3541d3-fb52-42a5-b977-9cc44ac396bc-original-pull-secret\") pod \"global-pull-secret-syncer-2p2bp\" (UID: \"5d3541d3-fb52-42a5-b977-9cc44ac396bc\") " pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:38:07.099029 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:07.098594    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:07.099029 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:07.098662    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d3541d3-fb52-42a5-b977-9cc44ac396bc-original-pull-secret podName:5d3541d3-fb52-42a5-b977-9cc44ac396bc nodeName:}" failed. No retries permitted until 2026-09-18 19:38:15.098641917 +0000 UTC m=+18.200930227 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d3541d3-fb52-42a5-b977-9cc44ac396bc-original-pull-secret") pod "global-pull-secret-syncer-2p2bp" (UID: "5d3541d3-fb52-42a5-b977-9cc44ac396bc") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:07.428036 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:07.428005    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:38:07.428205 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:07.428123    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9j6n" podUID="1173b48c-041e-4ec6-9708-98391b2bf0fe"
Sep 18 19:38:07.428572 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:07.428552    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67bgb"
Sep 18 19:38:07.428682 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:07.428647    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67bgb" podUID="0e5d82f6-4e09-448b-b9b7-8644ea3803a3"
Sep 18 19:38:08.427072 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:08.427033    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:38:08.427570 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:08.427176    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2p2bp" podUID="5d3541d3-fb52-42a5-b977-9cc44ac396bc"
Sep 18 19:38:09.426886 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:09.426794    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67bgb"
Sep 18 19:38:09.427046 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:09.426794    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:38:09.427046 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:09.426948    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67bgb" podUID="0e5d82f6-4e09-448b-b9b7-8644ea3803a3"
Sep 18 19:38:09.427046 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:09.426999    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9j6n" podUID="1173b48c-041e-4ec6-9708-98391b2bf0fe"
Sep 18 19:38:10.427341 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:10.427310    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:38:10.427719 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:10.427413    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2p2bp" podUID="5d3541d3-fb52-42a5-b977-9cc44ac396bc"
Sep 18 19:38:11.427400 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:11.427371    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:38:11.427797 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:11.427404    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67bgb"
Sep 18 19:38:11.427797 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:11.427512    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9j6n" podUID="1173b48c-041e-4ec6-9708-98391b2bf0fe"
Sep 18 19:38:11.427797 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:11.427620    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67bgb" podUID="0e5d82f6-4e09-448b-b9b7-8644ea3803a3"
Sep 18 19:38:12.426707 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:12.426671    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:38:12.426869 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:12.426796    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2p2bp" podUID="5d3541d3-fb52-42a5-b977-9cc44ac396bc"
Sep 18 19:38:13.426944 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:13.426905    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:38:13.427373 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:13.426959    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67bgb"
Sep 18 19:38:13.427373 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:13.427043    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9j6n" podUID="1173b48c-041e-4ec6-9708-98391b2bf0fe"
Sep 18 19:38:13.427373 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:13.427169    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67bgb" podUID="0e5d82f6-4e09-448b-b9b7-8644ea3803a3"
Sep 18 19:38:14.427194 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:14.427162    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:38:14.427640 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:14.427284    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2p2bp" podUID="5d3541d3-fb52-42a5-b977-9cc44ac396bc"
Sep 18 19:38:15.061261 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:15.061225    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1173b48c-041e-4ec6-9708-98391b2bf0fe-metrics-certs\") pod \"network-metrics-daemon-s9j6n\" (UID: \"1173b48c-041e-4ec6-9708-98391b2bf0fe\") " pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:38:15.061433 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:15.061285    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-842cm\" (UniqueName: \"kubernetes.io/projected/0e5d82f6-4e09-448b-b9b7-8644ea3803a3-kube-api-access-842cm\") pod \"network-check-target-67bgb\" (UID: \"0e5d82f6-4e09-448b-b9b7-8644ea3803a3\") " pod="openshift-network-diagnostics/network-check-target-67bgb"
Sep 18 19:38:15.061433 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:15.061386    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:15.061433 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:15.061399    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:15.061433 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:15.061415    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:15.061433 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:15.061424    2581 projected.go:196] Error preparing data for projected volume kube-api-access-842cm for pod openshift-network-diagnostics/network-check-target-67bgb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:15.061687 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:15.061460    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1173b48c-041e-4ec6-9708-98391b2bf0fe-metrics-certs podName:1173b48c-041e-4ec6-9708-98391b2bf0fe nodeName:}" failed. No retries permitted until 2026-09-18 19:38:31.061441179 +0000 UTC m=+34.163729489 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1173b48c-041e-4ec6-9708-98391b2bf0fe-metrics-certs") pod "network-metrics-daemon-s9j6n" (UID: "1173b48c-041e-4ec6-9708-98391b2bf0fe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:15.061687 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:15.061481    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e5d82f6-4e09-448b-b9b7-8644ea3803a3-kube-api-access-842cm podName:0e5d82f6-4e09-448b-b9b7-8644ea3803a3 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:31.061471734 +0000 UTC m=+34.163760041 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-842cm" (UniqueName: "kubernetes.io/projected/0e5d82f6-4e09-448b-b9b7-8644ea3803a3-kube-api-access-842cm") pod "network-check-target-67bgb" (UID: "0e5d82f6-4e09-448b-b9b7-8644ea3803a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:15.161975 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:15.161936    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d3541d3-fb52-42a5-b977-9cc44ac396bc-original-pull-secret\") pod \"global-pull-secret-syncer-2p2bp\" (UID: \"5d3541d3-fb52-42a5-b977-9cc44ac396bc\") " pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:38:15.162156 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:15.162119    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:15.162223 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:15.162200    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d3541d3-fb52-42a5-b977-9cc44ac396bc-original-pull-secret podName:5d3541d3-fb52-42a5-b977-9cc44ac396bc nodeName:}" failed. No retries permitted until 2026-09-18 19:38:31.162179682 +0000 UTC m=+34.264468005 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d3541d3-fb52-42a5-b977-9cc44ac396bc-original-pull-secret") pod "global-pull-secret-syncer-2p2bp" (UID: "5d3541d3-fb52-42a5-b977-9cc44ac396bc") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:15.427207 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:15.427138    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67bgb"
Sep 18 19:38:15.427528 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:15.427138    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:38:15.427528 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:15.427260    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67bgb" podUID="0e5d82f6-4e09-448b-b9b7-8644ea3803a3"
Sep 18 19:38:15.427528 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:15.427294    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9j6n" podUID="1173b48c-041e-4ec6-9708-98391b2bf0fe"
Sep 18 19:38:16.427335 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:16.427308    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:38:16.427713 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:16.427415    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2p2bp" podUID="5d3541d3-fb52-42a5-b977-9cc44ac396bc"
Sep 18 19:38:17.427487 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:17.427460    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:38:17.427938 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:17.427572    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9j6n" podUID="1173b48c-041e-4ec6-9708-98391b2bf0fe"
Sep 18 19:38:17.427938 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:17.427655    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67bgb"
Sep 18 19:38:17.427938 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:17.427758    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67bgb" podUID="0e5d82f6-4e09-448b-b9b7-8644ea3803a3"
Sep 18 19:38:18.427588 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:18.427551    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:38:18.428018 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:18.427677    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2p2bp" podUID="5d3541d3-fb52-42a5-b977-9cc44ac396bc"
Sep 18 19:38:19.427688 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:19.427651    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67bgb"
Sep 18 19:38:19.428661 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:19.427691    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:38:19.428661 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:19.427785    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67bgb" podUID="0e5d82f6-4e09-448b-b9b7-8644ea3803a3"
Sep 18 19:38:19.428661 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:19.427929    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9j6n" podUID="1173b48c-041e-4ec6-9708-98391b2bf0fe"
Sep 18 19:38:19.623555 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:19.623527    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cg6mk" event={"ID":"6ff14ef8-6eac-448b-b297-b1fb15d59eac","Type":"ContainerStarted","Data":"b864b0e49f2a284d3e7148b8bc64fad45d1116c2827a6e82bc1947792c6de422"}
Sep 18 19:38:19.624936 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:19.624916    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lld5l" event={"ID":"2480d7a4-45ca-4ec7-ad05-9ef85b6fff0e","Type":"ContainerStarted","Data":"bc343dd733a34822fce8a14ed01b216639d65bf8c32f38eb4929134187d9bd9f"}
Sep 18 19:38:19.626505 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:19.626473    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4j6th" event={"ID":"697bb62f-6332-4a66-a1eb-84aa42f6a620","Type":"ContainerStarted","Data":"c2730820ea3761cf3dae4653f67f82fa18d2f4ac35744b29ad1990a9fcb64ed4"}
Sep 18 19:38:19.629502 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:19.629477    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7nrfp" event={"ID":"68f35ba9-52cf-4a53-bc81-b2863eeb0930","Type":"ContainerStarted","Data":"7e1132468d1d84abe4ae55e09546d7f2eff7e6cf6f3f34170111cec3d567ff8f"}
Sep 18 19:38:19.631114 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:19.631093    2581 generic.go:356] "Generic (PLEG): container finished" podID="cff0cd8b-6053-42d8-9b1e-20930e49616b" containerID="12ce9d7b79df96ba93573ac94d13d70b9ed14fee50b818df3d5af6001e935b2b" exitCode=0
Sep 18 19:38:19.631208 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:19.631156    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzsk4" event={"ID":"cff0cd8b-6053-42d8-9b1e-20930e49616b","Type":"ContainerDied","Data":"12ce9d7b79df96ba93573ac94d13d70b9ed14fee50b818df3d5af6001e935b2b"}
Sep 18 19:38:19.632830 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:19.632797    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jb5" event={"ID":"466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f","Type":"ContainerStarted","Data":"20db3b017f307afed83b3efebe73ecdd9ca6724135e9ced925540794c3a63d96"}
Sep 18 19:38:19.636617 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:19.635866    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2wj49" event={"ID":"a40538a7-7d46-4051-9292-9827fd5cb8af","Type":"ContainerStarted","Data":"76a7814599bc44fdfcc5e307e0b22736d48422e36c260d8399c8cdfa50e29ae7"}
Sep 18 19:38:19.636617 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:19.635952    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2wj49" event={"ID":"a40538a7-7d46-4051-9292-9827fd5cb8af","Type":"ContainerStarted","Data":"3f9f5dbfd8ae8d0fcd1a6a2407c71b98abb35b1cbcd948e08fa960c673e46477"}
Sep 18 19:38:19.636617 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:19.636009    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2wj49" event={"ID":"a40538a7-7d46-4051-9292-9827fd5cb8af","Type":"ContainerStarted","Data":"b855cb59d8d22d1cacb202eed00ffae0a7cec02fc208c4be0c712eca85202225"}
Sep 18 19:38:19.636617 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:19.636060    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2wj49" event={"ID":"a40538a7-7d46-4051-9292-9827fd5cb8af","Type":"ContainerStarted","Data":"f68069b102b603e06b1ab3dec2115c2bc5634652ebc309f4040ae9c67f2e7181"}
Sep 18 19:38:19.636617 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:19.636108    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2wj49" event={"ID":"a40538a7-7d46-4051-9292-9827fd5cb8af","Type":"ContainerStarted","Data":"4fe0ce9599d1b4e89fee7348b887ae10546aac36d8d28cbdaac852e2a4f51dce"}
Sep 18 19:38:19.637299 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:19.637208    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cg6mk" podStartSLOduration=3.1241944249999998 podStartE2EDuration="21.637193026s" totalImagesPullingTime="18.512998601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:19.637128792 +0000 UTC m=+22.739417121" watchObservedRunningTime="2026-09-18 19:38:19.637193026 +0000 UTC m=+22.739481354"
Sep 18 19:38:19.637427 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:19.637407    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-134.ec2.internal" podStartSLOduration=20.637398775 podStartE2EDuration="20.637398775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:37:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:02.603642537 +0000 UTC m=+5.705930871" watchObservedRunningTime="2026-09-18 19:38:19.637398775 +0000 UTC m=+22.739687103"
Sep 18 19:38:19.637512 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:19.637476    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w9dlm" event={"ID":"58fa049b-20bb-4466-bb38-8b46448db93b","Type":"ContainerStarted","Data":"e1c80792774e424c5faac838c089fad310db9d40c1209e7198681f43b8874753"}
Sep 18 19:38:19.671915 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:19.671871    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4j6th" podStartSLOduration=3.088607574 podStartE2EDuration="21.671856286s" totalImagesPullingTime="18.583248712s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:19.671777373 +0000 UTC m=+22.774065701" watchObservedRunningTime="2026-09-18 19:38:19.671856286 +0000 UTC m=+22.774144609"
Sep 18 19:38:19.695943 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:19.695892    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lld5l" podStartSLOduration=3.133739384 podStartE2EDuration="21.695877882s" totalImagesPullingTime="18.562138498s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:19.695719299 +0000 UTC m=+22.798007628" watchObservedRunningTime="2026-09-18 19:38:19.695877882 +0000 UTC m=+22.798166210"
Sep 18 19:38:19.708946 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:19.708906    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7nrfp" podStartSLOduration=3.181931378 podStartE2EDuration="21.708888754s" totalImagesPullingTime="18.526957376s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:19.708794376 +0000 UTC m=+22.811082704" watchObservedRunningTime="2026-09-18 19:38:19.708888754 +0000 UTC m=+22.811177081"
Sep 18 19:38:20.427827 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:20.427629    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:38:20.428252 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:20.427940    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2p2bp" podUID="5d3541d3-fb52-42a5-b977-9cc44ac396bc"
Sep 18 19:38:20.514723 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:20.514700    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 19:38:20.640967 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:20.640865    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s2vsh" event={"ID":"ee926faa-2422-4be9-82b5-ad1f0ec42ecd","Type":"ContainerStarted","Data":"84cc1c7fbfd4939293e6b9a82dd079f6751106d6ad0e164bc4724134f70c178b"}
Sep 18 19:38:20.642698 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:20.642674    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jb5" event={"ID":"466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f","Type":"ContainerStarted","Data":"c32d48f137fc5129bf9bae7d0145110e364cde4a38b16e6713642915ef388645"}
Sep 18 19:38:20.645755 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:20.645724    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2wj49" event={"ID":"a40538a7-7d46-4051-9292-9827fd5cb8af","Type":"ContainerStarted","Data":"c303524aa53c672af6da451e69b4a0bf94e552b9ed07dc0fd20b7e04507654d5"}
Sep 18 19:38:20.657115 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:20.657064    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-w9dlm" podStartSLOduration=4.120378055 podStartE2EDuration="22.65705204s" totalImagesPullingTime="18.536673985s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:19.731350613 +0000 UTC m=+22.833638940" watchObservedRunningTime="2026-09-18 19:38:20.65705204 +0000 UTC m=+23.759340368"
Sep 18 19:38:20.657530 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:20.657506    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-s2vsh" podStartSLOduration=4.130126058 podStartE2EDuration="22.657499677s" totalImagesPullingTime="18.527373619s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:20.657008742 +0000 UTC m=+23.759297070" watchObservedRunningTime="2026-09-18 19:38:20.657499677 +0000 UTC m=+23.759788005"
Sep 18 19:38:21.398269 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:21.398146    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T19:38:20.514719669Z","UUID":"0f074ca8-5b71-49f6-82c5-56e4e7f21345","Handler":null,"Name":"","Endpoint":""}
Sep 18 19:38:21.399813 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:21.399787    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 19:38:21.399813 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:21.399818    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 19:38:21.427439 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:21.427413    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:38:21.427595 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:21.427448    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67bgb"
Sep 18 19:38:21.427595 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:21.427513    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9j6n" podUID="1173b48c-041e-4ec6-9708-98391b2bf0fe"
Sep 18 19:38:21.427721 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:21.427623    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67bgb" podUID="0e5d82f6-4e09-448b-b9b7-8644ea3803a3"
Sep 18 19:38:21.651004 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:21.650929    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jb5" event={"ID":"466b6af3-c62d-4cb1-ab20-6a3fa7e35a2f","Type":"ContainerStarted","Data":"331e98e37407608cd67bf5f589faa3e4654052f24832df4b85b9a7383633ddf5"}
Sep 18 19:38:21.668438 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:21.668394    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jb5" podStartSLOduration=3.041295306 podStartE2EDuration="23.668379926s" totalImagesPullingTime="20.62708462s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:37:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:21.66778272 +0000 UTC m=+24.770071047" watchObservedRunningTime="2026-09-18 19:38:21.668379926 +0000 UTC m=+24.770668253"
Sep 18 19:38:22.427761 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:22.427733    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:38:22.427953 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:22.427846    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2p2bp" podUID="5d3541d3-fb52-42a5-b977-9cc44ac396bc"
Sep 18 19:38:22.656746 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:22.656704    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2wj49" event={"ID":"a40538a7-7d46-4051-9292-9827fd5cb8af","Type":"ContainerStarted","Data":"86b9ff947807e5fc7da741d89eefe99677ced38d491beb466695f02aa833e98e"}
Sep 18 19:38:23.431745 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:23.431714    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67bgb"
Sep 18 19:38:23.431926 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:23.431754    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:38:23.431926 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:23.431898    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67bgb" podUID="0e5d82f6-4e09-448b-b9b7-8644ea3803a3"
Sep 18 19:38:23.432419 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:23.432386    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9j6n" podUID="1173b48c-041e-4ec6-9708-98391b2bf0fe"
Sep 18 19:38:23.957164 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:23.957128    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cg6mk"
Sep 18 19:38:23.958078 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:23.958056    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cg6mk"
Sep 18 19:38:24.427550 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:24.427378    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:38:24.427722 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:24.427640    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2p2bp" podUID="5d3541d3-fb52-42a5-b977-9cc44ac396bc"
Sep 18 19:38:24.662869 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:24.662815    2581 generic.go:356] "Generic (PLEG): container finished" podID="cff0cd8b-6053-42d8-9b1e-20930e49616b" containerID="4976fb0a85bd896f175b5a3998660fda05536cafe88b0a0ddabb7a109373b81a" exitCode=0
Sep 18 19:38:24.663010 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:24.662897    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzsk4" event={"ID":"cff0cd8b-6053-42d8-9b1e-20930e49616b","Type":"ContainerDied","Data":"4976fb0a85bd896f175b5a3998660fda05536cafe88b0a0ddabb7a109373b81a"}
Sep 18 19:38:24.666885 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:24.666858    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2wj49" event={"ID":"a40538a7-7d46-4051-9292-9827fd5cb8af","Type":"ContainerStarted","Data":"ac449e35bd910f59bc169d167a04111f967c557acdc1887ee3accd36f426c282"}
Sep 18 19:38:24.667095 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:24.667065    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cg6mk"
Sep 18 19:38:24.667484 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:24.667470    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cg6mk"
Sep 18 19:38:24.718868 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:24.718810    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2wj49" podStartSLOduration=8.11826752 podStartE2EDuration="26.718797593s" totalImagesPullingTime="18.600530073s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:24.717546007 +0000 UTC m=+27.819834334" watchObservedRunningTime="2026-09-18 19:38:24.718797593 +0000 UTC m=+27.821085919"
Sep 18 19:38:25.429994 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:25.429966    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67bgb"
Sep 18 19:38:25.429994 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:25.429979    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:38:25.430756 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:25.430057    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67bgb" podUID="0e5d82f6-4e09-448b-b9b7-8644ea3803a3"
Sep 18 19:38:25.430756 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:25.430192    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9j6n" podUID="1173b48c-041e-4ec6-9708-98391b2bf0fe"
Sep 18 19:38:25.669422 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:25.669391    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:38:25.669635 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:25.669464    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:38:25.669635 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:25.669508    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:38:25.687022 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:25.686967    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:38:25.697257 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:25.697231    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:38:26.156757 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:26.155903    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2p2bp"]
Sep 18 19:38:26.156757 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:26.156216    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:38:26.156757 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:26.156289    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2p2bp" podUID="5d3541d3-fb52-42a5-b977-9cc44ac396bc"
Sep 18 19:38:26.160121 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:26.160087    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s9j6n"]
Sep 18 19:38:26.160223 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:26.160163    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-67bgb"]
Sep 18 19:38:26.160297 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:26.160254    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67bgb"
Sep 18 19:38:26.160456 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:26.160419    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67bgb" podUID="0e5d82f6-4e09-448b-b9b7-8644ea3803a3"
Sep 18 19:38:26.160549 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:26.160484    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:38:26.160606 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:26.160571    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9j6n" podUID="1173b48c-041e-4ec6-9708-98391b2bf0fe"
Sep 18 19:38:26.672513 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:26.672481    2581 generic.go:356] "Generic (PLEG): container finished" podID="cff0cd8b-6053-42d8-9b1e-20930e49616b" containerID="7236e498c350790707175883fef60a2ce5d1c1b0b49d5cc8ad1b2d237d405fc8" exitCode=0
Sep 18 19:38:26.672904 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:26.672561    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzsk4" event={"ID":"cff0cd8b-6053-42d8-9b1e-20930e49616b","Type":"ContainerDied","Data":"7236e498c350790707175883fef60a2ce5d1c1b0b49d5cc8ad1b2d237d405fc8"}
Sep 18 19:38:27.432091 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:27.432062    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67bgb"
Sep 18 19:38:27.432243 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:27.432150    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67bgb" podUID="0e5d82f6-4e09-448b-b9b7-8644ea3803a3"
Sep 18 19:38:28.427263 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:28.427233    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:38:28.427263 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:28.427262    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:38:28.427682 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:28.427367    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9j6n" podUID="1173b48c-041e-4ec6-9708-98391b2bf0fe"
Sep 18 19:38:28.427682 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:28.427522    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2p2bp" podUID="5d3541d3-fb52-42a5-b977-9cc44ac396bc"
Sep 18 19:38:28.679433 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:28.679363    2581 generic.go:356] "Generic (PLEG): container finished" podID="cff0cd8b-6053-42d8-9b1e-20930e49616b" containerID="f2b929ae7283c5d064d6113b2e852f9132691f29104512385a8fa33e19c6b87d" exitCode=0
Sep 18 19:38:28.679433 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:28.679398    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzsk4" event={"ID":"cff0cd8b-6053-42d8-9b1e-20930e49616b","Type":"ContainerDied","Data":"f2b929ae7283c5d064d6113b2e852f9132691f29104512385a8fa33e19c6b87d"}
Sep 18 19:38:29.427401 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:29.427371    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67bgb"
Sep 18 19:38:29.427908 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:29.427477    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67bgb" podUID="0e5d82f6-4e09-448b-b9b7-8644ea3803a3"
Sep 18 19:38:30.427152 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:30.427121    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:38:30.427326 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:30.427121    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:38:30.427326 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:30.427242    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2p2bp" podUID="5d3541d3-fb52-42a5-b977-9cc44ac396bc"
Sep 18 19:38:30.427432 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:30.427341    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s9j6n" podUID="1173b48c-041e-4ec6-9708-98391b2bf0fe"
Sep 18 19:38:31.078980 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.078943    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1173b48c-041e-4ec6-9708-98391b2bf0fe-metrics-certs\") pod \"network-metrics-daemon-s9j6n\" (UID: \"1173b48c-041e-4ec6-9708-98391b2bf0fe\") " pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:38:31.079164 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.079000    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-842cm\" (UniqueName: \"kubernetes.io/projected/0e5d82f6-4e09-448b-b9b7-8644ea3803a3-kube-api-access-842cm\") pod \"network-check-target-67bgb\" (UID: \"0e5d82f6-4e09-448b-b9b7-8644ea3803a3\") " pod="openshift-network-diagnostics/network-check-target-67bgb"
Sep 18 19:38:31.079164 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:31.079114    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:31.079267 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:31.079177    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1173b48c-041e-4ec6-9708-98391b2bf0fe-metrics-certs podName:1173b48c-041e-4ec6-9708-98391b2bf0fe nodeName:}" failed. No retries permitted until 2026-09-18 19:39:03.079161957 +0000 UTC m=+66.181450267 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1173b48c-041e-4ec6-9708-98391b2bf0fe-metrics-certs") pod "network-metrics-daemon-s9j6n" (UID: "1173b48c-041e-4ec6-9708-98391b2bf0fe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:31.079267 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:31.079117    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:31.079267 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:31.079242    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:31.079267 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:31.079259    2581 projected.go:196] Error preparing data for projected volume kube-api-access-842cm for pod openshift-network-diagnostics/network-check-target-67bgb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:31.079426 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:31.079316    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e5d82f6-4e09-448b-b9b7-8644ea3803a3-kube-api-access-842cm podName:0e5d82f6-4e09-448b-b9b7-8644ea3803a3 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:03.079300105 +0000 UTC m=+66.181588415 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-842cm" (UniqueName: "kubernetes.io/projected/0e5d82f6-4e09-448b-b9b7-8644ea3803a3-kube-api-access-842cm") pod "network-check-target-67bgb" (UID: "0e5d82f6-4e09-448b-b9b7-8644ea3803a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:31.179723 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.179695    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d3541d3-fb52-42a5-b977-9cc44ac396bc-original-pull-secret\") pod \"global-pull-secret-syncer-2p2bp\" (UID: \"5d3541d3-fb52-42a5-b977-9cc44ac396bc\") " pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:38:31.179873 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:31.179817    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:31.179921 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:31.179883    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d3541d3-fb52-42a5-b977-9cc44ac396bc-original-pull-secret podName:5d3541d3-fb52-42a5-b977-9cc44ac396bc nodeName:}" failed. No retries permitted until 2026-09-18 19:39:03.179870023 +0000 UTC m=+66.282158328 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d3541d3-fb52-42a5-b977-9cc44ac396bc-original-pull-secret") pod "global-pull-secret-syncer-2p2bp" (UID: "5d3541d3-fb52-42a5-b977-9cc44ac396bc") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:31.430084 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.430016    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67bgb"
Sep 18 19:38:31.430454 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:31.430115    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67bgb" podUID="0e5d82f6-4e09-448b-b9b7-8644ea3803a3"
Sep 18 19:38:31.651024 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.650999    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-134.ec2.internal" event="NodeReady"
Sep 18 19:38:31.651157 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.651144    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 19:38:31.685328 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.685262    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-86f46f546-98c44"]
Sep 18 19:38:31.712801 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.712779    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dwtpk"]
Sep 18 19:38:31.713766 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.713735    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:38:31.716115 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.716084    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ntrxj\""
Sep 18 19:38:31.716265 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.716092    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 19:38:31.716368 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.716299    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 19:38:31.716627 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.716609    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 19:38:31.722248 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.722227    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 19:38:31.728686 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.728668    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fn944"]
Sep 18 19:38:31.728816 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.728794    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dwtpk"
Sep 18 19:38:31.731374 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.731248    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 19:38:31.731374 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.731279    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 19:38:31.731374 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.731338    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5f8sq\""
Sep 18 19:38:31.746727 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.746701    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86f46f546-98c44"]
Sep 18 19:38:31.746727 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.746728    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fn944"]
Sep 18 19:38:31.746907 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.746744    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dwtpk"]
Sep 18 19:38:31.746907 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.746857    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fn944"
Sep 18 19:38:31.750123 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.750104    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 19:38:31.750204 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.750170    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 19:38:31.750261 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.750244    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-q4x2f\""
Sep 18 19:38:31.750325 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.750250    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 19:38:31.886015 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.885977    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/82551de1-f8ab-432e-8ff6-1da685c07fa9-config-volume\") pod \"dns-default-dwtpk\" (UID: \"82551de1-f8ab-432e-8ff6-1da685c07fa9\") " pod="openshift-dns/dns-default-dwtpk"
Sep 18 19:38:31.886148 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.886021    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c5tj\" (UniqueName: \"kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-kube-api-access-7c5tj\") pod \"image-registry-86f46f546-98c44\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") " pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:38:31.886148 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.886043    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82551de1-f8ab-432e-8ff6-1da685c07fa9-metrics-tls\") pod \"dns-default-dwtpk\" (UID: \"82551de1-f8ab-432e-8ff6-1da685c07fa9\") " pod="openshift-dns/dns-default-dwtpk"
Sep 18 19:38:31.886148 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.886070    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmb9s\" (UniqueName: \"kubernetes.io/projected/82551de1-f8ab-432e-8ff6-1da685c07fa9-kube-api-access-vmb9s\") pod \"dns-default-dwtpk\" (UID: \"82551de1-f8ab-432e-8ff6-1da685c07fa9\") " pod="openshift-dns/dns-default-dwtpk"
Sep 18 19:38:31.886148 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.886104    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-installation-pull-secrets\") pod \"image-registry-86f46f546-98c44\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") " pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:38:31.886321 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.886154    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82551de1-f8ab-432e-8ff6-1da685c07fa9-tmp-dir\") pod \"dns-default-dwtpk\" (UID: \"82551de1-f8ab-432e-8ff6-1da685c07fa9\") " pod="openshift-dns/dns-default-dwtpk"
Sep 18 19:38:31.886321 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.886216    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-tls\") pod \"image-registry-86f46f546-98c44\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") " pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:38:31.886321 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.886289    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-ca-trust-extracted\") pod \"image-registry-86f46f546-98c44\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") " pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:38:31.886441 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.886352    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-image-registry-private-configuration\") pod \"image-registry-86f46f546-98c44\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") " pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:38:31.886441 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.886381    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e7a69286-c751-476a-a2fc-631e26030284-cert\") pod \"ingress-canary-fn944\" (UID: \"e7a69286-c751-476a-a2fc-631e26030284\") " pod="openshift-ingress-canary/ingress-canary-fn944"
Sep 18 19:38:31.886441 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.886408    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-certificates\") pod \"image-registry-86f46f546-98c44\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") " pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:38:31.886441 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.886436    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-trusted-ca\") pod \"image-registry-86f46f546-98c44\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") " pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:38:31.886580 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.886461    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-bound-sa-token\") pod \"image-registry-86f46f546-98c44\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") " pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:38:31.886580 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.886520    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmwwd\" (UniqueName: \"kubernetes.io/projected/e7a69286-c751-476a-a2fc-631e26030284-kube-api-access-lmwwd\") pod \"ingress-canary-fn944\" (UID: \"e7a69286-c751-476a-a2fc-631e26030284\") " pod="openshift-ingress-canary/ingress-canary-fn944"
Sep 18 19:38:31.987894 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.987784    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/82551de1-f8ab-432e-8ff6-1da685c07fa9-config-volume\") pod \"dns-default-dwtpk\" (UID: \"82551de1-f8ab-432e-8ff6-1da685c07fa9\") " pod="openshift-dns/dns-default-dwtpk"
Sep 18 19:38:31.987894 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.987875    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c5tj\" (UniqueName: \"kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-kube-api-access-7c5tj\") pod \"image-registry-86f46f546-98c44\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") " pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:38:31.988100 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.987903    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82551de1-f8ab-432e-8ff6-1da685c07fa9-metrics-tls\") pod \"dns-default-dwtpk\" (UID: \"82551de1-f8ab-432e-8ff6-1da685c07fa9\") " pod="openshift-dns/dns-default-dwtpk"
Sep 18 19:38:31.988100 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.987923    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmb9s\" (UniqueName: \"kubernetes.io/projected/82551de1-f8ab-432e-8ff6-1da685c07fa9-kube-api-access-vmb9s\") pod \"dns-default-dwtpk\" (UID: \"82551de1-f8ab-432e-8ff6-1da685c07fa9\") " pod="openshift-dns/dns-default-dwtpk"
Sep 18 19:38:31.988100 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.987947    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-installation-pull-secrets\") pod \"image-registry-86f46f546-98c44\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") " pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:38:31.988100 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.987966    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82551de1-f8ab-432e-8ff6-1da685c07fa9-tmp-dir\") pod \"dns-default-dwtpk\" (UID: \"82551de1-f8ab-432e-8ff6-1da685c07fa9\") " pod="openshift-dns/dns-default-dwtpk"
Sep 18 19:38:31.988100 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.988012    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-tls\") pod \"image-registry-86f46f546-98c44\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") " pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:38:31.988100 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.988051    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-ca-trust-extracted\") pod \"image-registry-86f46f546-98c44\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") " pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:38:31.988100 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:31.988056    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:38:31.988100 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.988090    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-image-registry-private-configuration\") pod \"image-registry-86f46f546-98c44\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") " pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:38:31.988363 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.988112    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e7a69286-c751-476a-a2fc-631e26030284-cert\") pod \"ingress-canary-fn944\" (UID: \"e7a69286-c751-476a-a2fc-631e26030284\") " pod="openshift-ingress-canary/ingress-canary-fn944"
Sep 18 19:38:31.988363 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:31.988135    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82551de1-f8ab-432e-8ff6-1da685c07fa9-metrics-tls podName:82551de1-f8ab-432e-8ff6-1da685c07fa9 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:32.488113947 +0000 UTC m=+35.590402259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/82551de1-f8ab-432e-8ff6-1da685c07fa9-metrics-tls") pod "dns-default-dwtpk" (UID: "82551de1-f8ab-432e-8ff6-1da685c07fa9") : secret "dns-default-metrics-tls" not found
Sep 18 19:38:31.988363 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.988175    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-certificates\") pod \"image-registry-86f46f546-98c44\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") " pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:38:31.988363 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:31.988198    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:38:31.988363 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.988204    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-trusted-ca\") pod \"image-registry-86f46f546-98c44\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") " pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:38:31.988363 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:31.988213    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86f46f546-98c44: secret "image-registry-tls" not found
Sep 18 19:38:31.988363 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.988227    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-bound-sa-token\") pod \"image-registry-86f46f546-98c44\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") " pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:38:31.988363 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.988268    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmwwd\" (UniqueName: \"kubernetes.io/projected/e7a69286-c751-476a-a2fc-631e26030284-kube-api-access-lmwwd\") pod \"ingress-canary-fn944\" (UID: \"e7a69286-c751-476a-a2fc-631e26030284\") " pod="openshift-ingress-canary/ingress-canary-fn944"
Sep 18 19:38:31.988363 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:31.988285    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-tls podName:80dda80c-b4fe-4ab8-9f8b-ef73f7b04756 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:32.488268209 +0000 UTC m=+35.590556515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-tls") pod "image-registry-86f46f546-98c44" (UID: "80dda80c-b4fe-4ab8-9f8b-ef73f7b04756") : secret "image-registry-tls" not found
Sep 18 19:38:31.988617 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.988420    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/82551de1-f8ab-432e-8ff6-1da685c07fa9-config-volume\") pod \"dns-default-dwtpk\" (UID: \"82551de1-f8ab-432e-8ff6-1da685c07fa9\") " pod="openshift-dns/dns-default-dwtpk"
Sep 18 19:38:31.988617 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.988516    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82551de1-f8ab-432e-8ff6-1da685c07fa9-tmp-dir\") pod \"dns-default-dwtpk\" (UID: \"82551de1-f8ab-432e-8ff6-1da685c07fa9\") " pod="openshift-dns/dns-default-dwtpk"
Sep 18 19:38:31.988709 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.988616    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-ca-trust-extracted\") pod \"image-registry-86f46f546-98c44\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") " pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:38:31.989047 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.989023    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-certificates\") pod \"image-registry-86f46f546-98c44\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") " pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:38:31.989451 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.989428    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-trusted-ca\") pod \"image-registry-86f46f546-98c44\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") " pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:38:31.992540 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.992521    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-image-registry-private-configuration\") pod \"image-registry-86f46f546-98c44\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") " pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:38:31.992540 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.992530    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-installation-pull-secrets\") pod \"image-registry-86f46f546-98c44\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") " pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:38:31.992689 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.992595    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e7a69286-c751-476a-a2fc-631e26030284-cert\") pod \"ingress-canary-fn944\" (UID: \"e7a69286-c751-476a-a2fc-631e26030284\") " pod="openshift-ingress-canary/ingress-canary-fn944"
Sep 18 19:38:31.999244 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:31.999216    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c5tj\" (UniqueName: \"kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-kube-api-access-7c5tj\") pod \"image-registry-86f46f546-98c44\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") " pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:38:32.002507 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:32.002482    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmwwd\" (UniqueName: \"kubernetes.io/projected/e7a69286-c751-476a-a2fc-631e26030284-kube-api-access-lmwwd\") pod \"ingress-canary-fn944\" (UID: \"e7a69286-c751-476a-a2fc-631e26030284\") " pod="openshift-ingress-canary/ingress-canary-fn944"
Sep 18 19:38:32.003427 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:32.003402    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmb9s\" (UniqueName: \"kubernetes.io/projected/82551de1-f8ab-432e-8ff6-1da685c07fa9-kube-api-access-vmb9s\") pod \"dns-default-dwtpk\" (UID: \"82551de1-f8ab-432e-8ff6-1da685c07fa9\") " pod="openshift-dns/dns-default-dwtpk"
Sep 18 19:38:32.003525 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:32.003475    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-bound-sa-token\") pod \"image-registry-86f46f546-98c44\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") " pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:38:32.055906 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:32.055877    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fn944"
Sep 18 19:38:32.426812 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:32.426744    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:38:32.426991 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:32.426752    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:38:32.429457 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:32.429437    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:38:32.429611 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:32.429439    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:38:32.430462 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:32.430439    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8c9m8\""
Sep 18 19:38:32.492462 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:32.492437    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82551de1-f8ab-432e-8ff6-1da685c07fa9-metrics-tls\") pod \"dns-default-dwtpk\" (UID: \"82551de1-f8ab-432e-8ff6-1da685c07fa9\") " pod="openshift-dns/dns-default-dwtpk"
Sep 18 19:38:32.492591 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:32.492506    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-tls\") pod \"image-registry-86f46f546-98c44\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") " pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:38:32.492591 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:32.492541    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:38:32.492703 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:32.492599    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82551de1-f8ab-432e-8ff6-1da685c07fa9-metrics-tls podName:82551de1-f8ab-432e-8ff6-1da685c07fa9 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:33.49258081 +0000 UTC m=+36.594869119 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/82551de1-f8ab-432e-8ff6-1da685c07fa9-metrics-tls") pod "dns-default-dwtpk" (UID: "82551de1-f8ab-432e-8ff6-1da685c07fa9") : secret "dns-default-metrics-tls" not found
Sep 18 19:38:32.492703 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:32.492665    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:38:32.492703 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:32.492680    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86f46f546-98c44: secret "image-registry-tls" not found
Sep 18 19:38:32.492864 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:32.492735    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-tls podName:80dda80c-b4fe-4ab8-9f8b-ef73f7b04756 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:33.492720762 +0000 UTC m=+36.595009080 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-tls") pod "image-registry-86f46f546-98c44" (UID: "80dda80c-b4fe-4ab8-9f8b-ef73f7b04756") : secret "image-registry-tls" not found
Sep 18 19:38:33.141889 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:33.141796    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fn944"]
Sep 18 19:38:33.144603 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:38:33.144576    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7a69286_c751_476a_a2fc_631e26030284.slice/crio-dd634341e890222bbf5259bfecdbfce1a0b75befea2473f843dbdcd7de398f1b WatchSource:0}: Error finding container dd634341e890222bbf5259bfecdbfce1a0b75befea2473f843dbdcd7de398f1b: Status 404 returned error can't find the container with id dd634341e890222bbf5259bfecdbfce1a0b75befea2473f843dbdcd7de398f1b
Sep 18 19:38:33.426868 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:33.426818    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67bgb"
Sep 18 19:38:33.429548 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:33.429530    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:38:33.430386 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:33.430367    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pwxm5\""
Sep 18 19:38:33.430473 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:33.430371    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:38:33.500271 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:33.500247    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-tls\") pod \"image-registry-86f46f546-98c44\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") " pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:38:33.500350 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:33.500307    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82551de1-f8ab-432e-8ff6-1da685c07fa9-metrics-tls\") pod \"dns-default-dwtpk\" (UID: \"82551de1-f8ab-432e-8ff6-1da685c07fa9\") " pod="openshift-dns/dns-default-dwtpk"
Sep 18 19:38:33.500460 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:33.500394    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:38:33.500460 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:33.500411    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:38:33.500460 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:33.500427    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86f46f546-98c44: secret "image-registry-tls" not found
Sep 18 19:38:33.500460 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:33.500444    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82551de1-f8ab-432e-8ff6-1da685c07fa9-metrics-tls podName:82551de1-f8ab-432e-8ff6-1da685c07fa9 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:35.500427356 +0000 UTC m=+38.602715663 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/82551de1-f8ab-432e-8ff6-1da685c07fa9-metrics-tls") pod "dns-default-dwtpk" (UID: "82551de1-f8ab-432e-8ff6-1da685c07fa9") : secret "dns-default-metrics-tls" not found
Sep 18 19:38:33.500617 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:33.500485    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-tls podName:80dda80c-b4fe-4ab8-9f8b-ef73f7b04756 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:35.500469032 +0000 UTC m=+38.602757346 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-tls") pod "image-registry-86f46f546-98c44" (UID: "80dda80c-b4fe-4ab8-9f8b-ef73f7b04756") : secret "image-registry-tls" not found
Sep 18 19:38:33.693400 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:33.693324    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fn944" event={"ID":"e7a69286-c751-476a-a2fc-631e26030284","Type":"ContainerStarted","Data":"dd634341e890222bbf5259bfecdbfce1a0b75befea2473f843dbdcd7de398f1b"}
Sep 18 19:38:33.696729 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:33.696701    2581 generic.go:356] "Generic (PLEG): container finished" podID="cff0cd8b-6053-42d8-9b1e-20930e49616b" containerID="c0395bafb09f480d82c69fd1ceb6283a6b036e4644e00aa2c4c7b194868c8733" exitCode=0
Sep 18 19:38:33.696879 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:33.696740    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzsk4" event={"ID":"cff0cd8b-6053-42d8-9b1e-20930e49616b","Type":"ContainerDied","Data":"c0395bafb09f480d82c69fd1ceb6283a6b036e4644e00aa2c4c7b194868c8733"}
Sep 18 19:38:34.705177 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:34.705142    2581 generic.go:356] "Generic (PLEG): container finished" podID="cff0cd8b-6053-42d8-9b1e-20930e49616b" containerID="322b450b64f12d4ea69908094129176e8dcf6c7c59301f4e15600f33a971bc77" exitCode=0
Sep 18 19:38:34.705771 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:34.705208    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzsk4" event={"ID":"cff0cd8b-6053-42d8-9b1e-20930e49616b","Type":"ContainerDied","Data":"322b450b64f12d4ea69908094129176e8dcf6c7c59301f4e15600f33a971bc77"}
Sep 18 19:38:35.515697 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:35.515672    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82551de1-f8ab-432e-8ff6-1da685c07fa9-metrics-tls\") pod \"dns-default-dwtpk\" (UID: \"82551de1-f8ab-432e-8ff6-1da685c07fa9\") " pod="openshift-dns/dns-default-dwtpk"
Sep 18 19:38:35.515791 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:35.515727    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-tls\") pod \"image-registry-86f46f546-98c44\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") " pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:38:35.515852 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:35.515816    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:38:35.515893 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:35.515877    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:38:35.515893 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:35.515888    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86f46f546-98c44: secret "image-registry-tls" not found
Sep 18 19:38:35.515960 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:35.515901    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82551de1-f8ab-432e-8ff6-1da685c07fa9-metrics-tls podName:82551de1-f8ab-432e-8ff6-1da685c07fa9 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:39.515881877 +0000 UTC m=+42.618170183 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/82551de1-f8ab-432e-8ff6-1da685c07fa9-metrics-tls") pod "dns-default-dwtpk" (UID: "82551de1-f8ab-432e-8ff6-1da685c07fa9") : secret "dns-default-metrics-tls" not found
Sep 18 19:38:35.515960 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:35.515922    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-tls podName:80dda80c-b4fe-4ab8-9f8b-ef73f7b04756 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:39.515911576 +0000 UTC m=+42.618199886 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-tls") pod "image-registry-86f46f546-98c44" (UID: "80dda80c-b4fe-4ab8-9f8b-ef73f7b04756") : secret "image-registry-tls" not found
Sep 18 19:38:35.711761 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:35.711544    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzsk4" event={"ID":"cff0cd8b-6053-42d8-9b1e-20930e49616b","Type":"ContainerStarted","Data":"95100175d908b1c8e0a1b182eaf183c4860c11ad713ebd0e77e289859b719c73"}
Sep 18 19:38:35.732526 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:35.732478    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mzsk4" podStartSLOduration=7.625547715 podStartE2EDuration="37.732456975s" totalImagesPullingTime="30.10690926s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:37:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:35.731286101 +0000 UTC m=+38.833574423" watchObservedRunningTime="2026-09-18 19:38:35.732456975 +0000 UTC m=+38.834745306"
Sep 18 19:38:36.715634 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:36.715593    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fn944" event={"ID":"e7a69286-c751-476a-a2fc-631e26030284","Type":"ContainerStarted","Data":"9990091c6d1ae5f6375efd9c115d57d2dedd92898324629f4a54ffd17a65887a"}
Sep 18 19:38:36.733555 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:36.733515    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fn944" podStartSLOduration=3.263546722 podStartE2EDuration="5.733500565s" totalImagesPullingTime="2.469953843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:36.733204299 +0000 UTC m=+39.835492627" watchObservedRunningTime="2026-09-18 19:38:36.733500565 +0000 UTC m=+39.835788892"
Sep 18 19:38:39.543374 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:39.543344    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82551de1-f8ab-432e-8ff6-1da685c07fa9-metrics-tls\") pod \"dns-default-dwtpk\" (UID: \"82551de1-f8ab-432e-8ff6-1da685c07fa9\") " pod="openshift-dns/dns-default-dwtpk"
Sep 18 19:38:39.543721 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:39.543389    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-tls\") pod \"image-registry-86f46f546-98c44\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") " pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:38:39.543721 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:39.543487    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:38:39.543721 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:39.543487    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:38:39.543721 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:39.543498    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86f46f546-98c44: secret "image-registry-tls" not found
Sep 18 19:38:39.543721 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:39.543547    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82551de1-f8ab-432e-8ff6-1da685c07fa9-metrics-tls podName:82551de1-f8ab-432e-8ff6-1da685c07fa9 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:47.543531278 +0000 UTC m=+50.645819584 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/82551de1-f8ab-432e-8ff6-1da685c07fa9-metrics-tls") pod "dns-default-dwtpk" (UID: "82551de1-f8ab-432e-8ff6-1da685c07fa9") : secret "dns-default-metrics-tls" not found
Sep 18 19:38:39.543721 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:39.543560    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-tls podName:80dda80c-b4fe-4ab8-9f8b-ef73f7b04756 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:47.543554486 +0000 UTC m=+50.645842792 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-tls") pod "image-registry-86f46f546-98c44" (UID: "80dda80c-b4fe-4ab8-9f8b-ef73f7b04756") : secret "image-registry-tls" not found
Sep 18 19:38:47.597855 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:47.597800    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82551de1-f8ab-432e-8ff6-1da685c07fa9-metrics-tls\") pod \"dns-default-dwtpk\" (UID: \"82551de1-f8ab-432e-8ff6-1da685c07fa9\") " pod="openshift-dns/dns-default-dwtpk"
Sep 18 19:38:47.598384 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:47.597867    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-tls\") pod \"image-registry-86f46f546-98c44\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") " pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:38:47.598384 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:47.597965    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:38:47.598384 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:47.597985    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:38:47.598384 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:47.597996    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86f46f546-98c44: secret "image-registry-tls" not found
Sep 18 19:38:47.598384 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:47.598042    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82551de1-f8ab-432e-8ff6-1da685c07fa9-metrics-tls podName:82551de1-f8ab-432e-8ff6-1da685c07fa9 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:03.598022229 +0000 UTC m=+66.700310536 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/82551de1-f8ab-432e-8ff6-1da685c07fa9-metrics-tls") pod "dns-default-dwtpk" (UID: "82551de1-f8ab-432e-8ff6-1da685c07fa9") : secret "dns-default-metrics-tls" not found
Sep 18 19:38:47.598384 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:47.598061    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-tls podName:80dda80c-b4fe-4ab8-9f8b-ef73f7b04756 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:03.598051995 +0000 UTC m=+66.700340302 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-tls") pod "image-registry-86f46f546-98c44" (UID: "80dda80c-b4fe-4ab8-9f8b-ef73f7b04756") : secret "image-registry-tls" not found
Sep 18 19:38:56.612055 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.612018    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9h56d"]
Sep 18 19:38:56.613778 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.613759    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9h56d"
Sep 18 19:38:56.616249 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.616228    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:38:56.617003 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.616985    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-n975g\""
Sep 18 19:38:56.617062 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.617015    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 18 19:38:56.624328 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.624308    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9h56d"]
Sep 18 19:38:56.631306 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.631287    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rm6st"]
Sep 18 19:38:56.635518 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.635499    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-ffj79"]
Sep 18 19:38:56.635645 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.635630    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rm6st"
Sep 18 19:38:56.637052 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.637033    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-ffj79"
Sep 18 19:38:56.639712 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.639562    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 18 19:38:56.639712 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.639578    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 19:38:56.639712 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.639588    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 19:38:56.639712 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.639611    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 19:38:56.639712 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.639634    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 18 19:38:56.639712 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.639660    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-w9bv9\""
Sep 18 19:38:56.639712 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.639672    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 19:38:56.639712 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.639582    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 18 19:38:56.640098 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.639759    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 18 19:38:56.640532 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.640516    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6brmb\""
Sep 18 19:38:56.644024 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.644002    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 18 19:38:56.647735 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.647717    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-ffj79"]
Sep 18 19:38:56.648650 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.648631    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rm6st"]
Sep 18 19:38:56.661524 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.661498    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e764d634-1d03-429d-8454-d725e6be456f-serving-cert\") pod \"insights-operator-6f9c644dd9-ffj79\" (UID: \"e764d634-1d03-429d-8454-d725e6be456f\") " pod="openshift-insights/insights-operator-6f9c644dd9-ffj79"
Sep 18 19:38:56.661608 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.661591    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9dhq\" (UniqueName: \"kubernetes.io/projected/e764d634-1d03-429d-8454-d725e6be456f-kube-api-access-v9dhq\") pod \"insights-operator-6f9c644dd9-ffj79\" (UID: \"e764d634-1d03-429d-8454-d725e6be456f\") " pod="openshift-insights/insights-operator-6f9c644dd9-ffj79"
Sep 18 19:38:56.661654 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.661613    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e764d634-1d03-429d-8454-d725e6be456f-tmp\") pod \"insights-operator-6f9c644dd9-ffj79\" (UID: \"e764d634-1d03-429d-8454-d725e6be456f\") " pod="openshift-insights/insights-operator-6f9c644dd9-ffj79"
Sep 18 19:38:56.661654 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.661629    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e764d634-1d03-429d-8454-d725e6be456f-snapshots\") pod \"insights-operator-6f9c644dd9-ffj79\" (UID: \"e764d634-1d03-429d-8454-d725e6be456f\") " pod="openshift-insights/insights-operator-6f9c644dd9-ffj79"
Sep 18 19:38:56.661654 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.661643    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e764d634-1d03-429d-8454-d725e6be456f-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-ffj79\" (UID: \"e764d634-1d03-429d-8454-d725e6be456f\") " pod="openshift-insights/insights-operator-6f9c644dd9-ffj79"
Sep 18 19:38:56.661756 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.661696    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h64qb\" (UniqueName: \"kubernetes.io/projected/bf93b809-edab-49b2-8217-9e5ab4d42789-kube-api-access-h64qb\") pod \"volume-data-source-validator-5c4db8bdf9-9h56d\" (UID: \"bf93b809-edab-49b2-8217-9e5ab4d42789\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9h56d"
Sep 18 19:38:56.661756 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.661729    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/efe551fc-5e21-4174-bfaf-95e53c300ae2-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-rm6st\" (UID: \"efe551fc-5e21-4174-bfaf-95e53c300ae2\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rm6st"
Sep 18 19:38:56.661821 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.661789    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e764d634-1d03-429d-8454-d725e6be456f-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-ffj79\" (UID: \"e764d634-1d03-429d-8454-d725e6be456f\") " pod="openshift-insights/insights-operator-6f9c644dd9-ffj79"
Sep 18 19:38:56.661821 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.661814    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/efe551fc-5e21-4174-bfaf-95e53c300ae2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-rm6st\" (UID: \"efe551fc-5e21-4174-bfaf-95e53c300ae2\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rm6st"
Sep 18 19:38:56.661912 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.661865    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4k7b\" (UniqueName: \"kubernetes.io/projected/efe551fc-5e21-4174-bfaf-95e53c300ae2-kube-api-access-v4k7b\") pod \"cluster-monitoring-operator-847b8bd8d7-rm6st\" (UID: \"efe551fc-5e21-4174-bfaf-95e53c300ae2\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rm6st"
Sep 18 19:38:56.715400 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.715375    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-bcgcv"]
Sep 18 19:38:56.718232 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.718218    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-bcgcv"
Sep 18 19:38:56.720175 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.720155    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-8l864"]
Sep 18 19:38:56.722994 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.722978    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-8l864"
Sep 18 19:38:56.728096 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.728079    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-bcgcv"]
Sep 18 19:38:56.732274 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.732253    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 18 19:38:56.732392 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.732260    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 18 19:38:56.732609 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.732592    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-qbpkd\""
Sep 18 19:38:56.733861 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.733826    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-92bsq\""
Sep 18 19:38:56.733861 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.733856    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:38:56.734005 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.733940    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 18 19:38:56.740269 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.740254    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-8l864"]
Sep 18 19:38:56.762289 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.762270    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h64qb\" (UniqueName: \"kubernetes.io/projected/bf93b809-edab-49b2-8217-9e5ab4d42789-kube-api-access-h64qb\") pod \"volume-data-source-validator-5c4db8bdf9-9h56d\" (UID: \"bf93b809-edab-49b2-8217-9e5ab4d42789\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9h56d"
Sep 18 19:38:56.762372 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.762302    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/efe551fc-5e21-4174-bfaf-95e53c300ae2-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-rm6st\" (UID: \"efe551fc-5e21-4174-bfaf-95e53c300ae2\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rm6st"
Sep 18 19:38:56.762372 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.762349    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e764d634-1d03-429d-8454-d725e6be456f-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-ffj79\" (UID: \"e764d634-1d03-429d-8454-d725e6be456f\") " pod="openshift-insights/insights-operator-6f9c644dd9-ffj79"
Sep 18 19:38:56.762476 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.762373    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/efe551fc-5e21-4174-bfaf-95e53c300ae2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-rm6st\" (UID: \"efe551fc-5e21-4174-bfaf-95e53c300ae2\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rm6st"
Sep 18 19:38:56.762476 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.762397    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4k7b\" (UniqueName: \"kubernetes.io/projected/efe551fc-5e21-4174-bfaf-95e53c300ae2-kube-api-access-v4k7b\") pod \"cluster-monitoring-operator-847b8bd8d7-rm6st\" (UID: \"efe551fc-5e21-4174-bfaf-95e53c300ae2\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rm6st"
Sep 18 19:38:56.762476 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.762444    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsgz5\" (UniqueName: \"kubernetes.io/projected/43511802-2525-4048-abbb-76ccf9c0af0f-kube-api-access-gsgz5\") pod \"network-check-source-7cbc45fddc-bcgcv\" (UID: \"43511802-2525-4048-abbb-76ccf9c0af0f\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-bcgcv"
Sep 18 19:38:56.762593 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.762480    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e764d634-1d03-429d-8454-d725e6be456f-serving-cert\") pod \"insights-operator-6f9c644dd9-ffj79\" (UID: \"e764d634-1d03-429d-8454-d725e6be456f\") " pod="openshift-insights/insights-operator-6f9c644dd9-ffj79"
Sep 18 19:38:56.762593 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:56.762506    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:38:56.762693 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:56.762604    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efe551fc-5e21-4174-bfaf-95e53c300ae2-cluster-monitoring-operator-tls podName:efe551fc-5e21-4174-bfaf-95e53c300ae2 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:57.262583503 +0000 UTC m=+60.364871826 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/efe551fc-5e21-4174-bfaf-95e53c300ae2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-rm6st" (UID: "efe551fc-5e21-4174-bfaf-95e53c300ae2") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:38:56.762693 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.762667    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9dhq\" (UniqueName: \"kubernetes.io/projected/e764d634-1d03-429d-8454-d725e6be456f-kube-api-access-v9dhq\") pod \"insights-operator-6f9c644dd9-ffj79\" (UID: \"e764d634-1d03-429d-8454-d725e6be456f\") " pod="openshift-insights/insights-operator-6f9c644dd9-ffj79"
Sep 18 19:38:56.762799 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.762697    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e764d634-1d03-429d-8454-d725e6be456f-tmp\") pod \"insights-operator-6f9c644dd9-ffj79\" (UID: \"e764d634-1d03-429d-8454-d725e6be456f\") " pod="openshift-insights/insights-operator-6f9c644dd9-ffj79"
Sep 18 19:38:56.762799 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.762722    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e764d634-1d03-429d-8454-d725e6be456f-snapshots\") pod \"insights-operator-6f9c644dd9-ffj79\" (UID: \"e764d634-1d03-429d-8454-d725e6be456f\") " pod="openshift-insights/insights-operator-6f9c644dd9-ffj79"
Sep 18 19:38:56.762799 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.762747    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e764d634-1d03-429d-8454-d725e6be456f-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-ffj79\" (UID: \"e764d634-1d03-429d-8454-d725e6be456f\") " pod="openshift-insights/insights-operator-6f9c644dd9-ffj79"
Sep 18 19:38:56.763127 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.763111    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e764d634-1d03-429d-8454-d725e6be456f-tmp\") pod \"insights-operator-6f9c644dd9-ffj79\" (UID: \"e764d634-1d03-429d-8454-d725e6be456f\") " pod="openshift-insights/insights-operator-6f9c644dd9-ffj79"
Sep 18 19:38:56.763187 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.763126    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e764d634-1d03-429d-8454-d725e6be456f-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-ffj79\" (UID: \"e764d634-1d03-429d-8454-d725e6be456f\") " pod="openshift-insights/insights-operator-6f9c644dd9-ffj79"
Sep 18 19:38:56.763187 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.763162    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/efe551fc-5e21-4174-bfaf-95e53c300ae2-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-rm6st\" (UID: \"efe551fc-5e21-4174-bfaf-95e53c300ae2\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rm6st"
Sep 18 19:38:56.763359 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.763343    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e764d634-1d03-429d-8454-d725e6be456f-snapshots\") pod \"insights-operator-6f9c644dd9-ffj79\" (UID: \"e764d634-1d03-429d-8454-d725e6be456f\") " pod="openshift-insights/insights-operator-6f9c644dd9-ffj79"
Sep 18 19:38:56.763667 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.763650    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e764d634-1d03-429d-8454-d725e6be456f-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-ffj79\" (UID: \"e764d634-1d03-429d-8454-d725e6be456f\") " pod="openshift-insights/insights-operator-6f9c644dd9-ffj79"
Sep 18 19:38:56.766185 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.766166    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e764d634-1d03-429d-8454-d725e6be456f-serving-cert\") pod \"insights-operator-6f9c644dd9-ffj79\" (UID: \"e764d634-1d03-429d-8454-d725e6be456f\") " pod="openshift-insights/insights-operator-6f9c644dd9-ffj79"
Sep 18 19:38:56.778110 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.778090    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9dhq\" (UniqueName: \"kubernetes.io/projected/e764d634-1d03-429d-8454-d725e6be456f-kube-api-access-v9dhq\") pod \"insights-operator-6f9c644dd9-ffj79\" (UID: \"e764d634-1d03-429d-8454-d725e6be456f\") " pod="openshift-insights/insights-operator-6f9c644dd9-ffj79"
Sep 18 19:38:56.778185 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.778122    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4k7b\" (UniqueName: \"kubernetes.io/projected/efe551fc-5e21-4174-bfaf-95e53c300ae2-kube-api-access-v4k7b\") pod \"cluster-monitoring-operator-847b8bd8d7-rm6st\" (UID: \"efe551fc-5e21-4174-bfaf-95e53c300ae2\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rm6st"
Sep 18 19:38:56.778714 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.778691    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h64qb\" (UniqueName: \"kubernetes.io/projected/bf93b809-edab-49b2-8217-9e5ab4d42789-kube-api-access-h64qb\") pod \"volume-data-source-validator-5c4db8bdf9-9h56d\" (UID: \"bf93b809-edab-49b2-8217-9e5ab4d42789\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9h56d"
Sep 18 19:38:56.863896 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.863818    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8vs4\" (UniqueName: \"kubernetes.io/projected/85c9bace-8688-4f4a-ad42-dc32911c6263-kube-api-access-s8vs4\") pod \"kube-storage-version-migrator-operator-55889f7b47-8l864\" (UID: \"85c9bace-8688-4f4a-ad42-dc32911c6263\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-8l864"
Sep 18 19:38:56.863896 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.863870    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/85c9bace-8688-4f4a-ad42-dc32911c6263-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-8l864\" (UID: \"85c9bace-8688-4f4a-ad42-dc32911c6263\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-8l864"
Sep 18 19:38:56.864124 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.863914    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/85c9bace-8688-4f4a-ad42-dc32911c6263-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-8l864\" (UID: \"85c9bace-8688-4f4a-ad42-dc32911c6263\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-8l864"
Sep 18 19:38:56.864124 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.863989    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsgz5\" (UniqueName: \"kubernetes.io/projected/43511802-2525-4048-abbb-76ccf9c0af0f-kube-api-access-gsgz5\") pod \"network-check-source-7cbc45fddc-bcgcv\" (UID: \"43511802-2525-4048-abbb-76ccf9c0af0f\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-bcgcv"
Sep 18 19:38:56.873753 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.873730    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsgz5\" (UniqueName: \"kubernetes.io/projected/43511802-2525-4048-abbb-76ccf9c0af0f-kube-api-access-gsgz5\") pod \"network-check-source-7cbc45fddc-bcgcv\" (UID: \"43511802-2525-4048-abbb-76ccf9c0af0f\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-bcgcv"
Sep 18 19:38:56.922575 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.922546    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9h56d"
Sep 18 19:38:56.951152 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.951127    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-ffj79"
Sep 18 19:38:56.964795 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.964767    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8vs4\" (UniqueName: \"kubernetes.io/projected/85c9bace-8688-4f4a-ad42-dc32911c6263-kube-api-access-s8vs4\") pod \"kube-storage-version-migrator-operator-55889f7b47-8l864\" (UID: \"85c9bace-8688-4f4a-ad42-dc32911c6263\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-8l864"
Sep 18 19:38:56.964907 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.964814    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/85c9bace-8688-4f4a-ad42-dc32911c6263-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-8l864\" (UID: \"85c9bace-8688-4f4a-ad42-dc32911c6263\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-8l864"
Sep 18 19:38:56.964907 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.964876    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/85c9bace-8688-4f4a-ad42-dc32911c6263-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-8l864\" (UID: \"85c9bace-8688-4f4a-ad42-dc32911c6263\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-8l864"
Sep 18 19:38:56.965350 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.965331    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/85c9bace-8688-4f4a-ad42-dc32911c6263-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-8l864\" (UID: \"85c9bace-8688-4f4a-ad42-dc32911c6263\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-8l864"
Sep 18 19:38:56.967563 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.967522    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/85c9bace-8688-4f4a-ad42-dc32911c6263-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-8l864\" (UID: \"85c9bace-8688-4f4a-ad42-dc32911c6263\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-8l864"
Sep 18 19:38:56.976029 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:56.976004    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8vs4\" (UniqueName: \"kubernetes.io/projected/85c9bace-8688-4f4a-ad42-dc32911c6263-kube-api-access-s8vs4\") pod \"kube-storage-version-migrator-operator-55889f7b47-8l864\" (UID: \"85c9bace-8688-4f4a-ad42-dc32911c6263\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-8l864"
Sep 18 19:38:57.028241 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:57.028168    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-bcgcv"
Sep 18 19:38:57.033357 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:57.033332    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-8l864"
Sep 18 19:38:57.188907 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:57.188873    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9h56d"]
Sep 18 19:38:57.191350 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:38:57.191318    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf93b809_edab_49b2_8217_9e5ab4d42789.slice/crio-dc0cd9bf7d19b30beba7ca4e88f139ad8237e730b3803a60476630cc4ba370eb WatchSource:0}: Error finding container dc0cd9bf7d19b30beba7ca4e88f139ad8237e730b3803a60476630cc4ba370eb: Status 404 returned error can't find the container with id dc0cd9bf7d19b30beba7ca4e88f139ad8237e730b3803a60476630cc4ba370eb
Sep 18 19:38:57.226180 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:57.226152    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-ffj79"]
Sep 18 19:38:57.227270 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:38:57.227250    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode764d634_1d03_429d_8454_d725e6be456f.slice/crio-15d8cf4e0d67c6d522a2f6efc920f0cbe3e5952a8439e8bf8a0c966c59b103d2 WatchSource:0}: Error finding container 15d8cf4e0d67c6d522a2f6efc920f0cbe3e5952a8439e8bf8a0c966c59b103d2: Status 404 returned error can't find the container with id 15d8cf4e0d67c6d522a2f6efc920f0cbe3e5952a8439e8bf8a0c966c59b103d2
Sep 18 19:38:57.267403 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:57.267381    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/efe551fc-5e21-4174-bfaf-95e53c300ae2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-rm6st\" (UID: \"efe551fc-5e21-4174-bfaf-95e53c300ae2\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rm6st"
Sep 18 19:38:57.267541 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:57.267528    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:38:57.267627 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:57.267586    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efe551fc-5e21-4174-bfaf-95e53c300ae2-cluster-monitoring-operator-tls podName:efe551fc-5e21-4174-bfaf-95e53c300ae2 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:58.26756639 +0000 UTC m=+61.369854699 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/efe551fc-5e21-4174-bfaf-95e53c300ae2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-rm6st" (UID: "efe551fc-5e21-4174-bfaf-95e53c300ae2") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:38:57.323228 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:57.323200    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-bcgcv"]
Sep 18 19:38:57.325777 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:38:57.325757    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43511802_2525_4048_abbb_76ccf9c0af0f.slice/crio-6d447b20a564210e52e932e1ca7ad5e0d533f812dbef49ee5ba63c502b4cab71 WatchSource:0}: Error finding container 6d447b20a564210e52e932e1ca7ad5e0d533f812dbef49ee5ba63c502b4cab71: Status 404 returned error can't find the container with id 6d447b20a564210e52e932e1ca7ad5e0d533f812dbef49ee5ba63c502b4cab71
Sep 18 19:38:57.348645 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:57.348624    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-8l864"]
Sep 18 19:38:57.350981 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:38:57.350960    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85c9bace_8688_4f4a_ad42_dc32911c6263.slice/crio-7a8ad5c6fe3000d1d054303c8f9b4dec0029a31531f37a16d5e6c02d73c98437 WatchSource:0}: Error finding container 7a8ad5c6fe3000d1d054303c8f9b4dec0029a31531f37a16d5e6c02d73c98437: Status 404 returned error can't find the container with id 7a8ad5c6fe3000d1d054303c8f9b4dec0029a31531f37a16d5e6c02d73c98437
Sep 18 19:38:57.691424 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:57.691391    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2wj49"
Sep 18 19:38:57.762767 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:57.762729    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-bcgcv" event={"ID":"43511802-2525-4048-abbb-76ccf9c0af0f","Type":"ContainerStarted","Data":"6d447b20a564210e52e932e1ca7ad5e0d533f812dbef49ee5ba63c502b4cab71"}
Sep 18 19:38:57.763949 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:57.763924    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-ffj79" event={"ID":"e764d634-1d03-429d-8454-d725e6be456f","Type":"ContainerStarted","Data":"15d8cf4e0d67c6d522a2f6efc920f0cbe3e5952a8439e8bf8a0c966c59b103d2"}
Sep 18 19:38:57.765125 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:57.765106    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9h56d" event={"ID":"bf93b809-edab-49b2-8217-9e5ab4d42789","Type":"ContainerStarted","Data":"dc0cd9bf7d19b30beba7ca4e88f139ad8237e730b3803a60476630cc4ba370eb"}
Sep 18 19:38:57.766143 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:57.766123    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-8l864" event={"ID":"85c9bace-8688-4f4a-ad42-dc32911c6263","Type":"ContainerStarted","Data":"7a8ad5c6fe3000d1d054303c8f9b4dec0029a31531f37a16d5e6c02d73c98437"}
Sep 18 19:38:58.277129 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:58.276446    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/efe551fc-5e21-4174-bfaf-95e53c300ae2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-rm6st\" (UID: \"efe551fc-5e21-4174-bfaf-95e53c300ae2\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rm6st"
Sep 18 19:38:58.277129 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:58.276683    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:38:58.277129 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:38:58.276747    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efe551fc-5e21-4174-bfaf-95e53c300ae2-cluster-monitoring-operator-tls podName:efe551fc-5e21-4174-bfaf-95e53c300ae2 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:00.276727434 +0000 UTC m=+63.379015746 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/efe551fc-5e21-4174-bfaf-95e53c300ae2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-rm6st" (UID: "efe551fc-5e21-4174-bfaf-95e53c300ae2") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:38:58.772238 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:58.772200    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9h56d" event={"ID":"bf93b809-edab-49b2-8217-9e5ab4d42789","Type":"ContainerStarted","Data":"1da4425f53badbf257f82144df61c8c3b3f943c90680d031b154d789c52a74d7"}
Sep 18 19:38:58.790719 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:38:58.790145    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9h56d" podStartSLOduration=1.3547763640000001 podStartE2EDuration="2.790123618s" totalImagesPullingTime="1.435347254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:58.788408844 +0000 UTC m=+61.890697173" watchObservedRunningTime="2026-09-18 19:38:58.790123618 +0000 UTC m=+61.892411942"
Sep 18 19:39:00.294448 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:00.294390    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/efe551fc-5e21-4174-bfaf-95e53c300ae2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-rm6st\" (UID: \"efe551fc-5e21-4174-bfaf-95e53c300ae2\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rm6st"
Sep 18 19:39:00.295063 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:00.294560    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:39:00.295063 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:00.294651    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efe551fc-5e21-4174-bfaf-95e53c300ae2-cluster-monitoring-operator-tls podName:efe551fc-5e21-4174-bfaf-95e53c300ae2 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:04.294634303 +0000 UTC m=+67.396922611 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/efe551fc-5e21-4174-bfaf-95e53c300ae2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-rm6st" (UID: "efe551fc-5e21-4174-bfaf-95e53c300ae2") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:39:01.782496 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:01.782432    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-8l864" event={"ID":"85c9bace-8688-4f4a-ad42-dc32911c6263","Type":"ContainerStarted","Data":"ad31619dc7eda3f7fcfabdf96de59d3f41c14582cd6893ef27fa1083fec3ce51"}
Sep 18 19:39:01.784428 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:01.784402    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-bcgcv" event={"ID":"43511802-2525-4048-abbb-76ccf9c0af0f","Type":"ContainerStarted","Data":"c4871f41fad5fb83ab0f2d5fe1b29c8f90a9eed6d1d6e207289b9293e6642e50"}
Sep 18 19:39:01.786029 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:01.786009    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-ffj79" event={"ID":"e764d634-1d03-429d-8454-d725e6be456f","Type":"ContainerStarted","Data":"f54a8990146ae524131607b70247bdbffe3c3e479d125e68894a1f022b3deeba"}
Sep 18 19:39:01.797739 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:01.797679    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-8l864" podStartSLOduration=2.320404863 podStartE2EDuration="5.797662212s" totalImagesPullingTime="3.477257349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:01.796628382 +0000 UTC m=+64.898916710" watchObservedRunningTime="2026-09-18 19:39:01.797662212 +0000 UTC m=+64.899950535"
Sep 18 19:39:01.814867 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:01.814803    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-ffj79" podStartSLOduration=2.2170758680000002 podStartE2EDuration="5.814787521s" totalImagesPullingTime="3.597711653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:01.814372522 +0000 UTC m=+64.916660851" watchObservedRunningTime="2026-09-18 19:39:01.814787521 +0000 UTC m=+64.917075850"
Sep 18 19:39:01.829313 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:01.829272    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-bcgcv" podStartSLOduration=2.319607225 podStartE2EDuration="5.829255523s" totalImagesPullingTime="3.509648298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:01.828444452 +0000 UTC m=+64.930732777" watchObservedRunningTime="2026-09-18 19:39:01.829255523 +0000 UTC m=+64.931543851"
Sep 18 19:39:02.041637 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:02.041569    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qv2hc"]
Sep 18 19:39:02.046209 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:02.046183    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qv2hc"
Sep 18 19:39:02.048495 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:02.048463    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 19:39:02.048595 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:02.048464    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 19:39:02.048803 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:02.048786    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-696cc\""
Sep 18 19:39:02.055130 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:02.055108    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qv2hc"]
Sep 18 19:39:02.109329 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:02.109299    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9cd29832-b16f-48c7-a830-fd10ff432df8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qv2hc\" (UID: \"9cd29832-b16f-48c7-a830-fd10ff432df8\") " pod="openshift-insights/insights-runtime-extractor-qv2hc"
Sep 18 19:39:02.109469 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:02.109342    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9cd29832-b16f-48c7-a830-fd10ff432df8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qv2hc\" (UID: \"9cd29832-b16f-48c7-a830-fd10ff432df8\") " pod="openshift-insights/insights-runtime-extractor-qv2hc"
Sep 18 19:39:02.109469 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:02.109429    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9cd29832-b16f-48c7-a830-fd10ff432df8-data-volume\") pod \"insights-runtime-extractor-qv2hc\" (UID: \"9cd29832-b16f-48c7-a830-fd10ff432df8\") " pod="openshift-insights/insights-runtime-extractor-qv2hc"
Sep 18 19:39:02.109579 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:02.109524    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9cd29832-b16f-48c7-a830-fd10ff432df8-crio-socket\") pod \"insights-runtime-extractor-qv2hc\" (UID: \"9cd29832-b16f-48c7-a830-fd10ff432df8\") " pod="openshift-insights/insights-runtime-extractor-qv2hc"
Sep 18 19:39:02.109579 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:02.109558    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfq5k\" (UniqueName: \"kubernetes.io/projected/9cd29832-b16f-48c7-a830-fd10ff432df8-kube-api-access-gfq5k\") pod \"insights-runtime-extractor-qv2hc\" (UID: \"9cd29832-b16f-48c7-a830-fd10ff432df8\") " pod="openshift-insights/insights-runtime-extractor-qv2hc"
Sep 18 19:39:02.210878 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:02.210815    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9cd29832-b16f-48c7-a830-fd10ff432df8-crio-socket\") pod \"insights-runtime-extractor-qv2hc\" (UID: \"9cd29832-b16f-48c7-a830-fd10ff432df8\") " pod="openshift-insights/insights-runtime-extractor-qv2hc"
Sep 18 19:39:02.211051 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:02.210900    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfq5k\" (UniqueName: \"kubernetes.io/projected/9cd29832-b16f-48c7-a830-fd10ff432df8-kube-api-access-gfq5k\") pod \"insights-runtime-extractor-qv2hc\" (UID: \"9cd29832-b16f-48c7-a830-fd10ff432df8\") " pod="openshift-insights/insights-runtime-extractor-qv2hc"
Sep 18 19:39:02.211051 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:02.210952    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9cd29832-b16f-48c7-a830-fd10ff432df8-crio-socket\") pod \"insights-runtime-extractor-qv2hc\" (UID: \"9cd29832-b16f-48c7-a830-fd10ff432df8\") " pod="openshift-insights/insights-runtime-extractor-qv2hc"
Sep 18 19:39:02.211051 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:02.210990    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9cd29832-b16f-48c7-a830-fd10ff432df8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qv2hc\" (UID: \"9cd29832-b16f-48c7-a830-fd10ff432df8\") " pod="openshift-insights/insights-runtime-extractor-qv2hc"
Sep 18 19:39:02.211051 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:02.211021    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9cd29832-b16f-48c7-a830-fd10ff432df8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qv2hc\" (UID: \"9cd29832-b16f-48c7-a830-fd10ff432df8\") " pod="openshift-insights/insights-runtime-extractor-qv2hc"
Sep 18 19:39:02.211252 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:02.211058    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9cd29832-b16f-48c7-a830-fd10ff432df8-data-volume\") pod \"insights-runtime-extractor-qv2hc\" (UID: \"9cd29832-b16f-48c7-a830-fd10ff432df8\") " pod="openshift-insights/insights-runtime-extractor-qv2hc"
Sep 18 19:39:02.211252 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:02.211168    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:39:02.211252 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:02.211180    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:02.211252 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:02.211253    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cd29832-b16f-48c7-a830-fd10ff432df8-insights-runtime-extractor-tls podName:9cd29832-b16f-48c7-a830-fd10ff432df8 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:02.711235936 +0000 UTC m=+65.813524257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9cd29832-b16f-48c7-a830-fd10ff432df8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qv2hc" (UID: "9cd29832-b16f-48c7-a830-fd10ff432df8") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:02.211448 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:02.211278    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9cd29832-b16f-48c7-a830-fd10ff432df8-kube-rbac-proxy-cm podName:9cd29832-b16f-48c7-a830-fd10ff432df8 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:02.711268832 +0000 UTC m=+65.813557138 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9cd29832-b16f-48c7-a830-fd10ff432df8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qv2hc" (UID: "9cd29832-b16f-48c7-a830-fd10ff432df8") : configmap "kube-rbac-proxy" not found
Sep 18 19:39:02.211448 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:02.211432    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9cd29832-b16f-48c7-a830-fd10ff432df8-data-volume\") pod \"insights-runtime-extractor-qv2hc\" (UID: \"9cd29832-b16f-48c7-a830-fd10ff432df8\") " pod="openshift-insights/insights-runtime-extractor-qv2hc"
Sep 18 19:39:02.223706 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:02.223682    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfq5k\" (UniqueName: \"kubernetes.io/projected/9cd29832-b16f-48c7-a830-fd10ff432df8-kube-api-access-gfq5k\") pod \"insights-runtime-extractor-qv2hc\" (UID: \"9cd29832-b16f-48c7-a830-fd10ff432df8\") " pod="openshift-insights/insights-runtime-extractor-qv2hc"
Sep 18 19:39:02.719483 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:02.719448    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9cd29832-b16f-48c7-a830-fd10ff432df8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qv2hc\" (UID: \"9cd29832-b16f-48c7-a830-fd10ff432df8\") " pod="openshift-insights/insights-runtime-extractor-qv2hc"
Sep 18 19:39:02.719666 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:02.719495    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9cd29832-b16f-48c7-a830-fd10ff432df8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qv2hc\" (UID: \"9cd29832-b16f-48c7-a830-fd10ff432df8\") " pod="openshift-insights/insights-runtime-extractor-qv2hc"
Sep 18 19:39:02.719666 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:02.719618    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:02.719666 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:02.719636    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:39:02.719796 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:02.719720    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cd29832-b16f-48c7-a830-fd10ff432df8-insights-runtime-extractor-tls podName:9cd29832-b16f-48c7-a830-fd10ff432df8 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:03.71970184 +0000 UTC m=+66.821990161 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9cd29832-b16f-48c7-a830-fd10ff432df8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qv2hc" (UID: "9cd29832-b16f-48c7-a830-fd10ff432df8") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:02.719796 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:02.719742    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9cd29832-b16f-48c7-a830-fd10ff432df8-kube-rbac-proxy-cm podName:9cd29832-b16f-48c7-a830-fd10ff432df8 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:03.71973595 +0000 UTC m=+66.822024255 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9cd29832-b16f-48c7-a830-fd10ff432df8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qv2hc" (UID: "9cd29832-b16f-48c7-a830-fd10ff432df8") : configmap "kube-rbac-proxy" not found
Sep 18 19:39:03.122867 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:03.122820    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-842cm\" (UniqueName: \"kubernetes.io/projected/0e5d82f6-4e09-448b-b9b7-8644ea3803a3-kube-api-access-842cm\") pod \"network-check-target-67bgb\" (UID: \"0e5d82f6-4e09-448b-b9b7-8644ea3803a3\") " pod="openshift-network-diagnostics/network-check-target-67bgb"
Sep 18 19:39:03.123250 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:03.122890    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1173b48c-041e-4ec6-9708-98391b2bf0fe-metrics-certs\") pod \"network-metrics-daemon-s9j6n\" (UID: \"1173b48c-041e-4ec6-9708-98391b2bf0fe\") " pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:39:03.125193 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:03.125174    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:39:03.125521 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:03.125505    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-842cm\" (UniqueName: \"kubernetes.io/projected/0e5d82f6-4e09-448b-b9b7-8644ea3803a3-kube-api-access-842cm\") pod \"network-check-target-67bgb\" (UID: \"0e5d82f6-4e09-448b-b9b7-8644ea3803a3\") " pod="openshift-network-diagnostics/network-check-target-67bgb"
Sep 18 19:39:03.133903 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:03.133879    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 19:39:03.134017 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:03.133945    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1173b48c-041e-4ec6-9708-98391b2bf0fe-metrics-certs podName:1173b48c-041e-4ec6-9708-98391b2bf0fe nodeName:}" failed. No retries permitted until 2026-09-18 19:40:07.13392982 +0000 UTC m=+130.236218129 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1173b48c-041e-4ec6-9708-98391b2bf0fe-metrics-certs") pod "network-metrics-daemon-s9j6n" (UID: "1173b48c-041e-4ec6-9708-98391b2bf0fe") : secret "metrics-daemon-secret" not found
Sep 18 19:39:03.137103 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:03.137087    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pwxm5\""
Sep 18 19:39:03.145590 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:03.145577    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67bgb"
Sep 18 19:39:03.224142 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:03.224113    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d3541d3-fb52-42a5-b977-9cc44ac396bc-original-pull-secret\") pod \"global-pull-secret-syncer-2p2bp\" (UID: \"5d3541d3-fb52-42a5-b977-9cc44ac396bc\") " pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:39:03.227152 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:03.227122    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:39:03.237293 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:03.237277    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d3541d3-fb52-42a5-b977-9cc44ac396bc-original-pull-secret\") pod \"global-pull-secret-syncer-2p2bp\" (UID: \"5d3541d3-fb52-42a5-b977-9cc44ac396bc\") " pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:39:03.338120 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:03.338087    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2p2bp"
Sep 18 19:39:03.412127 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:03.412055    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-67bgb"]
Sep 18 19:39:03.414264 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:39:03.414241    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e5d82f6_4e09_448b_b9b7_8644ea3803a3.slice/crio-79f0d9a783a5758d31629ad04c76677fb46a88d1aceda9db5064839d31ac1366 WatchSource:0}: Error finding container 79f0d9a783a5758d31629ad04c76677fb46a88d1aceda9db5064839d31ac1366: Status 404 returned error can't find the container with id 79f0d9a783a5758d31629ad04c76677fb46a88d1aceda9db5064839d31ac1366
Sep 18 19:39:03.603782 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:03.603762    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2p2bp"]
Sep 18 19:39:03.606189 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:39:03.606161    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d3541d3_fb52_42a5_b977_9cc44ac396bc.slice/crio-03ccfb34b3887e057ad8be7668c289dcc13599008a760c56d81a0e3f15e057f2 WatchSource:0}: Error finding container 03ccfb34b3887e057ad8be7668c289dcc13599008a760c56d81a0e3f15e057f2: Status 404 returned error can't find the container with id 03ccfb34b3887e057ad8be7668c289dcc13599008a760c56d81a0e3f15e057f2
Sep 18 19:39:03.627807 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:03.627788    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82551de1-f8ab-432e-8ff6-1da685c07fa9-metrics-tls\") pod \"dns-default-dwtpk\" (UID: \"82551de1-f8ab-432e-8ff6-1da685c07fa9\") " pod="openshift-dns/dns-default-dwtpk"
Sep 18 19:39:03.627914 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:03.627830    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-tls\") pod \"image-registry-86f46f546-98c44\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") " pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:39:03.627960 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:03.627929    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 19:39:03.627960 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:03.627932    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:39:03.627960 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:03.627939    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86f46f546-98c44: secret "image-registry-tls" not found
Sep 18 19:39:03.628055 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:03.627983    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82551de1-f8ab-432e-8ff6-1da685c07fa9-metrics-tls podName:82551de1-f8ab-432e-8ff6-1da685c07fa9 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:35.627970082 +0000 UTC m=+98.730258387 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/82551de1-f8ab-432e-8ff6-1da685c07fa9-metrics-tls") pod "dns-default-dwtpk" (UID: "82551de1-f8ab-432e-8ff6-1da685c07fa9") : secret "dns-default-metrics-tls" not found
Sep 18 19:39:03.628055 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:03.628000    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-tls podName:80dda80c-b4fe-4ab8-9f8b-ef73f7b04756 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:35.627991186 +0000 UTC m=+98.730279491 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-tls") pod "image-registry-86f46f546-98c44" (UID: "80dda80c-b4fe-4ab8-9f8b-ef73f7b04756") : secret "image-registry-tls" not found
Sep 18 19:39:03.728650 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:03.728599    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9cd29832-b16f-48c7-a830-fd10ff432df8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qv2hc\" (UID: \"9cd29832-b16f-48c7-a830-fd10ff432df8\") " pod="openshift-insights/insights-runtime-extractor-qv2hc"
Sep 18 19:39:03.728650 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:03.728627    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9cd29832-b16f-48c7-a830-fd10ff432df8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qv2hc\" (UID: \"9cd29832-b16f-48c7-a830-fd10ff432df8\") " pod="openshift-insights/insights-runtime-extractor-qv2hc"
Sep 18 19:39:03.728803 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:03.728750    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:39:03.728803 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:03.728755    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:03.728803 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:03.728794    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9cd29832-b16f-48c7-a830-fd10ff432df8-kube-rbac-proxy-cm podName:9cd29832-b16f-48c7-a830-fd10ff432df8 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:05.728781693 +0000 UTC m=+68.831069999 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9cd29832-b16f-48c7-a830-fd10ff432df8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qv2hc" (UID: "9cd29832-b16f-48c7-a830-fd10ff432df8") : configmap "kube-rbac-proxy" not found
Sep 18 19:39:03.728962 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:03.728816    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cd29832-b16f-48c7-a830-fd10ff432df8-insights-runtime-extractor-tls podName:9cd29832-b16f-48c7-a830-fd10ff432df8 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:05.728799283 +0000 UTC m=+68.831087790 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9cd29832-b16f-48c7-a830-fd10ff432df8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qv2hc" (UID: "9cd29832-b16f-48c7-a830-fd10ff432df8") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:03.793439 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:03.793408    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2p2bp" event={"ID":"5d3541d3-fb52-42a5-b977-9cc44ac396bc","Type":"ContainerStarted","Data":"03ccfb34b3887e057ad8be7668c289dcc13599008a760c56d81a0e3f15e057f2"}
Sep 18 19:39:03.794793 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:03.794764    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-67bgb" event={"ID":"0e5d82f6-4e09-448b-b9b7-8644ea3803a3","Type":"ContainerStarted","Data":"c3b994808baaca4ddd3849a7b543d16d6d4db025e8266ca6d787674270bcdcf9"}
Sep 18 19:39:03.794880 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:03.794805    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-67bgb" event={"ID":"0e5d82f6-4e09-448b-b9b7-8644ea3803a3","Type":"ContainerStarted","Data":"79f0d9a783a5758d31629ad04c76677fb46a88d1aceda9db5064839d31ac1366"}
Sep 18 19:39:03.794880 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:03.794855    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-67bgb"
Sep 18 19:39:03.809886 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:03.809852    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-67bgb" podStartSLOduration=65.809827056 podStartE2EDuration="1m5.809827056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:03.809762706 +0000 UTC m=+66.912051032" watchObservedRunningTime="2026-09-18 19:39:03.809827056 +0000 UTC m=+66.912115383"
Sep 18 19:39:04.225898 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:04.225871    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w9dlm_58fa049b-20bb-4466-bb38-8b46448db93b/dns-node-resolver/0.log"
Sep 18 19:39:04.331869 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:04.331817    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/efe551fc-5e21-4174-bfaf-95e53c300ae2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-rm6st\" (UID: \"efe551fc-5e21-4174-bfaf-95e53c300ae2\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rm6st"
Sep 18 19:39:04.332036 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:04.331979    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:39:04.332102 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:04.332063    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efe551fc-5e21-4174-bfaf-95e53c300ae2-cluster-monitoring-operator-tls podName:efe551fc-5e21-4174-bfaf-95e53c300ae2 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:12.332041609 +0000 UTC m=+75.434329920 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/efe551fc-5e21-4174-bfaf-95e53c300ae2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-rm6st" (UID: "efe551fc-5e21-4174-bfaf-95e53c300ae2") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:39:04.822675 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:04.822649    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7nrfp_68f35ba9-52cf-4a53-bc81-b2863eeb0930/node-ca/0.log"
Sep 18 19:39:05.741966 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:05.741927    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9cd29832-b16f-48c7-a830-fd10ff432df8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qv2hc\" (UID: \"9cd29832-b16f-48c7-a830-fd10ff432df8\") " pod="openshift-insights/insights-runtime-extractor-qv2hc"
Sep 18 19:39:05.742342 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:05.741982    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9cd29832-b16f-48c7-a830-fd10ff432df8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qv2hc\" (UID: \"9cd29832-b16f-48c7-a830-fd10ff432df8\") " pod="openshift-insights/insights-runtime-extractor-qv2hc"
Sep 18 19:39:05.742342 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:05.742092    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:05.742342 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:05.742097    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:39:05.742342 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:05.742162    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9cd29832-b16f-48c7-a830-fd10ff432df8-kube-rbac-proxy-cm podName:9cd29832-b16f-48c7-a830-fd10ff432df8 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:09.742142352 +0000 UTC m=+72.844430664 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9cd29832-b16f-48c7-a830-fd10ff432df8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qv2hc" (UID: "9cd29832-b16f-48c7-a830-fd10ff432df8") : configmap "kube-rbac-proxy" not found
Sep 18 19:39:05.742342 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:05.742178    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cd29832-b16f-48c7-a830-fd10ff432df8-insights-runtime-extractor-tls podName:9cd29832-b16f-48c7-a830-fd10ff432df8 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:09.742169445 +0000 UTC m=+72.844457758 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9cd29832-b16f-48c7-a830-fd10ff432df8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qv2hc" (UID: "9cd29832-b16f-48c7-a830-fd10ff432df8") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:05.822866 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:05.822824    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fn944_e7a69286-c751-476a-a2fc-631e26030284/serve-healthcheck-canary/0.log"
Sep 18 19:39:08.813354 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:08.813317    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2p2bp" event={"ID":"5d3541d3-fb52-42a5-b977-9cc44ac396bc","Type":"ContainerStarted","Data":"1740c33263f8e4d18a1d483ac584719eed10bb6b0198c57d39c3982ec4a88be5"}
Sep 18 19:39:08.827741 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:08.827692    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2p2bp" podStartSLOduration=66.669149675 podStartE2EDuration="1m10.827678227s" totalImagesPullingTime="4.158528552s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:08.827336049 +0000 UTC m=+71.929624378" watchObservedRunningTime="2026-09-18 19:39:08.827678227 +0000 UTC m=+71.929966599"
Sep 18 19:39:09.773726 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:09.773683    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9cd29832-b16f-48c7-a830-fd10ff432df8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qv2hc\" (UID: \"9cd29832-b16f-48c7-a830-fd10ff432df8\") " pod="openshift-insights/insights-runtime-extractor-qv2hc"
Sep 18 19:39:09.773726 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:09.773732    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9cd29832-b16f-48c7-a830-fd10ff432df8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qv2hc\" (UID: \"9cd29832-b16f-48c7-a830-fd10ff432df8\") " pod="openshift-insights/insights-runtime-extractor-qv2hc"
Sep 18 19:39:09.773974 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:09.773829    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:09.773974 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:09.773879    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:39:09.773974 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:09.773907    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cd29832-b16f-48c7-a830-fd10ff432df8-insights-runtime-extractor-tls podName:9cd29832-b16f-48c7-a830-fd10ff432df8 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:17.773892844 +0000 UTC m=+80.876181151 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9cd29832-b16f-48c7-a830-fd10ff432df8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qv2hc" (UID: "9cd29832-b16f-48c7-a830-fd10ff432df8") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:09.773974 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:09.773928    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9cd29832-b16f-48c7-a830-fd10ff432df8-kube-rbac-proxy-cm podName:9cd29832-b16f-48c7-a830-fd10ff432df8 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:17.773915772 +0000 UTC m=+80.876204082 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9cd29832-b16f-48c7-a830-fd10ff432df8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qv2hc" (UID: "9cd29832-b16f-48c7-a830-fd10ff432df8") : configmap "kube-rbac-proxy" not found
Sep 18 19:39:12.394397 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:12.394344    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/efe551fc-5e21-4174-bfaf-95e53c300ae2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-rm6st\" (UID: \"efe551fc-5e21-4174-bfaf-95e53c300ae2\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rm6st"
Sep 18 19:39:12.394977 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:12.394487    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 19:39:12.394977 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:12.394555    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efe551fc-5e21-4174-bfaf-95e53c300ae2-cluster-monitoring-operator-tls podName:efe551fc-5e21-4174-bfaf-95e53c300ae2 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:28.3945384 +0000 UTC m=+91.496826705 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/efe551fc-5e21-4174-bfaf-95e53c300ae2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-rm6st" (UID: "efe551fc-5e21-4174-bfaf-95e53c300ae2") : secret "cluster-monitoring-operator-tls" not found
Sep 18 19:39:17.834126 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:17.834082    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9cd29832-b16f-48c7-a830-fd10ff432df8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qv2hc\" (UID: \"9cd29832-b16f-48c7-a830-fd10ff432df8\") " pod="openshift-insights/insights-runtime-extractor-qv2hc"
Sep 18 19:39:17.834126 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:17.834129    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9cd29832-b16f-48c7-a830-fd10ff432df8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qv2hc\" (UID: \"9cd29832-b16f-48c7-a830-fd10ff432df8\") " pod="openshift-insights/insights-runtime-extractor-qv2hc"
Sep 18 19:39:17.834543 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:17.834220    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:17.834543 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:17.834283    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cd29832-b16f-48c7-a830-fd10ff432df8-insights-runtime-extractor-tls podName:9cd29832-b16f-48c7-a830-fd10ff432df8 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:33.834265162 +0000 UTC m=+96.936553488 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9cd29832-b16f-48c7-a830-fd10ff432df8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qv2hc" (UID: "9cd29832-b16f-48c7-a830-fd10ff432df8") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:39:17.834618 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:17.834566    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9cd29832-b16f-48c7-a830-fd10ff432df8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qv2hc\" (UID: \"9cd29832-b16f-48c7-a830-fd10ff432df8\") " pod="openshift-insights/insights-runtime-extractor-qv2hc"
Sep 18 19:39:26.208903 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.208768    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d47b8565-k8tf2"]
Sep 18 19:39:26.211388 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.211365    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d47b8565-k8tf2"
Sep 18 19:39:26.211520 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.211497    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-f7sc6"]
Sep 18 19:39:26.213035 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.213016    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-f7sc6"
Sep 18 19:39:26.214381 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.214359    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 18 19:39:26.215429 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.215404    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 18 19:39:26.215429 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.215413    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 18 19:39:26.215579 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.215418    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 18 19:39:26.215755 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.215738    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-p6hqb\""
Sep 18 19:39:26.215865 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.215826    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 18 19:39:26.216038 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.216024    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 18 19:39:26.227101 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.227082    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d47b8565-k8tf2"]
Sep 18 19:39:26.227873 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.227857    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-f7sc6"]
Sep 18 19:39:26.228567 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.228552    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd865f6cd-vm6gc"]
Sep 18 19:39:26.230030 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.230012    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd865f6cd-vm6gc"
Sep 18 19:39:26.232935 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.232916    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-xjs86\""
Sep 18 19:39:26.233030 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.232990    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 18 19:39:26.244341 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.244324    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd865f6cd-vm6gc"]
Sep 18 19:39:26.295674 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.295637    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bd6280eb-423c-4d05-8137-f4578bb8d0b0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5cd865f6cd-vm6gc\" (UID: \"bd6280eb-423c-4d05-8137-f4578bb8d0b0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd865f6cd-vm6gc"
Sep 18 19:39:26.295825 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.295696    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxrv8\" (UniqueName: \"kubernetes.io/projected/bd6280eb-423c-4d05-8137-f4578bb8d0b0-kube-api-access-cxrv8\") pod \"managed-serviceaccount-addon-agent-5cd865f6cd-vm6gc\" (UID: \"bd6280eb-423c-4d05-8137-f4578bb8d0b0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd865f6cd-vm6gc"
Sep 18 19:39:26.295825 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.295783    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c9f1c72-512c-415c-85fc-fc7fad90e25d-tmp\") pod \"klusterlet-addon-workmgr-6d47b8565-k8tf2\" (UID: \"4c9f1c72-512c-415c-85fc-fc7fad90e25d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d47b8565-k8tf2"
Sep 18 19:39:26.295825 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.295806    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9dtl\" (UniqueName: \"kubernetes.io/projected/4c9f1c72-512c-415c-85fc-fc7fad90e25d-kube-api-access-z9dtl\") pod \"klusterlet-addon-workmgr-6d47b8565-k8tf2\" (UID: \"4c9f1c72-512c-415c-85fc-fc7fad90e25d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d47b8565-k8tf2"
Sep 18 19:39:26.295957 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.295830    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b1138025-f527-48a3-baa5-499c4bfbe124-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-f7sc6\" (UID: \"b1138025-f527-48a3-baa5-499c4bfbe124\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-f7sc6"
Sep 18 19:39:26.295957 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.295869    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4c9f1c72-512c-415c-85fc-fc7fad90e25d-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d47b8565-k8tf2\" (UID: \"4c9f1c72-512c-415c-85fc-fc7fad90e25d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d47b8565-k8tf2"
Sep 18 19:39:26.295957 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.295920    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b1138025-f527-48a3-baa5-499c4bfbe124-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-f7sc6\" (UID: \"b1138025-f527-48a3-baa5-499c4bfbe124\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-f7sc6"
Sep 18 19:39:26.396178 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.396150    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c9f1c72-512c-415c-85fc-fc7fad90e25d-tmp\") pod \"klusterlet-addon-workmgr-6d47b8565-k8tf2\" (UID: \"4c9f1c72-512c-415c-85fc-fc7fad90e25d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d47b8565-k8tf2"
Sep 18 19:39:26.396178 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.396176    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9dtl\" (UniqueName: \"kubernetes.io/projected/4c9f1c72-512c-415c-85fc-fc7fad90e25d-kube-api-access-z9dtl\") pod \"klusterlet-addon-workmgr-6d47b8565-k8tf2\" (UID: \"4c9f1c72-512c-415c-85fc-fc7fad90e25d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d47b8565-k8tf2"
Sep 18 19:39:26.396365 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.396195    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b1138025-f527-48a3-baa5-499c4bfbe124-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-f7sc6\" (UID: \"b1138025-f527-48a3-baa5-499c4bfbe124\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-f7sc6"
Sep 18 19:39:26.396365 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.396211    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4c9f1c72-512c-415c-85fc-fc7fad90e25d-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d47b8565-k8tf2\" (UID: \"4c9f1c72-512c-415c-85fc-fc7fad90e25d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d47b8565-k8tf2"
Sep 18 19:39:26.396365 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.396353    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b1138025-f527-48a3-baa5-499c4bfbe124-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-f7sc6\" (UID: \"b1138025-f527-48a3-baa5-499c4bfbe124\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-f7sc6"
Sep 18 19:39:26.396509 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.396408    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bd6280eb-423c-4d05-8137-f4578bb8d0b0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5cd865f6cd-vm6gc\" (UID: \"bd6280eb-423c-4d05-8137-f4578bb8d0b0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd865f6cd-vm6gc"
Sep 18 19:39:26.396509 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.396474    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxrv8\" (UniqueName: \"kubernetes.io/projected/bd6280eb-423c-4d05-8137-f4578bb8d0b0-kube-api-access-cxrv8\") pod \"managed-serviceaccount-addon-agent-5cd865f6cd-vm6gc\" (UID: \"bd6280eb-423c-4d05-8137-f4578bb8d0b0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd865f6cd-vm6gc"
Sep 18 19:39:26.396646 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.396571    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c9f1c72-512c-415c-85fc-fc7fad90e25d-tmp\") pod \"klusterlet-addon-workmgr-6d47b8565-k8tf2\" (UID: \"4c9f1c72-512c-415c-85fc-fc7fad90e25d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d47b8565-k8tf2"
Sep 18 19:39:26.396998 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.396977    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b1138025-f527-48a3-baa5-499c4bfbe124-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-f7sc6\" (UID: \"b1138025-f527-48a3-baa5-499c4bfbe124\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-f7sc6"
Sep 18 19:39:26.398946 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.398926    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b1138025-f527-48a3-baa5-499c4bfbe124-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-f7sc6\" (UID: \"b1138025-f527-48a3-baa5-499c4bfbe124\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-f7sc6"
Sep 18 19:39:26.399084 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.399066    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bd6280eb-423c-4d05-8137-f4578bb8d0b0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5cd865f6cd-vm6gc\" (UID: \"bd6280eb-423c-4d05-8137-f4578bb8d0b0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd865f6cd-vm6gc"
Sep 18 19:39:26.399168 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.399151    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4c9f1c72-512c-415c-85fc-fc7fad90e25d-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d47b8565-k8tf2\" (UID: \"4c9f1c72-512c-415c-85fc-fc7fad90e25d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d47b8565-k8tf2"
Sep 18 19:39:26.407917 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.407890    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxrv8\" (UniqueName: \"kubernetes.io/projected/bd6280eb-423c-4d05-8137-f4578bb8d0b0-kube-api-access-cxrv8\") pod \"managed-serviceaccount-addon-agent-5cd865f6cd-vm6gc\" (UID: \"bd6280eb-423c-4d05-8137-f4578bb8d0b0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd865f6cd-vm6gc"
Sep 18 19:39:26.408162 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.408136    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9dtl\" (UniqueName: \"kubernetes.io/projected/4c9f1c72-512c-415c-85fc-fc7fad90e25d-kube-api-access-z9dtl\") pod \"klusterlet-addon-workmgr-6d47b8565-k8tf2\" (UID: \"4c9f1c72-512c-415c-85fc-fc7fad90e25d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d47b8565-k8tf2"
Sep 18 19:39:26.522056 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.522033    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d47b8565-k8tf2"
Sep 18 19:39:26.527139 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.527125    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-f7sc6"
Sep 18 19:39:26.550433 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.550407    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd865f6cd-vm6gc"
Sep 18 19:39:26.794685 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.794614    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d47b8565-k8tf2"]
Sep 18 19:39:26.797528 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:39:26.797498    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c9f1c72_512c_415c_85fc_fc7fad90e25d.slice/crio-4a3974658e2a577956d4a8c9598f9c7e88966a199a9df8732e4f989091183f9e WatchSource:0}: Error finding container 4a3974658e2a577956d4a8c9598f9c7e88966a199a9df8732e4f989091183f9e: Status 404 returned error can't find the container with id 4a3974658e2a577956d4a8c9598f9c7e88966a199a9df8732e4f989091183f9e
Sep 18 19:39:26.798109 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.798002    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-f7sc6"]
Sep 18 19:39:26.800311 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:39:26.800291    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1138025_f527_48a3_baa5_499c4bfbe124.slice/crio-fe99a8e8cf6d48611068c58a20ee4282dec5ab81c7cbc4ae086f5e808b7f3812 WatchSource:0}: Error finding container fe99a8e8cf6d48611068c58a20ee4282dec5ab81c7cbc4ae086f5e808b7f3812: Status 404 returned error can't find the container with id fe99a8e8cf6d48611068c58a20ee4282dec5ab81c7cbc4ae086f5e808b7f3812
Sep 18 19:39:26.821698 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.821680    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd865f6cd-vm6gc"]
Sep 18 19:39:26.823766 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:39:26.823742    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd6280eb_423c_4d05_8137_f4578bb8d0b0.slice/crio-95e976d3f5913e9579d32f8f27ca8fee50d8c756428fc3191efe2e3142168b77 WatchSource:0}: Error finding container 95e976d3f5913e9579d32f8f27ca8fee50d8c756428fc3191efe2e3142168b77: Status 404 returned error can't find the container with id 95e976d3f5913e9579d32f8f27ca8fee50d8c756428fc3191efe2e3142168b77
Sep 18 19:39:26.866754 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.866727    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-f7sc6" event={"ID":"b1138025-f527-48a3-baa5-499c4bfbe124","Type":"ContainerStarted","Data":"fe99a8e8cf6d48611068c58a20ee4282dec5ab81c7cbc4ae086f5e808b7f3812"}
Sep 18 19:39:26.867851 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.867811    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d47b8565-k8tf2" event={"ID":"4c9f1c72-512c-415c-85fc-fc7fad90e25d","Type":"ContainerStarted","Data":"4a3974658e2a577956d4a8c9598f9c7e88966a199a9df8732e4f989091183f9e"}
Sep 18 19:39:26.868820 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:26.868804    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd865f6cd-vm6gc" event={"ID":"bd6280eb-423c-4d05-8137-f4578bb8d0b0","Type":"ContainerStarted","Data":"95e976d3f5913e9579d32f8f27ca8fee50d8c756428fc3191efe2e3142168b77"}
Sep 18 19:39:28.413395 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:28.413348    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/efe551fc-5e21-4174-bfaf-95e53c300ae2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-rm6st\" (UID: \"efe551fc-5e21-4174-bfaf-95e53c300ae2\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rm6st"
Sep 18 19:39:28.418001 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:28.417802    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/efe551fc-5e21-4174-bfaf-95e53c300ae2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-rm6st\" (UID: \"efe551fc-5e21-4174-bfaf-95e53c300ae2\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rm6st"
Sep 18 19:39:28.447370 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:28.447155    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-w9bv9\""
Sep 18 19:39:28.455251 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:28.455067    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rm6st"
Sep 18 19:39:31.510440 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:31.510332    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rm6st"]
Sep 18 19:39:31.512386 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:39:31.512361    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefe551fc_5e21_4174_bfaf_95e53c300ae2.slice/crio-cf0c89f1c1016171306ff8298d7c6b0d0c5245124d737d341979bae17df7d67d WatchSource:0}: Error finding container cf0c89f1c1016171306ff8298d7c6b0d0c5245124d737d341979bae17df7d67d: Status 404 returned error can't find the container with id cf0c89f1c1016171306ff8298d7c6b0d0c5245124d737d341979bae17df7d67d
Sep 18 19:39:31.889510 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:31.889473    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd865f6cd-vm6gc" event={"ID":"bd6280eb-423c-4d05-8137-f4578bb8d0b0","Type":"ContainerStarted","Data":"5f3b7a697c5123bffaed267a1372972129a8129b1805e2b44f61638332e2f516"}
Sep 18 19:39:31.891437 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:31.891404    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-f7sc6" event={"ID":"b1138025-f527-48a3-baa5-499c4bfbe124","Type":"ContainerStarted","Data":"81eb6ea40a8cafc218a7c936515eae7329e92abb4fff1ec292e7debc766d29f1"}
Sep 18 19:39:31.893422 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:31.893393    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d47b8565-k8tf2" event={"ID":"4c9f1c72-512c-415c-85fc-fc7fad90e25d","Type":"ContainerStarted","Data":"6496bae9567439f900cb990a8699267bd43f140c601eceff25b9f4f03b16e903"}
Sep 18 19:39:31.893767 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:31.893725    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d47b8565-k8tf2"
Sep 18 19:39:31.894676 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:31.894653    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rm6st" event={"ID":"efe551fc-5e21-4174-bfaf-95e53c300ae2","Type":"ContainerStarted","Data":"cf0c89f1c1016171306ff8298d7c6b0d0c5245124d737d341979bae17df7d67d"}
Sep 18 19:39:31.895609 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:31.895588    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d47b8565-k8tf2"
Sep 18 19:39:31.903935 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:31.903891    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cd865f6cd-vm6gc" podStartSLOduration=1.460911187 podStartE2EDuration="5.903875301s" totalImagesPullingTime="4.442964114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:31.903530422 +0000 UTC m=+95.005818751" watchObservedRunningTime="2026-09-18 19:39:31.903875301 +0000 UTC m=+95.006163633"
Sep 18 19:39:31.920767 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:31.920726    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d47b8565-k8tf2" podStartSLOduration=1.427845483 podStartE2EDuration="5.920711293s" totalImagesPullingTime="4.49286581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:31.920058399 +0000 UTC m=+95.022346752" watchObservedRunningTime="2026-09-18 19:39:31.920711293 +0000 UTC m=+95.022999625"
Sep 18 19:39:31.934994 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:31.933952    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-f7sc6" podStartSLOduration=1.4892954440000001 podStartE2EDuration="5.933936058s" totalImagesPullingTime="4.444640614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:31.933589098 +0000 UTC m=+95.035877428" watchObservedRunningTime="2026-09-18 19:39:31.933936058 +0000 UTC m=+95.036224387"
Sep 18 19:39:33.860380 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:33.860343    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9cd29832-b16f-48c7-a830-fd10ff432df8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qv2hc\" (UID: \"9cd29832-b16f-48c7-a830-fd10ff432df8\") " pod="openshift-insights/insights-runtime-extractor-qv2hc"
Sep 18 19:39:33.862659 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:33.862640    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9cd29832-b16f-48c7-a830-fd10ff432df8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qv2hc\" (UID: \"9cd29832-b16f-48c7-a830-fd10ff432df8\") " pod="openshift-insights/insights-runtime-extractor-qv2hc"
Sep 18 19:39:33.902766 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:33.902734    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rm6st" event={"ID":"efe551fc-5e21-4174-bfaf-95e53c300ae2","Type":"ContainerStarted","Data":"e2862145517d8a9500fbdbe5ec5ddf850381271e5667f6fc27dc7a596593afa8"}
Sep 18 19:39:33.919025 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:33.918976    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-rm6st" podStartSLOduration=36.51348207 podStartE2EDuration="37.91896304s" totalImagesPullingTime="1.40548097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:33.91809854 +0000 UTC m=+97.020386867" watchObservedRunningTime="2026-09-18 19:39:33.91896304 +0000 UTC m=+97.021251368"
Sep 18 19:39:34.156562 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:34.156498    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qv2hc"
Sep 18 19:39:34.458424 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:34.458347    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qv2hc"]
Sep 18 19:39:34.461855 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:39:34.461810    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd29832_b16f_48c7_a830_fd10ff432df8.slice/crio-39f577667eea815432e28e0d6480994df01ad919c1fc3b7fd2323f442dbaa3c6 WatchSource:0}: Error finding container 39f577667eea815432e28e0d6480994df01ad919c1fc3b7fd2323f442dbaa3c6: Status 404 returned error can't find the container with id 39f577667eea815432e28e0d6480994df01ad919c1fc3b7fd2323f442dbaa3c6
Sep 18 19:39:34.802366 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:34.802341    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-67bgb"
Sep 18 19:39:34.907437 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:34.907408    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qv2hc" event={"ID":"9cd29832-b16f-48c7-a830-fd10ff432df8","Type":"ContainerStarted","Data":"fac9c2a6bf150cc98532bb8fb5b2478d90f4a6342ec14b355f289f8bacd0524b"}
Sep 18 19:39:34.907784 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:34.907457    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qv2hc" event={"ID":"9cd29832-b16f-48c7-a830-fd10ff432df8","Type":"ContainerStarted","Data":"39f577667eea815432e28e0d6480994df01ad919c1fc3b7fd2323f442dbaa3c6"}
Sep 18 19:39:35.674598 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:35.674571    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82551de1-f8ab-432e-8ff6-1da685c07fa9-metrics-tls\") pod \"dns-default-dwtpk\" (UID: \"82551de1-f8ab-432e-8ff6-1da685c07fa9\") " pod="openshift-dns/dns-default-dwtpk"
Sep 18 19:39:35.674759 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:35.674606    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-tls\") pod \"image-registry-86f46f546-98c44\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") " pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:39:35.676959 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:35.676930    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/82551de1-f8ab-432e-8ff6-1da685c07fa9-metrics-tls\") pod \"dns-default-dwtpk\" (UID: \"82551de1-f8ab-432e-8ff6-1da685c07fa9\") " pod="openshift-dns/dns-default-dwtpk"
Sep 18 19:39:35.676959 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:35.676940    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-tls\") pod \"image-registry-86f46f546-98c44\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") " pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:39:35.911821 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:35.911795    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qv2hc" event={"ID":"9cd29832-b16f-48c7-a830-fd10ff432df8","Type":"ContainerStarted","Data":"27cd76d662ba70d3cd41f1aa842dfa7f77524c7aba9ad9ee84e04d4cb3739388"}
Sep 18 19:39:35.928269 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:35.928214    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ntrxj\""
Sep 18 19:39:35.936228 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:35.936212    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:39:35.939589 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:35.939573    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5f8sq\""
Sep 18 19:39:35.947694 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:35.947668    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dwtpk"
Sep 18 19:39:36.227249 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:36.227176    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86f46f546-98c44"]
Sep 18 19:39:36.231645 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:39:36.231609    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80dda80c_b4fe_4ab8_9f8b_ef73f7b04756.slice/crio-23d044069180f86e1f111e2dd0ba81d60fbcd322ff340b3f44a01a19df8e2746 WatchSource:0}: Error finding container 23d044069180f86e1f111e2dd0ba81d60fbcd322ff340b3f44a01a19df8e2746: Status 404 returned error can't find the container with id 23d044069180f86e1f111e2dd0ba81d60fbcd322ff340b3f44a01a19df8e2746
Sep 18 19:39:36.251514 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:36.251491    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dwtpk"]
Sep 18 19:39:36.259852 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:39:36.259806    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82551de1_f8ab_432e_8ff6_1da685c07fa9.slice/crio-981e33013fe7b62a9ba2e109e52524e94de64682028f702116fa2c7cdd754c1b WatchSource:0}: Error finding container 981e33013fe7b62a9ba2e109e52524e94de64682028f702116fa2c7cdd754c1b: Status 404 returned error can't find the container with id 981e33013fe7b62a9ba2e109e52524e94de64682028f702116fa2c7cdd754c1b
Sep 18 19:39:36.920599 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:36.920562    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86f46f546-98c44" event={"ID":"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756","Type":"ContainerStarted","Data":"63e4ee6e4a2a1b83565ab59c48437aac6652535a5676c0aeab13cd654fb8f284"}
Sep 18 19:39:36.921185 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:36.920645    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86f46f546-98c44" event={"ID":"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756","Type":"ContainerStarted","Data":"23d044069180f86e1f111e2dd0ba81d60fbcd322ff340b3f44a01a19df8e2746"}
Sep 18 19:39:36.921185 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:36.920672    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:39:36.922070 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:36.922034    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dwtpk" event={"ID":"82551de1-f8ab-432e-8ff6-1da685c07fa9","Type":"ContainerStarted","Data":"981e33013fe7b62a9ba2e109e52524e94de64682028f702116fa2c7cdd754c1b"}
Sep 18 19:39:36.940114 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:36.940061    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-86f46f546-98c44" podStartSLOduration=98.940046506 podStartE2EDuration="1m38.940046506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:36.938255304 +0000 UTC m=+100.040543645" watchObservedRunningTime="2026-09-18 19:39:36.940046506 +0000 UTC m=+100.042334833"
Sep 18 19:39:37.928061 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:37.928019    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qv2hc" event={"ID":"9cd29832-b16f-48c7-a830-fd10ff432df8","Type":"ContainerStarted","Data":"484fb49c15a4ecc2195073883e4db673cc65175e5f3e01f492e45f492dfe9d0f"}
Sep 18 19:39:37.946881 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:37.946815    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qv2hc" podStartSLOduration=33.588421636 podStartE2EDuration="35.946794167s" totalImagesPullingTime="2.358372531s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:39:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:37.945045956 +0000 UTC m=+101.047334285" watchObservedRunningTime="2026-09-18 19:39:37.946794167 +0000 UTC m=+101.049082520"
Sep 18 19:39:38.934118 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:38.934079    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dwtpk" event={"ID":"82551de1-f8ab-432e-8ff6-1da685c07fa9","Type":"ContainerStarted","Data":"510f926975b3d6c6177e4fa6f0b1d21d43f502ecc4e3148368aafed876d4a2c8"}
Sep 18 19:39:38.934555 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:38.934142    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dwtpk" event={"ID":"82551de1-f8ab-432e-8ff6-1da685c07fa9","Type":"ContainerStarted","Data":"795e102bd5917dbf4867cdb526902b36c0711331eb146284a0cd1648d0a58e66"}
Sep 18 19:39:38.950226 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:38.950179    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dwtpk" podStartSLOduration=66.038857615 podStartE2EDuration="1m7.950159632s" totalImagesPullingTime="1.911302017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:38.949863665 +0000 UTC m=+102.052152004" watchObservedRunningTime="2026-09-18 19:39:38.950159632 +0000 UTC m=+102.052447961"
Sep 18 19:39:39.937845 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:39.937803    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dwtpk"
Sep 18 19:39:41.830485 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:41.830451    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bdftw"]
Sep 18 19:39:41.862814 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:41.862789    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bdftw"
Sep 18 19:39:41.866444 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:41.866417    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qmpz8\""
Sep 18 19:39:41.866639 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:41.866415    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 19:39:41.866639 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:41.866487    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 19:39:41.866639 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:41.866423    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 19:39:41.866639 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:41.866417    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 19:39:42.021300 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:42.021273    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c2ad484-0f94-4f69-84a8-4fb65d4e8d53-node-exporter-wtmp\") pod \"node-exporter-bdftw\" (UID: \"9c2ad484-0f94-4f69-84a8-4fb65d4e8d53\") " pod="openshift-monitoring/node-exporter-bdftw"
Sep 18 19:39:42.021300 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:42.021300    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c2ad484-0f94-4f69-84a8-4fb65d4e8d53-metrics-client-ca\") pod \"node-exporter-bdftw\" (UID: \"9c2ad484-0f94-4f69-84a8-4fb65d4e8d53\") " pod="openshift-monitoring/node-exporter-bdftw"
Sep 18 19:39:42.021546 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:42.021318    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c2ad484-0f94-4f69-84a8-4fb65d4e8d53-node-exporter-textfile\") pod \"node-exporter-bdftw\" (UID: \"9c2ad484-0f94-4f69-84a8-4fb65d4e8d53\") " pod="openshift-monitoring/node-exporter-bdftw"
Sep 18 19:39:42.021546 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:42.021337    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c2ad484-0f94-4f69-84a8-4fb65d4e8d53-node-exporter-tls\") pod \"node-exporter-bdftw\" (UID: \"9c2ad484-0f94-4f69-84a8-4fb65d4e8d53\") " pod="openshift-monitoring/node-exporter-bdftw"
Sep 18 19:39:42.021546 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:42.021397    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6wrx\" (UniqueName: \"kubernetes.io/projected/9c2ad484-0f94-4f69-84a8-4fb65d4e8d53-kube-api-access-b6wrx\") pod \"node-exporter-bdftw\" (UID: \"9c2ad484-0f94-4f69-84a8-4fb65d4e8d53\") " pod="openshift-monitoring/node-exporter-bdftw"
Sep 18 19:39:42.021546 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:42.021529    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c2ad484-0f94-4f69-84a8-4fb65d4e8d53-node-exporter-accelerators-collector-config\") pod \"node-exporter-bdftw\" (UID: \"9c2ad484-0f94-4f69-84a8-4fb65d4e8d53\") " pod="openshift-monitoring/node-exporter-bdftw"
Sep 18 19:39:42.021738 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:42.021578    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c2ad484-0f94-4f69-84a8-4fb65d4e8d53-sys\") pod \"node-exporter-bdftw\" (UID: \"9c2ad484-0f94-4f69-84a8-4fb65d4e8d53\") " pod="openshift-monitoring/node-exporter-bdftw"
Sep 18 19:39:42.021738 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:42.021610    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c2ad484-0f94-4f69-84a8-4fb65d4e8d53-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bdftw\" (UID: \"9c2ad484-0f94-4f69-84a8-4fb65d4e8d53\") " pod="openshift-monitoring/node-exporter-bdftw"
Sep 18 19:39:42.021738 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:42.021670    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c2ad484-0f94-4f69-84a8-4fb65d4e8d53-root\") pod \"node-exporter-bdftw\" (UID: \"9c2ad484-0f94-4f69-84a8-4fb65d4e8d53\") " pod="openshift-monitoring/node-exporter-bdftw"
Sep 18 19:39:42.123013 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:42.122939    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c2ad484-0f94-4f69-84a8-4fb65d4e8d53-sys\") pod \"node-exporter-bdftw\" (UID: \"9c2ad484-0f94-4f69-84a8-4fb65d4e8d53\") " pod="openshift-monitoring/node-exporter-bdftw"
Sep 18 19:39:42.123013 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:42.122986    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c2ad484-0f94-4f69-84a8-4fb65d4e8d53-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bdftw\" (UID: \"9c2ad484-0f94-4f69-84a8-4fb65d4e8d53\") " pod="openshift-monitoring/node-exporter-bdftw"
Sep 18 19:39:42.123013 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:42.123009    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c2ad484-0f94-4f69-84a8-4fb65d4e8d53-root\") pod \"node-exporter-bdftw\" (UID: \"9c2ad484-0f94-4f69-84a8-4fb65d4e8d53\") " pod="openshift-monitoring/node-exporter-bdftw"
Sep 18 19:39:42.123260 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:42.123029    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c2ad484-0f94-4f69-84a8-4fb65d4e8d53-node-exporter-wtmp\") pod \"node-exporter-bdftw\" (UID: \"9c2ad484-0f94-4f69-84a8-4fb65d4e8d53\") " pod="openshift-monitoring/node-exporter-bdftw"
Sep 18 19:39:42.123260 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:42.123044    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c2ad484-0f94-4f69-84a8-4fb65d4e8d53-metrics-client-ca\") pod \"node-exporter-bdftw\" (UID: \"9c2ad484-0f94-4f69-84a8-4fb65d4e8d53\") " pod="openshift-monitoring/node-exporter-bdftw"
Sep 18 19:39:42.123260 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:42.123061    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c2ad484-0f94-4f69-84a8-4fb65d4e8d53-sys\") pod \"node-exporter-bdftw\" (UID: \"9c2ad484-0f94-4f69-84a8-4fb65d4e8d53\") " pod="openshift-monitoring/node-exporter-bdftw"
Sep 18 19:39:42.123260 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:42.123065    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c2ad484-0f94-4f69-84a8-4fb65d4e8d53-node-exporter-textfile\") pod \"node-exporter-bdftw\" (UID: \"9c2ad484-0f94-4f69-84a8-4fb65d4e8d53\") " pod="openshift-monitoring/node-exporter-bdftw"
Sep 18 19:39:42.123260 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:42.123081    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9c2ad484-0f94-4f69-84a8-4fb65d4e8d53-root\") pod \"node-exporter-bdftw\" (UID: \"9c2ad484-0f94-4f69-84a8-4fb65d4e8d53\") " pod="openshift-monitoring/node-exporter-bdftw"
Sep 18 19:39:42.123260 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:42.123117    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c2ad484-0f94-4f69-84a8-4fb65d4e8d53-node-exporter-tls\") pod \"node-exporter-bdftw\" (UID: \"9c2ad484-0f94-4f69-84a8-4fb65d4e8d53\") " pod="openshift-monitoring/node-exporter-bdftw"
Sep 18 19:39:42.123260 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:42.123144    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6wrx\" (UniqueName: \"kubernetes.io/projected/9c2ad484-0f94-4f69-84a8-4fb65d4e8d53-kube-api-access-b6wrx\") pod \"node-exporter-bdftw\" (UID: \"9c2ad484-0f94-4f69-84a8-4fb65d4e8d53\") " pod="openshift-monitoring/node-exporter-bdftw"
Sep 18 19:39:42.123260 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:42.123170    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9c2ad484-0f94-4f69-84a8-4fb65d4e8d53-node-exporter-wtmp\") pod \"node-exporter-bdftw\" (UID: \"9c2ad484-0f94-4f69-84a8-4fb65d4e8d53\") " pod="openshift-monitoring/node-exporter-bdftw"
Sep 18 19:39:42.123260 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:42.123241    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c2ad484-0f94-4f69-84a8-4fb65d4e8d53-node-exporter-accelerators-collector-config\") pod \"node-exporter-bdftw\" (UID: \"9c2ad484-0f94-4f69-84a8-4fb65d4e8d53\") " pod="openshift-monitoring/node-exporter-bdftw"
Sep 18 19:39:42.123676 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:42.123437    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9c2ad484-0f94-4f69-84a8-4fb65d4e8d53-node-exporter-textfile\") pod \"node-exporter-bdftw\" (UID: \"9c2ad484-0f94-4f69-84a8-4fb65d4e8d53\") " pod="openshift-monitoring/node-exporter-bdftw"
Sep 18 19:39:42.123931 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:42.123721    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c2ad484-0f94-4f69-84a8-4fb65d4e8d53-metrics-client-ca\") pod \"node-exporter-bdftw\" (UID: \"9c2ad484-0f94-4f69-84a8-4fb65d4e8d53\") " pod="openshift-monitoring/node-exporter-bdftw"
Sep 18 19:39:42.123931 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:42.123731    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9c2ad484-0f94-4f69-84a8-4fb65d4e8d53-node-exporter-accelerators-collector-config\") pod \"node-exporter-bdftw\" (UID: \"9c2ad484-0f94-4f69-84a8-4fb65d4e8d53\") " pod="openshift-monitoring/node-exporter-bdftw"
Sep 18 19:39:42.125498 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:42.125475    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c2ad484-0f94-4f69-84a8-4fb65d4e8d53-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bdftw\" (UID: \"9c2ad484-0f94-4f69-84a8-4fb65d4e8d53\") " pod="openshift-monitoring/node-exporter-bdftw"
Sep 18 19:39:42.125604 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:42.125517    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9c2ad484-0f94-4f69-84a8-4fb65d4e8d53-node-exporter-tls\") pod \"node-exporter-bdftw\" (UID: \"9c2ad484-0f94-4f69-84a8-4fb65d4e8d53\") " pod="openshift-monitoring/node-exporter-bdftw"
Sep 18 19:39:42.130527 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:42.130506    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6wrx\" (UniqueName: \"kubernetes.io/projected/9c2ad484-0f94-4f69-84a8-4fb65d4e8d53-kube-api-access-b6wrx\") pod \"node-exporter-bdftw\" (UID: \"9c2ad484-0f94-4f69-84a8-4fb65d4e8d53\") " pod="openshift-monitoring/node-exporter-bdftw"
Sep 18 19:39:42.171974 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:42.171949    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bdftw"
Sep 18 19:39:42.201655 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:39:42.201631    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c2ad484_0f94_4f69_84a8_4fb65d4e8d53.slice/crio-f70d641f45b4ff9377c25267c36a7e189fe55787ae445a388ce7b7651d59ef88 WatchSource:0}: Error finding container f70d641f45b4ff9377c25267c36a7e189fe55787ae445a388ce7b7651d59ef88: Status 404 returned error can't find the container with id f70d641f45b4ff9377c25267c36a7e189fe55787ae445a388ce7b7651d59ef88
Sep 18 19:39:42.949710 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:42.949673    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bdftw" event={"ID":"9c2ad484-0f94-4f69-84a8-4fb65d4e8d53","Type":"ContainerStarted","Data":"f70d641f45b4ff9377c25267c36a7e189fe55787ae445a388ce7b7651d59ef88"}
Sep 18 19:39:43.918703 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:43.918672    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-78d686bcb7-gmvm2"]
Sep 18 19:39:43.921041 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:43.921025    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2"
Sep 18 19:39:43.923920 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:43.923884    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 18 19:39:43.924054 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:43.923989    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 18 19:39:43.924117 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:43.924074    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 18 19:39:43.924117 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:43.924085    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 18 19:39:43.924218 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:43.924190    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-jp6mv\""
Sep 18 19:39:43.924995 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:43.924975    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-drqld6gjo8p58\""
Sep 18 19:39:43.925107 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:43.925033    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 18 19:39:43.938660 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:43.938637    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-78d686bcb7-gmvm2"]
Sep 18 19:39:43.959985 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:43.959947    2581 generic.go:356] "Generic (PLEG): container finished" podID="9c2ad484-0f94-4f69-84a8-4fb65d4e8d53" containerID="fbf4891889107ab82ed0a1956badce62f99bab7ee1bc9e989ae5f15fd6360ff7" exitCode=0
Sep 18 19:39:43.960261 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:43.960025    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bdftw" event={"ID":"9c2ad484-0f94-4f69-84a8-4fb65d4e8d53","Type":"ContainerDied","Data":"fbf4891889107ab82ed0a1956badce62f99bab7ee1bc9e989ae5f15fd6360ff7"}
Sep 18 19:39:44.037161 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:44.037139    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/75f125aa-6a99-40d4-914d-67c8efd07d39-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78d686bcb7-gmvm2\" (UID: \"75f125aa-6a99-40d4-914d-67c8efd07d39\") " pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2"
Sep 18 19:39:44.037250 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:44.037185    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/75f125aa-6a99-40d4-914d-67c8efd07d39-secret-grpc-tls\") pod \"thanos-querier-78d686bcb7-gmvm2\" (UID: \"75f125aa-6a99-40d4-914d-67c8efd07d39\") " pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2"
Sep 18 19:39:44.037250 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:44.037221    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/75f125aa-6a99-40d4-914d-67c8efd07d39-secret-thanos-querier-tls\") pod \"thanos-querier-78d686bcb7-gmvm2\" (UID: \"75f125aa-6a99-40d4-914d-67c8efd07d39\") " pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2"
Sep 18 19:39:44.037250 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:44.037236    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75f125aa-6a99-40d4-914d-67c8efd07d39-metrics-client-ca\") pod \"thanos-querier-78d686bcb7-gmvm2\" (UID: \"75f125aa-6a99-40d4-914d-67c8efd07d39\") " pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2"
Sep 18 19:39:44.037421 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:44.037306    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75f125aa-6a99-40d4-914d-67c8efd07d39-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78d686bcb7-gmvm2\" (UID: \"75f125aa-6a99-40d4-914d-67c8efd07d39\") " pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2"
Sep 18 19:39:44.037421 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:44.037388    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75f125aa-6a99-40d4-914d-67c8efd07d39-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78d686bcb7-gmvm2\" (UID: \"75f125aa-6a99-40d4-914d-67c8efd07d39\") " pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2"
Sep 18 19:39:44.037522 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:44.037425    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55fdj\" (UniqueName: \"kubernetes.io/projected/75f125aa-6a99-40d4-914d-67c8efd07d39-kube-api-access-55fdj\") pod \"thanos-querier-78d686bcb7-gmvm2\" (UID: \"75f125aa-6a99-40d4-914d-67c8efd07d39\") " pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2"
Sep 18 19:39:44.037522 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:44.037457    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/75f125aa-6a99-40d4-914d-67c8efd07d39-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78d686bcb7-gmvm2\" (UID: \"75f125aa-6a99-40d4-914d-67c8efd07d39\") " pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2"
Sep 18 19:39:44.138562 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:44.138535    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75f125aa-6a99-40d4-914d-67c8efd07d39-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78d686bcb7-gmvm2\" (UID: \"75f125aa-6a99-40d4-914d-67c8efd07d39\") " pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2"
Sep 18 19:39:44.138781 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:44.138763    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55fdj\" (UniqueName: \"kubernetes.io/projected/75f125aa-6a99-40d4-914d-67c8efd07d39-kube-api-access-55fdj\") pod \"thanos-querier-78d686bcb7-gmvm2\" (UID: \"75f125aa-6a99-40d4-914d-67c8efd07d39\") " pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2"
Sep 18 19:39:44.138889 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:44.138813    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/75f125aa-6a99-40d4-914d-67c8efd07d39-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78d686bcb7-gmvm2\" (UID: \"75f125aa-6a99-40d4-914d-67c8efd07d39\") " pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2"
Sep 18 19:39:44.138889 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:44.138874    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/75f125aa-6a99-40d4-914d-67c8efd07d39-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78d686bcb7-gmvm2\" (UID: \"75f125aa-6a99-40d4-914d-67c8efd07d39\") " pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2"
Sep 18 19:39:44.139005 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:44.138944    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/75f125aa-6a99-40d4-914d-67c8efd07d39-secret-grpc-tls\") pod \"thanos-querier-78d686bcb7-gmvm2\" (UID: \"75f125aa-6a99-40d4-914d-67c8efd07d39\") " pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2"
Sep 18 19:39:44.139005 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:44.138999    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/75f125aa-6a99-40d4-914d-67c8efd07d39-secret-thanos-querier-tls\") pod \"thanos-querier-78d686bcb7-gmvm2\" (UID: \"75f125aa-6a99-40d4-914d-67c8efd07d39\") " pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2"
Sep 18 19:39:44.139102 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:44.139050    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75f125aa-6a99-40d4-914d-67c8efd07d39-metrics-client-ca\") pod \"thanos-querier-78d686bcb7-gmvm2\" (UID: \"75f125aa-6a99-40d4-914d-67c8efd07d39\") " pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2"
Sep 18 19:39:44.139156 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:44.139114    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75f125aa-6a99-40d4-914d-67c8efd07d39-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78d686bcb7-gmvm2\" (UID: \"75f125aa-6a99-40d4-914d-67c8efd07d39\") " pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2"
Sep 18 19:39:44.139826 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:44.139800    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75f125aa-6a99-40d4-914d-67c8efd07d39-metrics-client-ca\") pod \"thanos-querier-78d686bcb7-gmvm2\" (UID: \"75f125aa-6a99-40d4-914d-67c8efd07d39\") " pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2"
Sep 18 19:39:44.141863 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:44.141807    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75f125aa-6a99-40d4-914d-67c8efd07d39-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-78d686bcb7-gmvm2\" (UID: \"75f125aa-6a99-40d4-914d-67c8efd07d39\") " pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2"
Sep 18 19:39:44.141863 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:44.141856    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75f125aa-6a99-40d4-914d-67c8efd07d39-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-78d686bcb7-gmvm2\" (UID: \"75f125aa-6a99-40d4-914d-67c8efd07d39\") " pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2"
Sep 18 19:39:44.142003 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:44.141952    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/75f125aa-6a99-40d4-914d-67c8efd07d39-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-78d686bcb7-gmvm2\" (UID: \"75f125aa-6a99-40d4-914d-67c8efd07d39\") " pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2"
Sep 18 19:39:44.142116 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:44.142094    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/75f125aa-6a99-40d4-914d-67c8efd07d39-secret-thanos-querier-tls\") pod \"thanos-querier-78d686bcb7-gmvm2\" (UID: \"75f125aa-6a99-40d4-914d-67c8efd07d39\") " pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2"
Sep 18 19:39:44.142219 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:44.142203    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/75f125aa-6a99-40d4-914d-67c8efd07d39-secret-grpc-tls\") pod \"thanos-querier-78d686bcb7-gmvm2\" (UID: \"75f125aa-6a99-40d4-914d-67c8efd07d39\") " pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2"
Sep 18 19:39:44.142322 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:44.142303    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/75f125aa-6a99-40d4-914d-67c8efd07d39-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-78d686bcb7-gmvm2\" (UID: \"75f125aa-6a99-40d4-914d-67c8efd07d39\") " pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2"
Sep 18 19:39:44.146607 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:44.146590    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55fdj\" (UniqueName: \"kubernetes.io/projected/75f125aa-6a99-40d4-914d-67c8efd07d39-kube-api-access-55fdj\") pod \"thanos-querier-78d686bcb7-gmvm2\" (UID: \"75f125aa-6a99-40d4-914d-67c8efd07d39\") " pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2"
Sep 18 19:39:44.230243 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:44.230192    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2"
Sep 18 19:39:44.504584 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:44.504562    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-78d686bcb7-gmvm2"]
Sep 18 19:39:44.506484 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:39:44.506455    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f125aa_6a99_40d4_914d_67c8efd07d39.slice/crio-5989f8fc7ea83c85fbeacd9ba1ab594d2a30c02b06edfacd86a8131007e553d7 WatchSource:0}: Error finding container 5989f8fc7ea83c85fbeacd9ba1ab594d2a30c02b06edfacd86a8131007e553d7: Status 404 returned error can't find the container with id 5989f8fc7ea83c85fbeacd9ba1ab594d2a30c02b06edfacd86a8131007e553d7
Sep 18 19:39:44.967245 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:44.967210    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bdftw" event={"ID":"9c2ad484-0f94-4f69-84a8-4fb65d4e8d53","Type":"ContainerStarted","Data":"fd0e8d4c559d0f79b72da9be76ac5b73cdac9533cb002fe2a7215223bfdabe2d"}
Sep 18 19:39:44.967721 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:44.967275    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bdftw" event={"ID":"9c2ad484-0f94-4f69-84a8-4fb65d4e8d53","Type":"ContainerStarted","Data":"5424401efe13e57da02314972456b5f779ad543d1c9815e38cd681a52af94367"}
Sep 18 19:39:44.968995 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:44.968967    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2" event={"ID":"75f125aa-6a99-40d4-914d-67c8efd07d39","Type":"ContainerStarted","Data":"5989f8fc7ea83c85fbeacd9ba1ab594d2a30c02b06edfacd86a8131007e553d7"}
Sep 18 19:39:44.989415 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:44.989368    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bdftw" podStartSLOduration=2.93769927 podStartE2EDuration="3.989352691s" totalImagesPullingTime="982.196788ms" totalInitContainerRuntime="69.456633ms" isStatefulPod=true podCreationTimestamp="2026-09-18 19:39:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:44.988077966 +0000 UTC m=+108.090366299" watchObservedRunningTime="2026-09-18 19:39:44.989352691 +0000 UTC m=+108.091641053"
Sep 18 19:39:46.591913 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:46.591884    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-r4twg"]
Sep 18 19:39:46.593988 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:46.593967    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-r4twg"
Sep 18 19:39:46.596264 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:46.596242    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-2s7gq\""
Sep 18 19:39:46.596407 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:46.596388    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 18 19:39:46.603434 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:46.603416    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-r4twg"]
Sep 18 19:39:46.762468 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:46.762430    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrtq9\" (UniqueName: \"kubernetes.io/projected/44301539-88b6-4be8-8475-a1e987e81370-kube-api-access-mrtq9\") pod \"monitoring-plugin-5984459ff8-r4twg\" (UID: \"44301539-88b6-4be8-8475-a1e987e81370\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-r4twg"
Sep 18 19:39:46.762616 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:46.762541    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/44301539-88b6-4be8-8475-a1e987e81370-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-r4twg\" (UID: \"44301539-88b6-4be8-8475-a1e987e81370\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-r4twg"
Sep 18 19:39:46.863929 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:46.863846    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/44301539-88b6-4be8-8475-a1e987e81370-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-r4twg\" (UID: \"44301539-88b6-4be8-8475-a1e987e81370\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-r4twg"
Sep 18 19:39:46.864101 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:46.863930    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrtq9\" (UniqueName: \"kubernetes.io/projected/44301539-88b6-4be8-8475-a1e987e81370-kube-api-access-mrtq9\") pod \"monitoring-plugin-5984459ff8-r4twg\" (UID: \"44301539-88b6-4be8-8475-a1e987e81370\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-r4twg"
Sep 18 19:39:46.864101 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:46.863965    2581 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 18 19:39:46.864101 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:39:46.864043    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44301539-88b6-4be8-8475-a1e987e81370-monitoring-plugin-cert podName:44301539-88b6-4be8-8475-a1e987e81370 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:47.364021712 +0000 UTC m=+110.466310023 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/44301539-88b6-4be8-8475-a1e987e81370-monitoring-plugin-cert") pod "monitoring-plugin-5984459ff8-r4twg" (UID: "44301539-88b6-4be8-8475-a1e987e81370") : secret "monitoring-plugin-cert" not found
Sep 18 19:39:46.872985 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:46.872961    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrtq9\" (UniqueName: \"kubernetes.io/projected/44301539-88b6-4be8-8475-a1e987e81370-kube-api-access-mrtq9\") pod \"monitoring-plugin-5984459ff8-r4twg\" (UID: \"44301539-88b6-4be8-8475-a1e987e81370\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-r4twg"
Sep 18 19:39:46.980000 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:46.979966    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2" event={"ID":"75f125aa-6a99-40d4-914d-67c8efd07d39","Type":"ContainerStarted","Data":"78396138d1b3e30c5d3d1a7fdbd4ee465647191f6cb8e3b9fe6156d533b1ef54"}
Sep 18 19:39:46.980150 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:46.980042    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2" event={"ID":"75f125aa-6a99-40d4-914d-67c8efd07d39","Type":"ContainerStarted","Data":"1538cc3a82475b959842c2b657d1c505e31982555f7d92dd6c10f648a09cf4fe"}
Sep 18 19:39:46.980150 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:46.980078    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2" event={"ID":"75f125aa-6a99-40d4-914d-67c8efd07d39","Type":"ContainerStarted","Data":"41b74f40b9b1bc41b85bb8af29b68e241d222a40ed3edcca182d9d0badd5724f"}
Sep 18 19:39:47.368387 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:47.368310    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/44301539-88b6-4be8-8475-a1e987e81370-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-r4twg\" (UID: \"44301539-88b6-4be8-8475-a1e987e81370\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-r4twg"
Sep 18 19:39:47.370720 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:47.370691    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/44301539-88b6-4be8-8475-a1e987e81370-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-r4twg\" (UID: \"44301539-88b6-4be8-8475-a1e987e81370\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-r4twg"
Sep 18 19:39:47.502900 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:47.502874    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-r4twg"
Sep 18 19:39:47.770705 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:47.770678    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-r4twg"]
Sep 18 19:39:47.773307 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:39:47.773286    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44301539_88b6_4be8_8475_a1e987e81370.slice/crio-6669e9c6b62b69b01c8ce88aa8e0c31d577a3b830490e4ab6341d005f50ed046 WatchSource:0}: Error finding container 6669e9c6b62b69b01c8ce88aa8e0c31d577a3b830490e4ab6341d005f50ed046: Status 404 returned error can't find the container with id 6669e9c6b62b69b01c8ce88aa8e0c31d577a3b830490e4ab6341d005f50ed046
Sep 18 19:39:47.986738 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:47.986694    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2" event={"ID":"75f125aa-6a99-40d4-914d-67c8efd07d39","Type":"ContainerStarted","Data":"eff1bd4be4be1aa390bdd1998fc310604e238dfe26e0ec922974369a39f792d7"}
Sep 18 19:39:47.986934 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:47.986769    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2" event={"ID":"75f125aa-6a99-40d4-914d-67c8efd07d39","Type":"ContainerStarted","Data":"89c96d211a8bec3896bcb45c5f18a01705ae8984859bc7ecfc83665ed756d5fc"}
Sep 18 19:39:47.986934 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:47.986818    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2"
Sep 18 19:39:47.986934 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:47.986868    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2" event={"ID":"75f125aa-6a99-40d4-914d-67c8efd07d39","Type":"ContainerStarted","Data":"0baddc714e9a11ec647f1f614e61d9a973dfe4444950436ee6bf82195a0d3ba6"}
Sep 18 19:39:47.987930 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:47.987908    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-r4twg" event={"ID":"44301539-88b6-4be8-8475-a1e987e81370","Type":"ContainerStarted","Data":"6669e9c6b62b69b01c8ce88aa8e0c31d577a3b830490e4ab6341d005f50ed046"}
Sep 18 19:39:48.009994 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:48.009950    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2" podStartSLOduration=2.637581635 podStartE2EDuration="5.009936913s" totalImagesPullingTime="2.372355278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:48.007935704 +0000 UTC m=+111.110224033" watchObservedRunningTime="2026-09-18 19:39:48.009936913 +0000 UTC m=+111.112225273"
Sep 18 19:39:48.558221 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:48.558190    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-86f46f546-98c44"]
Sep 18 19:39:49.943195 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:49.943166    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dwtpk"
Sep 18 19:39:49.997339 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:49.997304    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-r4twg" event={"ID":"44301539-88b6-4be8-8475-a1e987e81370","Type":"ContainerStarted","Data":"4daf109d36c828fac49ce29694045a0a626ea9669e18bfdf18c7c3becb8c7eda"}
Sep 18 19:39:49.997544 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:49.997524    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-r4twg"
Sep 18 19:39:50.002307 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:50.002284    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-r4twg"
Sep 18 19:39:50.012443 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:50.012399    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-r4twg" podStartSLOduration=2.834048869 podStartE2EDuration="4.012385956s" totalImagesPullingTime="1.178337087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:50.011273934 +0000 UTC m=+113.113562263" watchObservedRunningTime="2026-09-18 19:39:50.012385956 +0000 UTC m=+113.114674283"
Sep 18 19:39:53.997870 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:53.997826    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-78d686bcb7-gmvm2"
Sep 18 19:39:58.564205 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:39:58.564173    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:40:07.208562 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:07.208529    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1173b48c-041e-4ec6-9708-98391b2bf0fe-metrics-certs\") pod \"network-metrics-daemon-s9j6n\" (UID: \"1173b48c-041e-4ec6-9708-98391b2bf0fe\") " pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:40:07.210964 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:07.210933    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1173b48c-041e-4ec6-9708-98391b2bf0fe-metrics-certs\") pod \"network-metrics-daemon-s9j6n\" (UID: \"1173b48c-041e-4ec6-9708-98391b2bf0fe\") " pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:40:07.247959 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:07.247936    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8c9m8\""
Sep 18 19:40:07.255950 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:07.255932    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s9j6n"
Sep 18 19:40:07.617136 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:07.616959    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s9j6n"]
Sep 18 19:40:07.619620 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:40:07.619598    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1173b48c_041e_4ec6_9708_98391b2bf0fe.slice/crio-2ddf3b92cfc56999331d822e6ccb9b485ac65568d980ebb3f8099955c07e625c WatchSource:0}: Error finding container 2ddf3b92cfc56999331d822e6ccb9b485ac65568d980ebb3f8099955c07e625c: Status 404 returned error can't find the container with id 2ddf3b92cfc56999331d822e6ccb9b485ac65568d980ebb3f8099955c07e625c
Sep 18 19:40:08.063697 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:08.063664    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s9j6n" event={"ID":"1173b48c-041e-4ec6-9708-98391b2bf0fe","Type":"ContainerStarted","Data":"2ddf3b92cfc56999331d822e6ccb9b485ac65568d980ebb3f8099955c07e625c"}
Sep 18 19:40:10.072959 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:10.072923    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s9j6n" event={"ID":"1173b48c-041e-4ec6-9708-98391b2bf0fe","Type":"ContainerStarted","Data":"664b56c2c52fb1a7f467931ab19c242c4aa7f122a63139db4e9b63a8a0f2b6d2"}
Sep 18 19:40:10.073367 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:10.072976    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s9j6n" event={"ID":"1173b48c-041e-4ec6-9708-98391b2bf0fe","Type":"ContainerStarted","Data":"77c464f5a8e4b5a4d358882432bd578f081f12b9001707c621f8393c73b550a1"}
Sep 18 19:40:10.088607 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:10.088553    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-s9j6n" podStartSLOduration=130.625212876 podStartE2EDuration="2m12.088536691s" totalImagesPullingTime="1.463323815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:40:10.087005239 +0000 UTC m=+133.189293567" watchObservedRunningTime="2026-09-18 19:40:10.088536691 +0000 UTC m=+133.190825018"
Sep 18 19:40:12.083474 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:12.083440    2581 generic.go:356] "Generic (PLEG): container finished" podID="e764d634-1d03-429d-8454-d725e6be456f" containerID="f54a8990146ae524131607b70247bdbffe3c3e479d125e68894a1f022b3deeba" exitCode=0
Sep 18 19:40:12.083888 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:12.083486    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-ffj79" event={"ID":"e764d634-1d03-429d-8454-d725e6be456f","Type":"ContainerDied","Data":"f54a8990146ae524131607b70247bdbffe3c3e479d125e68894a1f022b3deeba"}
Sep 18 19:40:12.083888 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:12.083816    2581 scope.go:117] "RemoveContainer" containerID="f54a8990146ae524131607b70247bdbffe3c3e479d125e68894a1f022b3deeba"
Sep 18 19:40:13.089263 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.089229    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-ffj79" event={"ID":"e764d634-1d03-429d-8454-d725e6be456f","Type":"ContainerStarted","Data":"a8c8ea44880821c09428fee1cfcff8c0f3156cb54bbb27f4ff1b3a3a49ffe428"}
Sep 18 19:40:13.124011 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.123984    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b797584fc-mzjpt"]
Sep 18 19:40:13.126052 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.126036    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b797584fc-mzjpt"
Sep 18 19:40:13.128359 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.128337    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 19:40:13.128569 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.128553    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 19:40:13.128655 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.128574    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 19:40:13.128655 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.128615    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 19:40:13.128655 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.128553    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 19:40:13.129018 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.128992    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 19:40:13.129096 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.129016    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hrjgw\""
Sep 18 19:40:13.129646 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.129629    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 19:40:13.134089 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.134066    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 18 19:40:13.137228 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.137207    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b797584fc-mzjpt"]
Sep 18 19:40:13.152769 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.152735    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd27da47-cec6-4b54-9115-9df09bf39017-console-serving-cert\") pod \"console-6b797584fc-mzjpt\" (UID: \"cd27da47-cec6-4b54-9115-9df09bf39017\") " pod="openshift-console/console-6b797584fc-mzjpt"
Sep 18 19:40:13.152942 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.152788    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hp5k\" (UniqueName: \"kubernetes.io/projected/cd27da47-cec6-4b54-9115-9df09bf39017-kube-api-access-6hp5k\") pod \"console-6b797584fc-mzjpt\" (UID: \"cd27da47-cec6-4b54-9115-9df09bf39017\") " pod="openshift-console/console-6b797584fc-mzjpt"
Sep 18 19:40:13.153190 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.153168    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd27da47-cec6-4b54-9115-9df09bf39017-oauth-serving-cert\") pod \"console-6b797584fc-mzjpt\" (UID: \"cd27da47-cec6-4b54-9115-9df09bf39017\") " pod="openshift-console/console-6b797584fc-mzjpt"
Sep 18 19:40:13.153282 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.153239    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd27da47-cec6-4b54-9115-9df09bf39017-console-oauth-config\") pod \"console-6b797584fc-mzjpt\" (UID: \"cd27da47-cec6-4b54-9115-9df09bf39017\") " pod="openshift-console/console-6b797584fc-mzjpt"
Sep 18 19:40:13.153282 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.153270    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd27da47-cec6-4b54-9115-9df09bf39017-service-ca\") pod \"console-6b797584fc-mzjpt\" (UID: \"cd27da47-cec6-4b54-9115-9df09bf39017\") " pod="openshift-console/console-6b797584fc-mzjpt"
Sep 18 19:40:13.153433 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.153307    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd27da47-cec6-4b54-9115-9df09bf39017-trusted-ca-bundle\") pod \"console-6b797584fc-mzjpt\" (UID: \"cd27da47-cec6-4b54-9115-9df09bf39017\") " pod="openshift-console/console-6b797584fc-mzjpt"
Sep 18 19:40:13.153517 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.153437    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cd27da47-cec6-4b54-9115-9df09bf39017-console-config\") pod \"console-6b797584fc-mzjpt\" (UID: \"cd27da47-cec6-4b54-9115-9df09bf39017\") " pod="openshift-console/console-6b797584fc-mzjpt"
Sep 18 19:40:13.254553 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.254522    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd27da47-cec6-4b54-9115-9df09bf39017-oauth-serving-cert\") pod \"console-6b797584fc-mzjpt\" (UID: \"cd27da47-cec6-4b54-9115-9df09bf39017\") " pod="openshift-console/console-6b797584fc-mzjpt"
Sep 18 19:40:13.254553 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.254557    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd27da47-cec6-4b54-9115-9df09bf39017-console-oauth-config\") pod \"console-6b797584fc-mzjpt\" (UID: \"cd27da47-cec6-4b54-9115-9df09bf39017\") " pod="openshift-console/console-6b797584fc-mzjpt"
Sep 18 19:40:13.254814 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.254574    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd27da47-cec6-4b54-9115-9df09bf39017-service-ca\") pod \"console-6b797584fc-mzjpt\" (UID: \"cd27da47-cec6-4b54-9115-9df09bf39017\") " pod="openshift-console/console-6b797584fc-mzjpt"
Sep 18 19:40:13.254814 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.254701    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd27da47-cec6-4b54-9115-9df09bf39017-trusted-ca-bundle\") pod \"console-6b797584fc-mzjpt\" (UID: \"cd27da47-cec6-4b54-9115-9df09bf39017\") " pod="openshift-console/console-6b797584fc-mzjpt"
Sep 18 19:40:13.254814 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.254734    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cd27da47-cec6-4b54-9115-9df09bf39017-console-config\") pod \"console-6b797584fc-mzjpt\" (UID: \"cd27da47-cec6-4b54-9115-9df09bf39017\") " pod="openshift-console/console-6b797584fc-mzjpt"
Sep 18 19:40:13.254814 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.254763    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd27da47-cec6-4b54-9115-9df09bf39017-console-serving-cert\") pod \"console-6b797584fc-mzjpt\" (UID: \"cd27da47-cec6-4b54-9115-9df09bf39017\") " pod="openshift-console/console-6b797584fc-mzjpt"
Sep 18 19:40:13.254814 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.254790    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hp5k\" (UniqueName: \"kubernetes.io/projected/cd27da47-cec6-4b54-9115-9df09bf39017-kube-api-access-6hp5k\") pod \"console-6b797584fc-mzjpt\" (UID: \"cd27da47-cec6-4b54-9115-9df09bf39017\") " pod="openshift-console/console-6b797584fc-mzjpt"
Sep 18 19:40:13.255506 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.255407    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cd27da47-cec6-4b54-9115-9df09bf39017-console-config\") pod \"console-6b797584fc-mzjpt\" (UID: \"cd27da47-cec6-4b54-9115-9df09bf39017\") " pod="openshift-console/console-6b797584fc-mzjpt"
Sep 18 19:40:13.255506 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.255431    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd27da47-cec6-4b54-9115-9df09bf39017-oauth-serving-cert\") pod \"console-6b797584fc-mzjpt\" (UID: \"cd27da47-cec6-4b54-9115-9df09bf39017\") " pod="openshift-console/console-6b797584fc-mzjpt"
Sep 18 19:40:13.255661 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.255577    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd27da47-cec6-4b54-9115-9df09bf39017-trusted-ca-bundle\") pod \"console-6b797584fc-mzjpt\" (UID: \"cd27da47-cec6-4b54-9115-9df09bf39017\") " pod="openshift-console/console-6b797584fc-mzjpt"
Sep 18 19:40:13.255661 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.255603    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd27da47-cec6-4b54-9115-9df09bf39017-service-ca\") pod \"console-6b797584fc-mzjpt\" (UID: \"cd27da47-cec6-4b54-9115-9df09bf39017\") " pod="openshift-console/console-6b797584fc-mzjpt"
Sep 18 19:40:13.257264 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.257241    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd27da47-cec6-4b54-9115-9df09bf39017-console-serving-cert\") pod \"console-6b797584fc-mzjpt\" (UID: \"cd27da47-cec6-4b54-9115-9df09bf39017\") " pod="openshift-console/console-6b797584fc-mzjpt"
Sep 18 19:40:13.257358 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.257270    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd27da47-cec6-4b54-9115-9df09bf39017-console-oauth-config\") pod \"console-6b797584fc-mzjpt\" (UID: \"cd27da47-cec6-4b54-9115-9df09bf39017\") " pod="openshift-console/console-6b797584fc-mzjpt"
Sep 18 19:40:13.262762 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.262739    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hp5k\" (UniqueName: \"kubernetes.io/projected/cd27da47-cec6-4b54-9115-9df09bf39017-kube-api-access-6hp5k\") pod \"console-6b797584fc-mzjpt\" (UID: \"cd27da47-cec6-4b54-9115-9df09bf39017\") " pod="openshift-console/console-6b797584fc-mzjpt"
Sep 18 19:40:13.436718 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.436647    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b797584fc-mzjpt"
Sep 18 19:40:13.581040 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.580994    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-86f46f546-98c44" podUID="80dda80c-b4fe-4ab8-9f8b-ef73f7b04756" containerName="registry" containerID="cri-o://63e4ee6e4a2a1b83565ab59c48437aac6652535a5676c0aeab13cd654fb8f284" gracePeriod=30
Sep 18 19:40:13.707381 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.705962    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b797584fc-mzjpt"]
Sep 18 19:40:13.709647 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:40:13.709619    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd27da47_cec6_4b54_9115_9df09bf39017.slice/crio-391488ec472dc63b044e5e3923595701a71f881fee29d41c66e44d48a7686184 WatchSource:0}: Error finding container 391488ec472dc63b044e5e3923595701a71f881fee29d41c66e44d48a7686184: Status 404 returned error can't find the container with id 391488ec472dc63b044e5e3923595701a71f881fee29d41c66e44d48a7686184
Sep 18 19:40:13.808546 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.808524    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:40:13.860877 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.860830    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-certificates\") pod \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") "
Sep 18 19:40:13.861025 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.860883    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-trusted-ca\") pod \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") "
Sep 18 19:40:13.861025 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.860909    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-ca-trust-extracted\") pod \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") "
Sep 18 19:40:13.861025 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.860939    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-image-registry-private-configuration\") pod \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") "
Sep 18 19:40:13.861025 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.860967    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-tls\") pod \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") "
Sep 18 19:40:13.861025 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.860999    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7c5tj\" (UniqueName: \"kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-kube-api-access-7c5tj\") pod \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") "
Sep 18 19:40:13.861025 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.861022    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-installation-pull-secrets\") pod \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") "
Sep 18 19:40:13.861285 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.861048    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-bound-sa-token\") pod \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\" (UID: \"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756\") "
Sep 18 19:40:13.861341 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.861283    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "80dda80c-b4fe-4ab8-9f8b-ef73f7b04756" (UID: "80dda80c-b4fe-4ab8-9f8b-ef73f7b04756"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:40:13.861392 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.861375    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "80dda80c-b4fe-4ab8-9f8b-ef73f7b04756" (UID: "80dda80c-b4fe-4ab8-9f8b-ef73f7b04756"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:40:13.863718 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.863693    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-kube-api-access-7c5tj" (OuterVolumeSpecName: "kube-api-access-7c5tj") pod "80dda80c-b4fe-4ab8-9f8b-ef73f7b04756" (UID: "80dda80c-b4fe-4ab8-9f8b-ef73f7b04756"). InnerVolumeSpecName "kube-api-access-7c5tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:40:13.863862 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.863728    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "80dda80c-b4fe-4ab8-9f8b-ef73f7b04756" (UID: "80dda80c-b4fe-4ab8-9f8b-ef73f7b04756"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:40:13.863862 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.863742    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "80dda80c-b4fe-4ab8-9f8b-ef73f7b04756" (UID: "80dda80c-b4fe-4ab8-9f8b-ef73f7b04756"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:40:13.863955 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.863917    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "80dda80c-b4fe-4ab8-9f8b-ef73f7b04756" (UID: "80dda80c-b4fe-4ab8-9f8b-ef73f7b04756"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:40:13.863955 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.863940    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "80dda80c-b4fe-4ab8-9f8b-ef73f7b04756" (UID: "80dda80c-b4fe-4ab8-9f8b-ef73f7b04756"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:40:13.869325 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.869305    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "80dda80c-b4fe-4ab8-9f8b-ef73f7b04756" (UID: "80dda80c-b4fe-4ab8-9f8b-ef73f7b04756"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:40:13.962272 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.962189    2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-image-registry-private-configuration\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:40:13.962272 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.962216    2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-tls\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:40:13.962272 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.962228    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c5tj\" (UniqueName: \"kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-kube-api-access-7c5tj\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:40:13.962272 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.962237    2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-installation-pull-secrets\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:40:13.962272 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.962247    2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-bound-sa-token\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:40:13.962272 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.962255    2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-registry-certificates\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:40:13.962272 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.962262    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-trusted-ca\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:40:13.962272 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:13.962270    2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756-ca-trust-extracted\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:40:14.097756 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:14.097705    2581 generic.go:356] "Generic (PLEG): container finished" podID="80dda80c-b4fe-4ab8-9f8b-ef73f7b04756" containerID="63e4ee6e4a2a1b83565ab59c48437aac6652535a5676c0aeab13cd654fb8f284" exitCode=0
Sep 18 19:40:14.098229 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:14.097825    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86f46f546-98c44" event={"ID":"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756","Type":"ContainerDied","Data":"63e4ee6e4a2a1b83565ab59c48437aac6652535a5676c0aeab13cd654fb8f284"}
Sep 18 19:40:14.098229 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:14.097894    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86f46f546-98c44"
Sep 18 19:40:14.098229 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:14.097933    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86f46f546-98c44" event={"ID":"80dda80c-b4fe-4ab8-9f8b-ef73f7b04756","Type":"ContainerDied","Data":"23d044069180f86e1f111e2dd0ba81d60fbcd322ff340b3f44a01a19df8e2746"}
Sep 18 19:40:14.098229 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:14.097962    2581 scope.go:117] "RemoveContainer" containerID="63e4ee6e4a2a1b83565ab59c48437aac6652535a5676c0aeab13cd654fb8f284"
Sep 18 19:40:14.100426 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:14.100394    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b797584fc-mzjpt" event={"ID":"cd27da47-cec6-4b54-9115-9df09bf39017","Type":"ContainerStarted","Data":"391488ec472dc63b044e5e3923595701a71f881fee29d41c66e44d48a7686184"}
Sep 18 19:40:14.109652 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:14.109629    2581 scope.go:117] "RemoveContainer" containerID="63e4ee6e4a2a1b83565ab59c48437aac6652535a5676c0aeab13cd654fb8f284"
Sep 18 19:40:14.110256 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:40:14.110227    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63e4ee6e4a2a1b83565ab59c48437aac6652535a5676c0aeab13cd654fb8f284\": container with ID starting with 63e4ee6e4a2a1b83565ab59c48437aac6652535a5676c0aeab13cd654fb8f284 not found: ID does not exist" containerID="63e4ee6e4a2a1b83565ab59c48437aac6652535a5676c0aeab13cd654fb8f284"
Sep 18 19:40:14.110344 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:14.110269    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63e4ee6e4a2a1b83565ab59c48437aac6652535a5676c0aeab13cd654fb8f284"} err="failed to get container status \"63e4ee6e4a2a1b83565ab59c48437aac6652535a5676c0aeab13cd654fb8f284\": rpc error: code = NotFound desc = could not find container \"63e4ee6e4a2a1b83565ab59c48437aac6652535a5676c0aeab13cd654fb8f284\": container with ID starting with 63e4ee6e4a2a1b83565ab59c48437aac6652535a5676c0aeab13cd654fb8f284 not found: ID does not exist"
Sep 18 19:40:14.125023 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:14.124983    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-86f46f546-98c44"]
Sep 18 19:40:14.129150 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:14.129127    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-86f46f546-98c44"]
Sep 18 19:40:15.434182 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:15.434147    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80dda80c-b4fe-4ab8-9f8b-ef73f7b04756" path="/var/lib/kubelet/pods/80dda80c-b4fe-4ab8-9f8b-ef73f7b04756/volumes"
Sep 18 19:40:18.121829 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:18.121795    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b797584fc-mzjpt" event={"ID":"cd27da47-cec6-4b54-9115-9df09bf39017","Type":"ContainerStarted","Data":"cf17820eed8b97467194cadab93e313bb07ef6bfcda2fbb1e0b30d568b4826e2"}
Sep 18 19:40:18.138729 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:18.138676    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b797584fc-mzjpt" podStartSLOduration=1.34574384 podStartE2EDuration="5.138657064s" totalImagesPullingTime="3.792913224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:40:18.13794461 +0000 UTC m=+141.240232937" watchObservedRunningTime="2026-09-18 19:40:18.138657064 +0000 UTC m=+141.240945393"
Sep 18 19:40:23.437355 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:23.437327    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b797584fc-mzjpt"
Sep 18 19:40:23.437778 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:23.437376    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b797584fc-mzjpt"
Sep 18 19:40:23.443059 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:23.443038    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b797584fc-mzjpt"
Sep 18 19:40:24.155347 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:24.155307    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b797584fc-mzjpt"
Sep 18 19:40:33.185489 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:33.185450    2581 generic.go:356] "Generic (PLEG): container finished" podID="85c9bace-8688-4f4a-ad42-dc32911c6263" containerID="ad31619dc7eda3f7fcfabdf96de59d3f41c14582cd6893ef27fa1083fec3ce51" exitCode=0
Sep 18 19:40:33.185871 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:33.185525    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-8l864" event={"ID":"85c9bace-8688-4f4a-ad42-dc32911c6263","Type":"ContainerDied","Data":"ad31619dc7eda3f7fcfabdf96de59d3f41c14582cd6893ef27fa1083fec3ce51"}
Sep 18 19:40:33.185871 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:33.185810    2581 scope.go:117] "RemoveContainer" containerID="ad31619dc7eda3f7fcfabdf96de59d3f41c14582cd6893ef27fa1083fec3ce51"
Sep 18 19:40:34.191088 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:40:34.191058    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-8l864" event={"ID":"85c9bace-8688-4f4a-ad42-dc32911c6263","Type":"ContainerStarted","Data":"1a6b1c623afc3595fe45f2ba10a1c2cdecf69b914b24893f1172368acd5ccfb5"}
Sep 18 19:41:03.802885 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:03.802830    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:41:03.803367 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:03.803150    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80dda80c-b4fe-4ab8-9f8b-ef73f7b04756" containerName="registry"
Sep 18 19:41:03.803367 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:03.803164    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="80dda80c-b4fe-4ab8-9f8b-ef73f7b04756" containerName="registry"
Sep 18 19:41:03.803367 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:03.803219    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="80dda80c-b4fe-4ab8-9f8b-ef73f7b04756" containerName="registry"
Sep 18 19:41:03.807432 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:03.807414    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:03.809737 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:03.809717    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 18 19:41:03.810041 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:03.810021    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 18 19:41:03.810177 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:03.810158    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 18 19:41:03.810258 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:03.810118    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 18 19:41:03.810258 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:03.810116    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 18 19:41:03.810258 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:03.810239    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-k9cl7\""
Sep 18 19:41:03.810258 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:03.810167    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 18 19:41:03.810521 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:03.810138    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 18 19:41:03.810521 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:03.810118    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 18 19:41:03.814924 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:03.814903    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 18 19:41:03.819076 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:03.819055    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:41:03.957863 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:03.957805    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:03.958033 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:03.957876    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-config-out\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:03.958033 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:03.957903    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:03.958033 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:03.957930    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:03.958033 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:03.957957    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:03.958033 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:03.958003    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:03.958033 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:03.958018    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:03.958217 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:03.958043    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:03.958217 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:03.958069    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbgt6\" (UniqueName: \"kubernetes.io/projected/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-kube-api-access-sbgt6\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:03.958217 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:03.958086    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:03.958217 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:03.958145    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-web-config\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:03.958217 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:03.958174    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-config-volume\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:03.958364 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:03.958219    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:04.058611 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:04.058525    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbgt6\" (UniqueName: \"kubernetes.io/projected/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-kube-api-access-sbgt6\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:04.058611 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:04.058558    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:04.058611 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:04.058576    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-web-config\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:04.058611 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:04.058592    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-config-volume\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:04.058925 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:04.058625    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:04.058925 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:04.058656    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:04.058925 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:04.058678    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-config-out\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:04.058925 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:04.058705    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:04.058925 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:04.058742    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:04.058925 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:04.058763    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:04.058925 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:04.058792    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:04.058925 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:04.058815    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:04.058925 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:04.058859    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:04.060425 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:04.060148    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:04.060425 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:04.060321    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:04.060425 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:04.060368    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:04.062338 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:04.062302    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:04.062458 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:04.062428    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-config-volume\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:04.062564 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:04.062487    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:04.062631 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:04.062580    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:04.063128 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:04.063104    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-web-config\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:04.063128 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:04.063120    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:04.063242 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:04.063107    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-config-out\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:04.063317 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:04.063299    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:04.063571 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:04.063552    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:04.066729 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:04.066710    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbgt6\" (UniqueName: \"kubernetes.io/projected/ee7a81a5-d588-4320-8b6f-5dbb28fa5b87-kube-api-access-sbgt6\") pod \"alertmanager-main-0\" (UID: \"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:04.117107 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:04.117080    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 19:41:04.398177 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:04.398149    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 19:41:04.398557 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:41:04.398534    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee7a81a5_d588_4320_8b6f_5dbb28fa5b87.slice/crio-034b6a40f03f3662e0a321e85c77c9ac4c39e2ec01774e5dfbf7c622c565e2f1 WatchSource:0}: Error finding container 034b6a40f03f3662e0a321e85c77c9ac4c39e2ec01774e5dfbf7c622c565e2f1: Status 404 returned error can't find the container with id 034b6a40f03f3662e0a321e85c77c9ac4c39e2ec01774e5dfbf7c622c565e2f1
Sep 18 19:41:05.307561 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:05.307522    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87","Type":"ContainerStarted","Data":"034b6a40f03f3662e0a321e85c77c9ac4c39e2ec01774e5dfbf7c622c565e2f1"}
Sep 18 19:41:06.188225 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.188189    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-57b7d86db6-pbz6t"]
Sep 18 19:41:06.190294 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.190279    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-57b7d86db6-pbz6t"
Sep 18 19:41:06.194358 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.194339    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 18 19:41:06.194614 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.194597    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-xvdwl\""
Sep 18 19:41:06.194970 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.194953    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 18 19:41:06.195694 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.195678    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 18 19:41:06.196141 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.195980    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 18 19:41:06.196141 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.195979    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 18 19:41:06.199577 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.199558    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 18 19:41:06.205280 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.205261    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-57b7d86db6-pbz6t"]
Sep 18 19:41:06.275989 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.275952    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xm97\" (UniqueName: \"kubernetes.io/projected/0eddddab-ac2f-4633-bb6a-01a896804307-kube-api-access-7xm97\") pod \"telemeter-client-57b7d86db6-pbz6t\" (UID: \"0eddddab-ac2f-4633-bb6a-01a896804307\") " pod="openshift-monitoring/telemeter-client-57b7d86db6-pbz6t"
Sep 18 19:41:06.275989 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.275991    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eddddab-ac2f-4633-bb6a-01a896804307-serving-certs-ca-bundle\") pod \"telemeter-client-57b7d86db6-pbz6t\" (UID: \"0eddddab-ac2f-4633-bb6a-01a896804307\") " pod="openshift-monitoring/telemeter-client-57b7d86db6-pbz6t"
Sep 18 19:41:06.276188 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.276022    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eddddab-ac2f-4633-bb6a-01a896804307-telemeter-trusted-ca-bundle\") pod \"telemeter-client-57b7d86db6-pbz6t\" (UID: \"0eddddab-ac2f-4633-bb6a-01a896804307\") " pod="openshift-monitoring/telemeter-client-57b7d86db6-pbz6t"
Sep 18 19:41:06.276188 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.276105    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0eddddab-ac2f-4633-bb6a-01a896804307-secret-telemeter-client\") pod \"telemeter-client-57b7d86db6-pbz6t\" (UID: \"0eddddab-ac2f-4633-bb6a-01a896804307\") " pod="openshift-monitoring/telemeter-client-57b7d86db6-pbz6t"
Sep 18 19:41:06.276188 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.276159    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0eddddab-ac2f-4633-bb6a-01a896804307-federate-client-tls\") pod \"telemeter-client-57b7d86db6-pbz6t\" (UID: \"0eddddab-ac2f-4633-bb6a-01a896804307\") " pod="openshift-monitoring/telemeter-client-57b7d86db6-pbz6t"
Sep 18 19:41:06.276188 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.276184    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0eddddab-ac2f-4633-bb6a-01a896804307-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-57b7d86db6-pbz6t\" (UID: \"0eddddab-ac2f-4633-bb6a-01a896804307\") " pod="openshift-monitoring/telemeter-client-57b7d86db6-pbz6t"
Sep 18 19:41:06.276311 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.276213    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0eddddab-ac2f-4633-bb6a-01a896804307-telemeter-client-tls\") pod \"telemeter-client-57b7d86db6-pbz6t\" (UID: \"0eddddab-ac2f-4633-bb6a-01a896804307\") " pod="openshift-monitoring/telemeter-client-57b7d86db6-pbz6t"
Sep 18 19:41:06.276311 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.276261    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0eddddab-ac2f-4633-bb6a-01a896804307-metrics-client-ca\") pod \"telemeter-client-57b7d86db6-pbz6t\" (UID: \"0eddddab-ac2f-4633-bb6a-01a896804307\") " pod="openshift-monitoring/telemeter-client-57b7d86db6-pbz6t"
Sep 18 19:41:06.313056 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.313030    2581 generic.go:356] "Generic (PLEG): container finished" podID="ee7a81a5-d588-4320-8b6f-5dbb28fa5b87" containerID="321269609fef80be20c00aff236f636614b8301c8b7a2c8915da4e599945b4b8" exitCode=0
Sep 18 19:41:06.313378 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.313124    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87","Type":"ContainerDied","Data":"321269609fef80be20c00aff236f636614b8301c8b7a2c8915da4e599945b4b8"}
Sep 18 19:41:06.377088 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.377054    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eddddab-ac2f-4633-bb6a-01a896804307-serving-certs-ca-bundle\") pod \"telemeter-client-57b7d86db6-pbz6t\" (UID: \"0eddddab-ac2f-4633-bb6a-01a896804307\") " pod="openshift-monitoring/telemeter-client-57b7d86db6-pbz6t"
Sep 18 19:41:06.377088 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.377096    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eddddab-ac2f-4633-bb6a-01a896804307-telemeter-trusted-ca-bundle\") pod \"telemeter-client-57b7d86db6-pbz6t\" (UID: \"0eddddab-ac2f-4633-bb6a-01a896804307\") " pod="openshift-monitoring/telemeter-client-57b7d86db6-pbz6t"
Sep 18 19:41:06.377326 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.377154    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0eddddab-ac2f-4633-bb6a-01a896804307-secret-telemeter-client\") pod \"telemeter-client-57b7d86db6-pbz6t\" (UID: \"0eddddab-ac2f-4633-bb6a-01a896804307\") " pod="openshift-monitoring/telemeter-client-57b7d86db6-pbz6t"
Sep 18 19:41:06.377326 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.377253    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0eddddab-ac2f-4633-bb6a-01a896804307-federate-client-tls\") pod \"telemeter-client-57b7d86db6-pbz6t\" (UID: \"0eddddab-ac2f-4633-bb6a-01a896804307\") " pod="openshift-monitoring/telemeter-client-57b7d86db6-pbz6t"
Sep 18 19:41:06.377326 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.377314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0eddddab-ac2f-4633-bb6a-01a896804307-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-57b7d86db6-pbz6t\" (UID: \"0eddddab-ac2f-4633-bb6a-01a896804307\") " pod="openshift-monitoring/telemeter-client-57b7d86db6-pbz6t"
Sep 18 19:41:06.377488 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.377386    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0eddddab-ac2f-4633-bb6a-01a896804307-telemeter-client-tls\") pod \"telemeter-client-57b7d86db6-pbz6t\" (UID: \"0eddddab-ac2f-4633-bb6a-01a896804307\") " pod="openshift-monitoring/telemeter-client-57b7d86db6-pbz6t"
Sep 18 19:41:06.377488 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.377463    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0eddddab-ac2f-4633-bb6a-01a896804307-metrics-client-ca\") pod \"telemeter-client-57b7d86db6-pbz6t\" (UID: \"0eddddab-ac2f-4633-bb6a-01a896804307\") " pod="openshift-monitoring/telemeter-client-57b7d86db6-pbz6t"
Sep 18 19:41:06.377589 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.377515    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xm97\" (UniqueName: \"kubernetes.io/projected/0eddddab-ac2f-4633-bb6a-01a896804307-kube-api-access-7xm97\") pod \"telemeter-client-57b7d86db6-pbz6t\" (UID: \"0eddddab-ac2f-4633-bb6a-01a896804307\") " pod="openshift-monitoring/telemeter-client-57b7d86db6-pbz6t"
Sep 18 19:41:06.377953 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.377925    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eddddab-ac2f-4633-bb6a-01a896804307-serving-certs-ca-bundle\") pod \"telemeter-client-57b7d86db6-pbz6t\" (UID: \"0eddddab-ac2f-4633-bb6a-01a896804307\") " pod="openshift-monitoring/telemeter-client-57b7d86db6-pbz6t"
Sep 18 19:41:06.378120 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.378080    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eddddab-ac2f-4633-bb6a-01a896804307-telemeter-trusted-ca-bundle\") pod \"telemeter-client-57b7d86db6-pbz6t\" (UID: \"0eddddab-ac2f-4633-bb6a-01a896804307\") " pod="openshift-monitoring/telemeter-client-57b7d86db6-pbz6t"
Sep 18 19:41:06.378316 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.378295    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0eddddab-ac2f-4633-bb6a-01a896804307-metrics-client-ca\") pod \"telemeter-client-57b7d86db6-pbz6t\" (UID: \"0eddddab-ac2f-4633-bb6a-01a896804307\") " pod="openshift-monitoring/telemeter-client-57b7d86db6-pbz6t"
Sep 18 19:41:06.379932 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.379910    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0eddddab-ac2f-4633-bb6a-01a896804307-secret-telemeter-client\") pod \"telemeter-client-57b7d86db6-pbz6t\" (UID: \"0eddddab-ac2f-4633-bb6a-01a896804307\") " pod="openshift-monitoring/telemeter-client-57b7d86db6-pbz6t"
Sep 18 19:41:06.380069 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.380048    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0eddddab-ac2f-4633-bb6a-01a896804307-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-57b7d86db6-pbz6t\" (UID: \"0eddddab-ac2f-4633-bb6a-01a896804307\") " pod="openshift-monitoring/telemeter-client-57b7d86db6-pbz6t"
Sep 18 19:41:06.380128 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.380096    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0eddddab-ac2f-4633-bb6a-01a896804307-telemeter-client-tls\") pod \"telemeter-client-57b7d86db6-pbz6t\" (UID: \"0eddddab-ac2f-4633-bb6a-01a896804307\") " pod="openshift-monitoring/telemeter-client-57b7d86db6-pbz6t"
Sep 18 19:41:06.380792 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.380771    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0eddddab-ac2f-4633-bb6a-01a896804307-federate-client-tls\") pod \"telemeter-client-57b7d86db6-pbz6t\" (UID: \"0eddddab-ac2f-4633-bb6a-01a896804307\") " pod="openshift-monitoring/telemeter-client-57b7d86db6-pbz6t"
Sep 18 19:41:06.386257 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.386241    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xm97\" (UniqueName: \"kubernetes.io/projected/0eddddab-ac2f-4633-bb6a-01a896804307-kube-api-access-7xm97\") pod \"telemeter-client-57b7d86db6-pbz6t\" (UID: \"0eddddab-ac2f-4633-bb6a-01a896804307\") " pod="openshift-monitoring/telemeter-client-57b7d86db6-pbz6t"
Sep 18 19:41:06.500027 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.499949    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-57b7d86db6-pbz6t"
Sep 18 19:41:06.777274 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:06.777237    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-57b7d86db6-pbz6t"]
Sep 18 19:41:06.780466 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:41:06.780438    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eddddab_ac2f_4633_bb6a_01a896804307.slice/crio-80fbab5cfd4808c463b15d6bfbf0efee8a504a46c2edda7cb3932db7e8b692ed WatchSource:0}: Error finding container 80fbab5cfd4808c463b15d6bfbf0efee8a504a46c2edda7cb3932db7e8b692ed: Status 404 returned error can't find the container with id 80fbab5cfd4808c463b15d6bfbf0efee8a504a46c2edda7cb3932db7e8b692ed
Sep 18 19:41:07.318280 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:07.318247    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-57b7d86db6-pbz6t" event={"ID":"0eddddab-ac2f-4633-bb6a-01a896804307","Type":"ContainerStarted","Data":"80fbab5cfd4808c463b15d6bfbf0efee8a504a46c2edda7cb3932db7e8b692ed"}
Sep 18 19:41:08.326047 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:08.325969    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87","Type":"ContainerStarted","Data":"5bddde8bb193c247afb7b7f8b3b823312fd735d99013d6d4ce607fe2486a682c"}
Sep 18 19:41:08.326402 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:08.326052    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87","Type":"ContainerStarted","Data":"ed7bfde31d861844c393091cc9103a1dbeaa03d91a7e05536e1c13442c9b94ae"}
Sep 18 19:41:08.326402 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:08.326094    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87","Type":"ContainerStarted","Data":"dd8003c7f67165cd80cd8d83c5c35f3a4551d15ad9b18a19ceb7ead598ac122b"}
Sep 18 19:41:09.332349 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.332313    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-57b7d86db6-pbz6t" event={"ID":"0eddddab-ac2f-4633-bb6a-01a896804307","Type":"ContainerStarted","Data":"6402d45bcd59b3ac7e6caff9b9b7317ac5c116e61a48b1042dcc08e8acedff03"}
Sep 18 19:41:09.332760 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.332387    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-57b7d86db6-pbz6t" event={"ID":"0eddddab-ac2f-4633-bb6a-01a896804307","Type":"ContainerStarted","Data":"2ce64e39d2df13c5bb049dd97f587f8b8383300a941abd47911367917e43cdfd"}
Sep 18 19:41:09.332760 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.332419    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-57b7d86db6-pbz6t" event={"ID":"0eddddab-ac2f-4633-bb6a-01a896804307","Type":"ContainerStarted","Data":"e5a1fe3ab89022dce6728464549e0314610a3b243f8c65c9ffc3789281883d8b"}
Sep 18 19:41:09.335578 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.335556    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87","Type":"ContainerStarted","Data":"e16d01157f8f13ceab20e560ca83a819a7110b61d88c8d8c36456ffb78eb0bad"}
Sep 18 19:41:09.335689 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.335605    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87","Type":"ContainerStarted","Data":"6b9e96f9d8feba8e5e25c9574ef0bf8d37c607e8db082793ff428727c4e47047"}
Sep 18 19:41:09.335689 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.335630    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ee7a81a5-d588-4320-8b6f-5dbb28fa5b87","Type":"ContainerStarted","Data":"e250ed3a713e5aec8da3fccfa885fe7d4a9f7caa2f68c569d13328b6037caa88"}
Sep 18 19:41:09.357863 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.357793    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-57b7d86db6-pbz6t" podStartSLOduration=1.40094236 podStartE2EDuration="3.357776185s" totalImagesPullingTime="1.956833825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:41:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:41:09.35685538 +0000 UTC m=+192.459143703" watchObservedRunningTime="2026-09-18 19:41:09.357776185 +0000 UTC m=+192.460064512"
Sep 18 19:41:09.386004 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.385949    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.570629904 podStartE2EDuration="6.385930704s" totalImagesPullingTime="2.796043803s" totalInitContainerRuntime="19.256997ms" isStatefulPod=false podCreationTimestamp="2026-09-18 19:41:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:41:09.384148425 +0000 UTC m=+192.486436754" watchObservedRunningTime="2026-09-18 19:41:09.385930704 +0000 UTC m=+192.488219065"
Sep 18 19:41:09.432182 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.432152    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cdb69b5b8-x4jsz"]
Sep 18 19:41:09.435479 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.435460    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cdb69b5b8-x4jsz"
Sep 18 19:41:09.444807 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.444768    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cdb69b5b8-x4jsz"]
Sep 18 19:41:09.499926 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.499899    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98d92adf-99ee-49aa-b81a-0c1432451daf-console-serving-cert\") pod \"console-7cdb69b5b8-x4jsz\" (UID: \"98d92adf-99ee-49aa-b81a-0c1432451daf\") " pod="openshift-console/console-7cdb69b5b8-x4jsz"
Sep 18 19:41:09.500071 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.499934    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gkfz\" (UniqueName: \"kubernetes.io/projected/98d92adf-99ee-49aa-b81a-0c1432451daf-kube-api-access-4gkfz\") pod \"console-7cdb69b5b8-x4jsz\" (UID: \"98d92adf-99ee-49aa-b81a-0c1432451daf\") " pod="openshift-console/console-7cdb69b5b8-x4jsz"
Sep 18 19:41:09.500071 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.499962    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98d92adf-99ee-49aa-b81a-0c1432451daf-trusted-ca-bundle\") pod \"console-7cdb69b5b8-x4jsz\" (UID: \"98d92adf-99ee-49aa-b81a-0c1432451daf\") " pod="openshift-console/console-7cdb69b5b8-x4jsz"
Sep 18 19:41:09.500071 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.500034    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98d92adf-99ee-49aa-b81a-0c1432451daf-console-config\") pod \"console-7cdb69b5b8-x4jsz\" (UID: \"98d92adf-99ee-49aa-b81a-0c1432451daf\") " pod="openshift-console/console-7cdb69b5b8-x4jsz"
Sep 18 19:41:09.500174 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.500070    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98d92adf-99ee-49aa-b81a-0c1432451daf-oauth-serving-cert\") pod \"console-7cdb69b5b8-x4jsz\" (UID: \"98d92adf-99ee-49aa-b81a-0c1432451daf\") " pod="openshift-console/console-7cdb69b5b8-x4jsz"
Sep 18 19:41:09.500174 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.500105    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98d92adf-99ee-49aa-b81a-0c1432451daf-service-ca\") pod \"console-7cdb69b5b8-x4jsz\" (UID: \"98d92adf-99ee-49aa-b81a-0c1432451daf\") " pod="openshift-console/console-7cdb69b5b8-x4jsz"
Sep 18 19:41:09.500174 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.500124    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98d92adf-99ee-49aa-b81a-0c1432451daf-console-oauth-config\") pod \"console-7cdb69b5b8-x4jsz\" (UID: \"98d92adf-99ee-49aa-b81a-0c1432451daf\") " pod="openshift-console/console-7cdb69b5b8-x4jsz"
Sep 18 19:41:09.601354 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.601256    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98d92adf-99ee-49aa-b81a-0c1432451daf-console-config\") pod \"console-7cdb69b5b8-x4jsz\" (UID: \"98d92adf-99ee-49aa-b81a-0c1432451daf\") " pod="openshift-console/console-7cdb69b5b8-x4jsz"
Sep 18 19:41:09.601354 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.601307    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98d92adf-99ee-49aa-b81a-0c1432451daf-oauth-serving-cert\") pod \"console-7cdb69b5b8-x4jsz\" (UID: \"98d92adf-99ee-49aa-b81a-0c1432451daf\") " pod="openshift-console/console-7cdb69b5b8-x4jsz"
Sep 18 19:41:09.601354 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.601345    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98d92adf-99ee-49aa-b81a-0c1432451daf-service-ca\") pod \"console-7cdb69b5b8-x4jsz\" (UID: \"98d92adf-99ee-49aa-b81a-0c1432451daf\") " pod="openshift-console/console-7cdb69b5b8-x4jsz"
Sep 18 19:41:09.601625 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.601368    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98d92adf-99ee-49aa-b81a-0c1432451daf-console-oauth-config\") pod \"console-7cdb69b5b8-x4jsz\" (UID: \"98d92adf-99ee-49aa-b81a-0c1432451daf\") " pod="openshift-console/console-7cdb69b5b8-x4jsz"
Sep 18 19:41:09.601625 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.601405    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98d92adf-99ee-49aa-b81a-0c1432451daf-console-serving-cert\") pod \"console-7cdb69b5b8-x4jsz\" (UID: \"98d92adf-99ee-49aa-b81a-0c1432451daf\") " pod="openshift-console/console-7cdb69b5b8-x4jsz"
Sep 18 19:41:09.601625 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.601440    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gkfz\" (UniqueName: \"kubernetes.io/projected/98d92adf-99ee-49aa-b81a-0c1432451daf-kube-api-access-4gkfz\") pod \"console-7cdb69b5b8-x4jsz\" (UID: \"98d92adf-99ee-49aa-b81a-0c1432451daf\") " pod="openshift-console/console-7cdb69b5b8-x4jsz"
Sep 18 19:41:09.601625 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.601479    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98d92adf-99ee-49aa-b81a-0c1432451daf-trusted-ca-bundle\") pod \"console-7cdb69b5b8-x4jsz\" (UID: \"98d92adf-99ee-49aa-b81a-0c1432451daf\") " pod="openshift-console/console-7cdb69b5b8-x4jsz"
Sep 18 19:41:09.602122 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.602095    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98d92adf-99ee-49aa-b81a-0c1432451daf-console-config\") pod \"console-7cdb69b5b8-x4jsz\" (UID: \"98d92adf-99ee-49aa-b81a-0c1432451daf\") " pod="openshift-console/console-7cdb69b5b8-x4jsz"
Sep 18 19:41:09.602242 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.602160    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98d92adf-99ee-49aa-b81a-0c1432451daf-oauth-serving-cert\") pod \"console-7cdb69b5b8-x4jsz\" (UID: \"98d92adf-99ee-49aa-b81a-0c1432451daf\") " pod="openshift-console/console-7cdb69b5b8-x4jsz"
Sep 18 19:41:09.602326 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.602305    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98d92adf-99ee-49aa-b81a-0c1432451daf-service-ca\") pod \"console-7cdb69b5b8-x4jsz\" (UID: \"98d92adf-99ee-49aa-b81a-0c1432451daf\") " pod="openshift-console/console-7cdb69b5b8-x4jsz"
Sep 18 19:41:09.602552 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.602537    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98d92adf-99ee-49aa-b81a-0c1432451daf-trusted-ca-bundle\") pod \"console-7cdb69b5b8-x4jsz\" (UID: \"98d92adf-99ee-49aa-b81a-0c1432451daf\") " pod="openshift-console/console-7cdb69b5b8-x4jsz"
Sep 18 19:41:09.604356 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.604340    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98d92adf-99ee-49aa-b81a-0c1432451daf-console-oauth-config\") pod \"console-7cdb69b5b8-x4jsz\" (UID: \"98d92adf-99ee-49aa-b81a-0c1432451daf\") " pod="openshift-console/console-7cdb69b5b8-x4jsz"
Sep 18 19:41:09.604476 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.604458    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98d92adf-99ee-49aa-b81a-0c1432451daf-console-serving-cert\") pod \"console-7cdb69b5b8-x4jsz\" (UID: \"98d92adf-99ee-49aa-b81a-0c1432451daf\") " pod="openshift-console/console-7cdb69b5b8-x4jsz"
Sep 18 19:41:09.611452 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.611430    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gkfz\" (UniqueName: \"kubernetes.io/projected/98d92adf-99ee-49aa-b81a-0c1432451daf-kube-api-access-4gkfz\") pod \"console-7cdb69b5b8-x4jsz\" (UID: \"98d92adf-99ee-49aa-b81a-0c1432451daf\") " pod="openshift-console/console-7cdb69b5b8-x4jsz"
Sep 18 19:41:09.745444 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:09.745403    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cdb69b5b8-x4jsz"
Sep 18 19:41:10.015339 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:10.015317    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cdb69b5b8-x4jsz"]
Sep 18 19:41:10.016973 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:41:10.016946    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98d92adf_99ee_49aa_b81a_0c1432451daf.slice/crio-5220c03774222e56c9eda9bd2a33b01aeb604eadf5ff986bb670fb1c36bb7b72 WatchSource:0}: Error finding container 5220c03774222e56c9eda9bd2a33b01aeb604eadf5ff986bb670fb1c36bb7b72: Status 404 returned error can't find the container with id 5220c03774222e56c9eda9bd2a33b01aeb604eadf5ff986bb670fb1c36bb7b72
Sep 18 19:41:10.344340 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:10.344304    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cdb69b5b8-x4jsz" event={"ID":"98d92adf-99ee-49aa-b81a-0c1432451daf","Type":"ContainerStarted","Data":"10384fa8352613ceeb101ca6347cde11c08e2d2139b80218837c63da3b5766da"}
Sep 18 19:41:10.344806 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:10.344356    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cdb69b5b8-x4jsz" event={"ID":"98d92adf-99ee-49aa-b81a-0c1432451daf","Type":"ContainerStarted","Data":"5220c03774222e56c9eda9bd2a33b01aeb604eadf5ff986bb670fb1c36bb7b72"}
Sep 18 19:41:10.360613 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:10.360572    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7cdb69b5b8-x4jsz" podStartSLOduration=1.360557866 podStartE2EDuration="1.360557866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:41:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:41:10.359695665 +0000 UTC m=+193.461984007" watchObservedRunningTime="2026-09-18 19:41:10.360557866 +0000 UTC m=+193.462846193"
Sep 18 19:41:19.746431 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:19.746387    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cdb69b5b8-x4jsz"
Sep 18 19:41:19.746431 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:19.746444    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7cdb69b5b8-x4jsz"
Sep 18 19:41:19.751429 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:19.751405    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7cdb69b5b8-x4jsz"
Sep 18 19:41:20.384741 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:20.384714    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7cdb69b5b8-x4jsz"
Sep 18 19:41:20.431905 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:20.431866    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b797584fc-mzjpt"]
Sep 18 19:41:45.452137 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:45.452080    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b797584fc-mzjpt" podUID="cd27da47-cec6-4b54-9115-9df09bf39017" containerName="console" containerID="cri-o://cf17820eed8b97467194cadab93e313bb07ef6bfcda2fbb1e0b30d568b4826e2" gracePeriod=15
Sep 18 19:41:45.682954 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:45.682920    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b797584fc-mzjpt"
Sep 18 19:41:45.805765 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:45.805740    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd27da47-cec6-4b54-9115-9df09bf39017-trusted-ca-bundle\") pod \"cd27da47-cec6-4b54-9115-9df09bf39017\" (UID: \"cd27da47-cec6-4b54-9115-9df09bf39017\") "
Sep 18 19:41:45.805966 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:45.805771    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cd27da47-cec6-4b54-9115-9df09bf39017-console-config\") pod \"cd27da47-cec6-4b54-9115-9df09bf39017\" (UID: \"cd27da47-cec6-4b54-9115-9df09bf39017\") "
Sep 18 19:41:45.805966 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:45.805804    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hp5k\" (UniqueName: \"kubernetes.io/projected/cd27da47-cec6-4b54-9115-9df09bf39017-kube-api-access-6hp5k\") pod \"cd27da47-cec6-4b54-9115-9df09bf39017\" (UID: \"cd27da47-cec6-4b54-9115-9df09bf39017\") "
Sep 18 19:41:45.805966 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:45.805956    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd27da47-cec6-4b54-9115-9df09bf39017-service-ca\") pod \"cd27da47-cec6-4b54-9115-9df09bf39017\" (UID: \"cd27da47-cec6-4b54-9115-9df09bf39017\") "
Sep 18 19:41:45.806129 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:45.806001    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd27da47-cec6-4b54-9115-9df09bf39017-oauth-serving-cert\") pod \"cd27da47-cec6-4b54-9115-9df09bf39017\" (UID: \"cd27da47-cec6-4b54-9115-9df09bf39017\") "
Sep 18 19:41:45.806129 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:45.806038    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd27da47-cec6-4b54-9115-9df09bf39017-console-serving-cert\") pod \"cd27da47-cec6-4b54-9115-9df09bf39017\" (UID: \"cd27da47-cec6-4b54-9115-9df09bf39017\") "
Sep 18 19:41:45.806129 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:45.806087    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd27da47-cec6-4b54-9115-9df09bf39017-console-oauth-config\") pod \"cd27da47-cec6-4b54-9115-9df09bf39017\" (UID: \"cd27da47-cec6-4b54-9115-9df09bf39017\") "
Sep 18 19:41:45.806321 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:45.806258    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd27da47-cec6-4b54-9115-9df09bf39017-console-config" (OuterVolumeSpecName: "console-config") pod "cd27da47-cec6-4b54-9115-9df09bf39017" (UID: "cd27da47-cec6-4b54-9115-9df09bf39017"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:41:45.806376 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:45.806311    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd27da47-cec6-4b54-9115-9df09bf39017-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "cd27da47-cec6-4b54-9115-9df09bf39017" (UID: "cd27da47-cec6-4b54-9115-9df09bf39017"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:41:45.806376 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:45.806356    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd27da47-cec6-4b54-9115-9df09bf39017-service-ca" (OuterVolumeSpecName: "service-ca") pod "cd27da47-cec6-4b54-9115-9df09bf39017" (UID: "cd27da47-cec6-4b54-9115-9df09bf39017"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:41:45.806572 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:45.806550    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd27da47-cec6-4b54-9115-9df09bf39017-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cd27da47-cec6-4b54-9115-9df09bf39017" (UID: "cd27da47-cec6-4b54-9115-9df09bf39017"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:41:45.808312 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:45.808287    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd27da47-cec6-4b54-9115-9df09bf39017-kube-api-access-6hp5k" (OuterVolumeSpecName: "kube-api-access-6hp5k") pod "cd27da47-cec6-4b54-9115-9df09bf39017" (UID: "cd27da47-cec6-4b54-9115-9df09bf39017"). InnerVolumeSpecName "kube-api-access-6hp5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:41:45.808312 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:45.808285    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd27da47-cec6-4b54-9115-9df09bf39017-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cd27da47-cec6-4b54-9115-9df09bf39017" (UID: "cd27da47-cec6-4b54-9115-9df09bf39017"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:41:45.808441 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:45.808357    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd27da47-cec6-4b54-9115-9df09bf39017-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cd27da47-cec6-4b54-9115-9df09bf39017" (UID: "cd27da47-cec6-4b54-9115-9df09bf39017"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:41:45.906914 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:45.906877    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hp5k\" (UniqueName: \"kubernetes.io/projected/cd27da47-cec6-4b54-9115-9df09bf39017-kube-api-access-6hp5k\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:41:45.906914 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:45.906912    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd27da47-cec6-4b54-9115-9df09bf39017-service-ca\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:41:45.906914 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:45.906921    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd27da47-cec6-4b54-9115-9df09bf39017-oauth-serving-cert\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:41:45.907090 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:45.906930    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd27da47-cec6-4b54-9115-9df09bf39017-console-serving-cert\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:41:45.907090 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:45.906938    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd27da47-cec6-4b54-9115-9df09bf39017-console-oauth-config\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:41:45.907090 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:45.906947    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd27da47-cec6-4b54-9115-9df09bf39017-trusted-ca-bundle\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:41:45.907090 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:45.906955    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cd27da47-cec6-4b54-9115-9df09bf39017-console-config\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:41:46.484644 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:46.484605    2581 generic.go:356] "Generic (PLEG): container finished" podID="cd27da47-cec6-4b54-9115-9df09bf39017" containerID="cf17820eed8b97467194cadab93e313bb07ef6bfcda2fbb1e0b30d568b4826e2" exitCode=0
Sep 18 19:41:46.485109 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:46.484670    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b797584fc-mzjpt"
Sep 18 19:41:46.485109 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:46.484684    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b797584fc-mzjpt" event={"ID":"cd27da47-cec6-4b54-9115-9df09bf39017","Type":"ContainerDied","Data":"cf17820eed8b97467194cadab93e313bb07ef6bfcda2fbb1e0b30d568b4826e2"}
Sep 18 19:41:46.485109 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:46.484736    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b797584fc-mzjpt" event={"ID":"cd27da47-cec6-4b54-9115-9df09bf39017","Type":"ContainerDied","Data":"391488ec472dc63b044e5e3923595701a71f881fee29d41c66e44d48a7686184"}
Sep 18 19:41:46.485109 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:46.484756    2581 scope.go:117] "RemoveContainer" containerID="cf17820eed8b97467194cadab93e313bb07ef6bfcda2fbb1e0b30d568b4826e2"
Sep 18 19:41:46.493517 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:46.493503    2581 scope.go:117] "RemoveContainer" containerID="cf17820eed8b97467194cadab93e313bb07ef6bfcda2fbb1e0b30d568b4826e2"
Sep 18 19:41:46.493762 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:41:46.493747    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf17820eed8b97467194cadab93e313bb07ef6bfcda2fbb1e0b30d568b4826e2\": container with ID starting with cf17820eed8b97467194cadab93e313bb07ef6bfcda2fbb1e0b30d568b4826e2 not found: ID does not exist" containerID="cf17820eed8b97467194cadab93e313bb07ef6bfcda2fbb1e0b30d568b4826e2"
Sep 18 19:41:46.493799 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:46.493769    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf17820eed8b97467194cadab93e313bb07ef6bfcda2fbb1e0b30d568b4826e2"} err="failed to get container status \"cf17820eed8b97467194cadab93e313bb07ef6bfcda2fbb1e0b30d568b4826e2\": rpc error: code = NotFound desc = could not find container \"cf17820eed8b97467194cadab93e313bb07ef6bfcda2fbb1e0b30d568b4826e2\": container with ID starting with cf17820eed8b97467194cadab93e313bb07ef6bfcda2fbb1e0b30d568b4826e2 not found: ID does not exist"
Sep 18 19:41:46.504417 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:46.504390    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b797584fc-mzjpt"]
Sep 18 19:41:46.508062 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:46.508042    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b797584fc-mzjpt"]
Sep 18 19:41:47.432181 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:41:47.432140    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd27da47-cec6-4b54-9115-9df09bf39017" path="/var/lib/kubelet/pods/cd27da47-cec6-4b54-9115-9df09bf39017/volumes"
Sep 18 19:42:57.329952 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:42:57.329923    2581 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 19:43:02.470985 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:02.470946    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cdb69b5b8-x4jsz"]
Sep 18 19:43:27.491521 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:27.491465    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7cdb69b5b8-x4jsz" podUID="98d92adf-99ee-49aa-b81a-0c1432451daf" containerName="console" containerID="cri-o://10384fa8352613ceeb101ca6347cde11c08e2d2139b80218837c63da3b5766da" gracePeriod=15
Sep 18 19:43:27.732151 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:27.732128    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cdb69b5b8-x4jsz"
Sep 18 19:43:27.798194 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:27.798171    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98d92adf-99ee-49aa-b81a-0c1432451daf-oauth-serving-cert\") pod \"98d92adf-99ee-49aa-b81a-0c1432451daf\" (UID: \"98d92adf-99ee-49aa-b81a-0c1432451daf\") "
Sep 18 19:43:27.798308 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:27.798200    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98d92adf-99ee-49aa-b81a-0c1432451daf-service-ca\") pod \"98d92adf-99ee-49aa-b81a-0c1432451daf\" (UID: \"98d92adf-99ee-49aa-b81a-0c1432451daf\") "
Sep 18 19:43:27.798308 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:27.798232    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98d92adf-99ee-49aa-b81a-0c1432451daf-console-serving-cert\") pod \"98d92adf-99ee-49aa-b81a-0c1432451daf\" (UID: \"98d92adf-99ee-49aa-b81a-0c1432451daf\") "
Sep 18 19:43:27.798391 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:27.798368    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98d92adf-99ee-49aa-b81a-0c1432451daf-console-config\") pod \"98d92adf-99ee-49aa-b81a-0c1432451daf\" (UID: \"98d92adf-99ee-49aa-b81a-0c1432451daf\") "
Sep 18 19:43:27.798436 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:27.798412    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98d92adf-99ee-49aa-b81a-0c1432451daf-console-oauth-config\") pod \"98d92adf-99ee-49aa-b81a-0c1432451daf\" (UID: \"98d92adf-99ee-49aa-b81a-0c1432451daf\") "
Sep 18 19:43:27.798488 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:27.798465    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98d92adf-99ee-49aa-b81a-0c1432451daf-trusted-ca-bundle\") pod \"98d92adf-99ee-49aa-b81a-0c1432451daf\" (UID: \"98d92adf-99ee-49aa-b81a-0c1432451daf\") "
Sep 18 19:43:27.798540 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:27.798488    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gkfz\" (UniqueName: \"kubernetes.io/projected/98d92adf-99ee-49aa-b81a-0c1432451daf-kube-api-access-4gkfz\") pod \"98d92adf-99ee-49aa-b81a-0c1432451daf\" (UID: \"98d92adf-99ee-49aa-b81a-0c1432451daf\") "
Sep 18 19:43:27.798593 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:27.798540    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98d92adf-99ee-49aa-b81a-0c1432451daf-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "98d92adf-99ee-49aa-b81a-0c1432451daf" (UID: "98d92adf-99ee-49aa-b81a-0c1432451daf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:43:27.798773 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:27.798735    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98d92adf-99ee-49aa-b81a-0c1432451daf-oauth-serving-cert\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:43:27.798773 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:27.798738    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98d92adf-99ee-49aa-b81a-0c1432451daf-service-ca" (OuterVolumeSpecName: "service-ca") pod "98d92adf-99ee-49aa-b81a-0c1432451daf" (UID: "98d92adf-99ee-49aa-b81a-0c1432451daf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:43:27.798773 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:27.798749    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98d92adf-99ee-49aa-b81a-0c1432451daf-console-config" (OuterVolumeSpecName: "console-config") pod "98d92adf-99ee-49aa-b81a-0c1432451daf" (UID: "98d92adf-99ee-49aa-b81a-0c1432451daf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:43:27.799079 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:27.798962    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98d92adf-99ee-49aa-b81a-0c1432451daf-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "98d92adf-99ee-49aa-b81a-0c1432451daf" (UID: "98d92adf-99ee-49aa-b81a-0c1432451daf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:43:27.800475 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:27.800454    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98d92adf-99ee-49aa-b81a-0c1432451daf-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "98d92adf-99ee-49aa-b81a-0c1432451daf" (UID: "98d92adf-99ee-49aa-b81a-0c1432451daf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:43:27.800582 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:27.800564    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98d92adf-99ee-49aa-b81a-0c1432451daf-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "98d92adf-99ee-49aa-b81a-0c1432451daf" (UID: "98d92adf-99ee-49aa-b81a-0c1432451daf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:43:27.800642 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:27.800580    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98d92adf-99ee-49aa-b81a-0c1432451daf-kube-api-access-4gkfz" (OuterVolumeSpecName: "kube-api-access-4gkfz") pod "98d92adf-99ee-49aa-b81a-0c1432451daf" (UID: "98d92adf-99ee-49aa-b81a-0c1432451daf"). InnerVolumeSpecName "kube-api-access-4gkfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:43:27.881380 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:27.881349    2581 generic.go:356] "Generic (PLEG): container finished" podID="98d92adf-99ee-49aa-b81a-0c1432451daf" containerID="10384fa8352613ceeb101ca6347cde11c08e2d2139b80218837c63da3b5766da" exitCode=0
Sep 18 19:43:27.881495 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:27.881395    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cdb69b5b8-x4jsz"
Sep 18 19:43:27.881495 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:27.881409    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cdb69b5b8-x4jsz" event={"ID":"98d92adf-99ee-49aa-b81a-0c1432451daf","Type":"ContainerDied","Data":"10384fa8352613ceeb101ca6347cde11c08e2d2139b80218837c63da3b5766da"}
Sep 18 19:43:27.881495 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:27.881446    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cdb69b5b8-x4jsz" event={"ID":"98d92adf-99ee-49aa-b81a-0c1432451daf","Type":"ContainerDied","Data":"5220c03774222e56c9eda9bd2a33b01aeb604eadf5ff986bb670fb1c36bb7b72"}
Sep 18 19:43:27.881495 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:27.881464    2581 scope.go:117] "RemoveContainer" containerID="10384fa8352613ceeb101ca6347cde11c08e2d2139b80218837c63da3b5766da"
Sep 18 19:43:27.890391 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:27.890374    2581 scope.go:117] "RemoveContainer" containerID="10384fa8352613ceeb101ca6347cde11c08e2d2139b80218837c63da3b5766da"
Sep 18 19:43:27.890654 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:43:27.890639    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10384fa8352613ceeb101ca6347cde11c08e2d2139b80218837c63da3b5766da\": container with ID starting with 10384fa8352613ceeb101ca6347cde11c08e2d2139b80218837c63da3b5766da not found: ID does not exist" containerID="10384fa8352613ceeb101ca6347cde11c08e2d2139b80218837c63da3b5766da"
Sep 18 19:43:27.890733 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:27.890662    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10384fa8352613ceeb101ca6347cde11c08e2d2139b80218837c63da3b5766da"} err="failed to get container status \"10384fa8352613ceeb101ca6347cde11c08e2d2139b80218837c63da3b5766da\": rpc error: code = NotFound desc = could not find container \"10384fa8352613ceeb101ca6347cde11c08e2d2139b80218837c63da3b5766da\": container with ID starting with 10384fa8352613ceeb101ca6347cde11c08e2d2139b80218837c63da3b5766da not found: ID does not exist"
Sep 18 19:43:27.900059 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:27.900041    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98d92adf-99ee-49aa-b81a-0c1432451daf-trusted-ca-bundle\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:43:27.900112 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:27.900061    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gkfz\" (UniqueName: \"kubernetes.io/projected/98d92adf-99ee-49aa-b81a-0c1432451daf-kube-api-access-4gkfz\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:43:27.900112 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:27.900070    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98d92adf-99ee-49aa-b81a-0c1432451daf-service-ca\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:43:27.900112 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:27.900078    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98d92adf-99ee-49aa-b81a-0c1432451daf-console-serving-cert\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:43:27.900112 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:27.900087    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98d92adf-99ee-49aa-b81a-0c1432451daf-console-config\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:43:27.900112 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:27.900095    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98d92adf-99ee-49aa-b81a-0c1432451daf-console-oauth-config\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:43:27.904264 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:27.904245    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cdb69b5b8-x4jsz"]
Sep 18 19:43:27.911859 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:27.911824    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7cdb69b5b8-x4jsz"]
Sep 18 19:43:28.562073 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:28.562039    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx"]
Sep 18 19:43:28.562485 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:28.562339    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd27da47-cec6-4b54-9115-9df09bf39017" containerName="console"
Sep 18 19:43:28.562485 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:28.562349    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd27da47-cec6-4b54-9115-9df09bf39017" containerName="console"
Sep 18 19:43:28.562485 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:28.562359    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98d92adf-99ee-49aa-b81a-0c1432451daf" containerName="console"
Sep 18 19:43:28.562485 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:28.562364    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="98d92adf-99ee-49aa-b81a-0c1432451daf" containerName="console"
Sep 18 19:43:28.562485 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:28.562415    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="98d92adf-99ee-49aa-b81a-0c1432451daf" containerName="console"
Sep 18 19:43:28.562485 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:28.562425    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd27da47-cec6-4b54-9115-9df09bf39017" containerName="console"
Sep 18 19:43:28.566805 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:28.566789    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx"
Sep 18 19:43:28.569354 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:28.569331    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 19:43:28.569484 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:28.569393    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 19:43:28.570236 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:28.570220    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lg99k\""
Sep 18 19:43:28.575081 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:28.575061    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx"]
Sep 18 19:43:28.604615 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:28.604594    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7ba4810-09a8-498e-9f7c-60788b5fcacb-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx\" (UID: \"e7ba4810-09a8-498e-9f7c-60788b5fcacb\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx"
Sep 18 19:43:28.604707 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:28.604640    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tjhs\" (UniqueName: \"kubernetes.io/projected/e7ba4810-09a8-498e-9f7c-60788b5fcacb-kube-api-access-8tjhs\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx\" (UID: \"e7ba4810-09a8-498e-9f7c-60788b5fcacb\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx"
Sep 18 19:43:28.604707 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:28.604694    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7ba4810-09a8-498e-9f7c-60788b5fcacb-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx\" (UID: \"e7ba4810-09a8-498e-9f7c-60788b5fcacb\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx"
Sep 18 19:43:28.704982 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:28.704957    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7ba4810-09a8-498e-9f7c-60788b5fcacb-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx\" (UID: \"e7ba4810-09a8-498e-9f7c-60788b5fcacb\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx"
Sep 18 19:43:28.705090 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:28.705005    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7ba4810-09a8-498e-9f7c-60788b5fcacb-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx\" (UID: \"e7ba4810-09a8-498e-9f7c-60788b5fcacb\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx"
Sep 18 19:43:28.705090 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:28.705063    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tjhs\" (UniqueName: \"kubernetes.io/projected/e7ba4810-09a8-498e-9f7c-60788b5fcacb-kube-api-access-8tjhs\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx\" (UID: \"e7ba4810-09a8-498e-9f7c-60788b5fcacb\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx"
Sep 18 19:43:28.705290 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:28.705270    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7ba4810-09a8-498e-9f7c-60788b5fcacb-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx\" (UID: \"e7ba4810-09a8-498e-9f7c-60788b5fcacb\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx"
Sep 18 19:43:28.705357 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:28.705319    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7ba4810-09a8-498e-9f7c-60788b5fcacb-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx\" (UID: \"e7ba4810-09a8-498e-9f7c-60788b5fcacb\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx"
Sep 18 19:43:28.712505 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:28.712486    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tjhs\" (UniqueName: \"kubernetes.io/projected/e7ba4810-09a8-498e-9f7c-60788b5fcacb-kube-api-access-8tjhs\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx\" (UID: \"e7ba4810-09a8-498e-9f7c-60788b5fcacb\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx"
Sep 18 19:43:28.876688 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:28.876625    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx"
Sep 18 19:43:29.145371 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:29.145349    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx"]
Sep 18 19:43:29.147807 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:43:29.147778    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7ba4810_09a8_498e_9f7c_60788b5fcacb.slice/crio-14ca45ff9458849a202a147fe4807d79ee8d0fe8317df2df83e4c17b0070c998 WatchSource:0}: Error finding container 14ca45ff9458849a202a147fe4807d79ee8d0fe8317df2df83e4c17b0070c998: Status 404 returned error can't find the container with id 14ca45ff9458849a202a147fe4807d79ee8d0fe8317df2df83e4c17b0070c998
Sep 18 19:43:29.149767 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:29.149752    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:43:29.432321 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:29.432239    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98d92adf-99ee-49aa-b81a-0c1432451daf" path="/var/lib/kubelet/pods/98d92adf-99ee-49aa-b81a-0c1432451daf/volumes"
Sep 18 19:43:29.894579 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:29.894551    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx" event={"ID":"e7ba4810-09a8-498e-9f7c-60788b5fcacb","Type":"ContainerStarted","Data":"14ca45ff9458849a202a147fe4807d79ee8d0fe8317df2df83e4c17b0070c998"}
Sep 18 19:43:34.919387 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:34.919300    2581 generic.go:356] "Generic (PLEG): container finished" podID="e7ba4810-09a8-498e-9f7c-60788b5fcacb" containerID="82bb114af7e03df2bffbe492433c2c059df4d9c1aa6ce0b9cc57ab37c00f53cc" exitCode=0
Sep 18 19:43:34.919864 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:34.919391    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx" event={"ID":"e7ba4810-09a8-498e-9f7c-60788b5fcacb","Type":"ContainerDied","Data":"82bb114af7e03df2bffbe492433c2c059df4d9c1aa6ce0b9cc57ab37c00f53cc"}
Sep 18 19:43:36.931295 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:36.931268    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx" event={"ID":"e7ba4810-09a8-498e-9f7c-60788b5fcacb","Type":"ContainerStarted","Data":"603a200df4924d92cab26010b856808be4e5286aa5f57e54908abe326a1c0c77"}
Sep 18 19:43:37.937390 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:37.937353    2581 generic.go:356] "Generic (PLEG): container finished" podID="e7ba4810-09a8-498e-9f7c-60788b5fcacb" containerID="603a200df4924d92cab26010b856808be4e5286aa5f57e54908abe326a1c0c77" exitCode=0
Sep 18 19:43:37.937771 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:37.937435    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx" event={"ID":"e7ba4810-09a8-498e-9f7c-60788b5fcacb","Type":"ContainerDied","Data":"603a200df4924d92cab26010b856808be4e5286aa5f57e54908abe326a1c0c77"}
Sep 18 19:43:43.968301 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:43.968262    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx" event={"ID":"e7ba4810-09a8-498e-9f7c-60788b5fcacb","Type":"ContainerStarted","Data":"e8eb82ff534082c3fb89a85581a9074a4a7ddbe1b8cc096683962a13c925584a"}
Sep 18 19:43:43.987609 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:43.987561    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx" podStartSLOduration=2.818510121 podStartE2EDuration="15.987545103s" totalImagesPullingTime="13.169034982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:43:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:43:43.985976748 +0000 UTC m=+347.088265090" watchObservedRunningTime="2026-09-18 19:43:43.987545103 +0000 UTC m=+347.089833434"
Sep 18 19:43:44.974760 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:44.974722    2581 generic.go:356] "Generic (PLEG): container finished" podID="e7ba4810-09a8-498e-9f7c-60788b5fcacb" containerID="e8eb82ff534082c3fb89a85581a9074a4a7ddbe1b8cc096683962a13c925584a" exitCode=0
Sep 18 19:43:44.975199 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:44.974828    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx" event={"ID":"e7ba4810-09a8-498e-9f7c-60788b5fcacb","Type":"ContainerDied","Data":"e8eb82ff534082c3fb89a85581a9074a4a7ddbe1b8cc096683962a13c925584a"}
Sep 18 19:43:46.105237 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:46.105207    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx"
Sep 18 19:43:46.259239 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:46.259162    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tjhs\" (UniqueName: \"kubernetes.io/projected/e7ba4810-09a8-498e-9f7c-60788b5fcacb-kube-api-access-8tjhs\") pod \"e7ba4810-09a8-498e-9f7c-60788b5fcacb\" (UID: \"e7ba4810-09a8-498e-9f7c-60788b5fcacb\") "
Sep 18 19:43:46.259239 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:46.259199    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7ba4810-09a8-498e-9f7c-60788b5fcacb-util\") pod \"e7ba4810-09a8-498e-9f7c-60788b5fcacb\" (UID: \"e7ba4810-09a8-498e-9f7c-60788b5fcacb\") "
Sep 18 19:43:46.259239 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:46.259235    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7ba4810-09a8-498e-9f7c-60788b5fcacb-bundle\") pod \"e7ba4810-09a8-498e-9f7c-60788b5fcacb\" (UID: \"e7ba4810-09a8-498e-9f7c-60788b5fcacb\") "
Sep 18 19:43:46.259894 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:46.259871    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7ba4810-09a8-498e-9f7c-60788b5fcacb-bundle" (OuterVolumeSpecName: "bundle") pod "e7ba4810-09a8-498e-9f7c-60788b5fcacb" (UID: "e7ba4810-09a8-498e-9f7c-60788b5fcacb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:43:46.261502 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:46.261473    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7ba4810-09a8-498e-9f7c-60788b5fcacb-kube-api-access-8tjhs" (OuterVolumeSpecName: "kube-api-access-8tjhs") pod "e7ba4810-09a8-498e-9f7c-60788b5fcacb" (UID: "e7ba4810-09a8-498e-9f7c-60788b5fcacb"). InnerVolumeSpecName "kube-api-access-8tjhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:43:46.263417 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:46.263386    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7ba4810-09a8-498e-9f7c-60788b5fcacb-util" (OuterVolumeSpecName: "util") pod "e7ba4810-09a8-498e-9f7c-60788b5fcacb" (UID: "e7ba4810-09a8-498e-9f7c-60788b5fcacb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:43:46.360290 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:46.360256    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e7ba4810-09a8-498e-9f7c-60788b5fcacb-bundle\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:43:46.360290 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:46.360284    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tjhs\" (UniqueName: \"kubernetes.io/projected/e7ba4810-09a8-498e-9f7c-60788b5fcacb-kube-api-access-8tjhs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:43:46.360290 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:46.360293    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e7ba4810-09a8-498e-9f7c-60788b5fcacb-util\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:43:46.984459 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:46.984419    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx" event={"ID":"e7ba4810-09a8-498e-9f7c-60788b5fcacb","Type":"ContainerDied","Data":"14ca45ff9458849a202a147fe4807d79ee8d0fe8317df2df83e4c17b0070c998"}
Sep 18 19:43:46.984674 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:46.984491    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apb9sx"
Sep 18 19:43:46.984674 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:46.984496    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14ca45ff9458849a202a147fe4807d79ee8d0fe8317df2df83e4c17b0070c998"
Sep 18 19:43:55.557035 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:55.556957    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-2gqt9"]
Sep 18 19:43:55.557494 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:55.557317    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7ba4810-09a8-498e-9f7c-60788b5fcacb" containerName="extract"
Sep 18 19:43:55.557494 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:55.557329    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7ba4810-09a8-498e-9f7c-60788b5fcacb" containerName="extract"
Sep 18 19:43:55.557494 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:55.557344    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7ba4810-09a8-498e-9f7c-60788b5fcacb" containerName="pull"
Sep 18 19:43:55.557494 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:55.557349    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7ba4810-09a8-498e-9f7c-60788b5fcacb" containerName="pull"
Sep 18 19:43:55.557494 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:55.557359    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7ba4810-09a8-498e-9f7c-60788b5fcacb" containerName="util"
Sep 18 19:43:55.557494 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:55.557364    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7ba4810-09a8-498e-9f7c-60788b5fcacb" containerName="util"
Sep 18 19:43:55.557494 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:55.557416    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7ba4810-09a8-498e-9f7c-60788b5fcacb" containerName="extract"
Sep 18 19:43:55.559492 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:55.559477    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-2gqt9"
Sep 18 19:43:55.567257 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:55.567235    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 18 19:43:55.567879 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:55.567863    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 18 19:43:55.567972 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:55.567862    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-74cfb\""
Sep 18 19:43:55.567972 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:55.567911    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 18 19:43:55.567972 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:55.567936    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 18 19:43:55.568106 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:55.567982    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 18 19:43:55.575675 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:55.575654    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-2gqt9"]
Sep 18 19:43:55.737924 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:55.737894    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/5877f802-d2e0-40e5-85fa-1fb3eff9d264-cabundle0\") pod \"keda-operator-579c8958f7-2gqt9\" (UID: \"5877f802-d2e0-40e5-85fa-1fb3eff9d264\") " pod="openshift-keda/keda-operator-579c8958f7-2gqt9"
Sep 18 19:43:55.737924 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:55.737923    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsf88\" (UniqueName: \"kubernetes.io/projected/5877f802-d2e0-40e5-85fa-1fb3eff9d264-kube-api-access-jsf88\") pod \"keda-operator-579c8958f7-2gqt9\" (UID: \"5877f802-d2e0-40e5-85fa-1fb3eff9d264\") " pod="openshift-keda/keda-operator-579c8958f7-2gqt9"
Sep 18 19:43:55.738120 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:55.737966    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5877f802-d2e0-40e5-85fa-1fb3eff9d264-certificates\") pod \"keda-operator-579c8958f7-2gqt9\" (UID: \"5877f802-d2e0-40e5-85fa-1fb3eff9d264\") " pod="openshift-keda/keda-operator-579c8958f7-2gqt9"
Sep 18 19:43:55.839132 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:55.839051    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/5877f802-d2e0-40e5-85fa-1fb3eff9d264-cabundle0\") pod \"keda-operator-579c8958f7-2gqt9\" (UID: \"5877f802-d2e0-40e5-85fa-1fb3eff9d264\") " pod="openshift-keda/keda-operator-579c8958f7-2gqt9"
Sep 18 19:43:55.839132 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:55.839084    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsf88\" (UniqueName: \"kubernetes.io/projected/5877f802-d2e0-40e5-85fa-1fb3eff9d264-kube-api-access-jsf88\") pod \"keda-operator-579c8958f7-2gqt9\" (UID: \"5877f802-d2e0-40e5-85fa-1fb3eff9d264\") " pod="openshift-keda/keda-operator-579c8958f7-2gqt9"
Sep 18 19:43:55.839132 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:55.839120    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5877f802-d2e0-40e5-85fa-1fb3eff9d264-certificates\") pod \"keda-operator-579c8958f7-2gqt9\" (UID: \"5877f802-d2e0-40e5-85fa-1fb3eff9d264\") " pod="openshift-keda/keda-operator-579c8958f7-2gqt9"
Sep 18 19:43:55.839392 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:43:55.839211    2581 secret.go:281] references non-existent secret key: ca.crt
Sep 18 19:43:55.839392 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:43:55.839223    2581 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 18 19:43:55.839392 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:43:55.839230    2581 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-2gqt9: references non-existent secret key: ca.crt
Sep 18 19:43:55.839392 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:43:55.839285    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5877f802-d2e0-40e5-85fa-1fb3eff9d264-certificates podName:5877f802-d2e0-40e5-85fa-1fb3eff9d264 nodeName:}" failed. No retries permitted until 2026-09-18 19:43:56.339270121 +0000 UTC m=+359.441558427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/5877f802-d2e0-40e5-85fa-1fb3eff9d264-certificates") pod "keda-operator-579c8958f7-2gqt9" (UID: "5877f802-d2e0-40e5-85fa-1fb3eff9d264") : references non-existent secret key: ca.crt
Sep 18 19:43:55.839819 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:55.839798    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/5877f802-d2e0-40e5-85fa-1fb3eff9d264-cabundle0\") pod \"keda-operator-579c8958f7-2gqt9\" (UID: \"5877f802-d2e0-40e5-85fa-1fb3eff9d264\") " pod="openshift-keda/keda-operator-579c8958f7-2gqt9"
Sep 18 19:43:55.847761 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:55.847740    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsf88\" (UniqueName: \"kubernetes.io/projected/5877f802-d2e0-40e5-85fa-1fb3eff9d264-kube-api-access-jsf88\") pod \"keda-operator-579c8958f7-2gqt9\" (UID: \"5877f802-d2e0-40e5-85fa-1fb3eff9d264\") " pod="openshift-keda/keda-operator-579c8958f7-2gqt9"
Sep 18 19:43:56.344094 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:56.344062    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5877f802-d2e0-40e5-85fa-1fb3eff9d264-certificates\") pod \"keda-operator-579c8958f7-2gqt9\" (UID: \"5877f802-d2e0-40e5-85fa-1fb3eff9d264\") " pod="openshift-keda/keda-operator-579c8958f7-2gqt9"
Sep 18 19:43:56.344253 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:43:56.344214    2581 secret.go:281] references non-existent secret key: ca.crt
Sep 18 19:43:56.344253 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:43:56.344233    2581 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 18 19:43:56.344253 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:43:56.344241    2581 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-2gqt9: references non-existent secret key: ca.crt
Sep 18 19:43:56.344349 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:43:56.344303    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5877f802-d2e0-40e5-85fa-1fb3eff9d264-certificates podName:5877f802-d2e0-40e5-85fa-1fb3eff9d264 nodeName:}" failed. No retries permitted until 2026-09-18 19:43:57.344288171 +0000 UTC m=+360.446576476 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/5877f802-d2e0-40e5-85fa-1fb3eff9d264-certificates") pod "keda-operator-579c8958f7-2gqt9" (UID: "5877f802-d2e0-40e5-85fa-1fb3eff9d264") : references non-existent secret key: ca.crt
Sep 18 19:43:56.625238 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:56.625163    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-kt4tm"]
Sep 18 19:43:56.627344 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:56.627329    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-kt4tm"
Sep 18 19:43:56.629537 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:56.629515    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 18 19:43:56.638435 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:56.638417    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-kt4tm"]
Sep 18 19:43:56.748066 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:56.748034    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b8adc2ba-295b-4e13-a652-b600e3867ef9-certificates\") pod \"keda-admission-6c4d864b86-kt4tm\" (UID: \"b8adc2ba-295b-4e13-a652-b600e3867ef9\") " pod="openshift-keda/keda-admission-6c4d864b86-kt4tm"
Sep 18 19:43:56.748265 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:56.748151    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6ftz\" (UniqueName: \"kubernetes.io/projected/b8adc2ba-295b-4e13-a652-b600e3867ef9-kube-api-access-v6ftz\") pod \"keda-admission-6c4d864b86-kt4tm\" (UID: \"b8adc2ba-295b-4e13-a652-b600e3867ef9\") " pod="openshift-keda/keda-admission-6c4d864b86-kt4tm"
Sep 18 19:43:56.849041 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:56.849010    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6ftz\" (UniqueName: \"kubernetes.io/projected/b8adc2ba-295b-4e13-a652-b600e3867ef9-kube-api-access-v6ftz\") pod \"keda-admission-6c4d864b86-kt4tm\" (UID: \"b8adc2ba-295b-4e13-a652-b600e3867ef9\") " pod="openshift-keda/keda-admission-6c4d864b86-kt4tm"
Sep 18 19:43:56.849193 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:56.849056    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b8adc2ba-295b-4e13-a652-b600e3867ef9-certificates\") pod \"keda-admission-6c4d864b86-kt4tm\" (UID: \"b8adc2ba-295b-4e13-a652-b600e3867ef9\") " pod="openshift-keda/keda-admission-6c4d864b86-kt4tm"
Sep 18 19:43:56.851714 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:56.851683    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b8adc2ba-295b-4e13-a652-b600e3867ef9-certificates\") pod \"keda-admission-6c4d864b86-kt4tm\" (UID: \"b8adc2ba-295b-4e13-a652-b600e3867ef9\") " pod="openshift-keda/keda-admission-6c4d864b86-kt4tm"
Sep 18 19:43:56.860250 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:56.860230    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6ftz\" (UniqueName: \"kubernetes.io/projected/b8adc2ba-295b-4e13-a652-b600e3867ef9-kube-api-access-v6ftz\") pod \"keda-admission-6c4d864b86-kt4tm\" (UID: \"b8adc2ba-295b-4e13-a652-b600e3867ef9\") " pod="openshift-keda/keda-admission-6c4d864b86-kt4tm"
Sep 18 19:43:56.937208 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:56.937146    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-kt4tm"
Sep 18 19:43:57.209433 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:57.209330    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-kt4tm"]
Sep 18 19:43:57.212185 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:43:57.212157    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8adc2ba_295b_4e13_a652_b600e3867ef9.slice/crio-46d7702a14295890c7ddca52f1802decd38d1976e3c5769eb85ade20cf7487cd WatchSource:0}: Error finding container 46d7702a14295890c7ddca52f1802decd38d1976e3c5769eb85ade20cf7487cd: Status 404 returned error can't find the container with id 46d7702a14295890c7ddca52f1802decd38d1976e3c5769eb85ade20cf7487cd
Sep 18 19:43:57.352497 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:57.352477    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5877f802-d2e0-40e5-85fa-1fb3eff9d264-certificates\") pod \"keda-operator-579c8958f7-2gqt9\" (UID: \"5877f802-d2e0-40e5-85fa-1fb3eff9d264\") " pod="openshift-keda/keda-operator-579c8958f7-2gqt9"
Sep 18 19:43:57.354962 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:57.354935    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5877f802-d2e0-40e5-85fa-1fb3eff9d264-certificates\") pod \"keda-operator-579c8958f7-2gqt9\" (UID: \"5877f802-d2e0-40e5-85fa-1fb3eff9d264\") " pod="openshift-keda/keda-operator-579c8958f7-2gqt9"
Sep 18 19:43:57.369919 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:57.369899    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-2gqt9"
Sep 18 19:43:57.635573 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:57.635549    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-2gqt9"]
Sep 18 19:43:57.637881 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:43:57.637858    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5877f802_d2e0_40e5_85fa_1fb3eff9d264.slice/crio-3a41d5c1035138d521d8ccfabcf50b8319386e6c3924ef8b837c544bbdd78cf3 WatchSource:0}: Error finding container 3a41d5c1035138d521d8ccfabcf50b8319386e6c3924ef8b837c544bbdd78cf3: Status 404 returned error can't find the container with id 3a41d5c1035138d521d8ccfabcf50b8319386e6c3924ef8b837c544bbdd78cf3
Sep 18 19:43:58.029603 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:58.029562    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-2gqt9" event={"ID":"5877f802-d2e0-40e5-85fa-1fb3eff9d264","Type":"ContainerStarted","Data":"3a41d5c1035138d521d8ccfabcf50b8319386e6c3924ef8b837c544bbdd78cf3"}
Sep 18 19:43:58.030985 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:58.030949    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-kt4tm" event={"ID":"b8adc2ba-295b-4e13-a652-b600e3867ef9","Type":"ContainerStarted","Data":"46d7702a14295890c7ddca52f1802decd38d1976e3c5769eb85ade20cf7487cd"}
Sep 18 19:43:59.037367 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:59.037329    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-kt4tm" event={"ID":"b8adc2ba-295b-4e13-a652-b600e3867ef9","Type":"ContainerStarted","Data":"7205a54f5036c2c940d2a90ca3d470131feb7709efc0c9ca12071ff050507c7d"}
Sep 18 19:43:59.037870 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:59.037536    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-kt4tm"
Sep 18 19:43:59.055373 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:43:59.055328    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-kt4tm" podStartSLOduration=1.607148765 podStartE2EDuration="3.055314218s" totalImagesPullingTime="1.448165453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:43:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:43:59.053180342 +0000 UTC m=+362.155468671" watchObservedRunningTime="2026-09-18 19:43:59.055314218 +0000 UTC m=+362.157602545"
Sep 18 19:44:01.047584 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:01.047551    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-2gqt9" event={"ID":"5877f802-d2e0-40e5-85fa-1fb3eff9d264","Type":"ContainerStarted","Data":"59af47a649febf8c338a703545d22b4f94e2ef05c760d15bab913f3f418b5aed"}
Sep 18 19:44:01.048009 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:01.047687    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-2gqt9"
Sep 18 19:44:01.065457 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:01.065408    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-2gqt9" podStartSLOduration=3.246027462 podStartE2EDuration="6.065392617s" totalImagesPullingTime="2.819365155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:43:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:44:01.064194721 +0000 UTC m=+364.166483049" watchObservedRunningTime="2026-09-18 19:44:01.065392617 +0000 UTC m=+364.167680945"
Sep 18 19:44:20.044246 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:20.044216    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-kt4tm"
Sep 18 19:44:22.053960 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:22.053932    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-2gqt9"
Sep 18 19:44:48.378110 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:48.378073    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754066c2z"]
Sep 18 19:44:48.381995 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:48.381975    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754066c2z"
Sep 18 19:44:48.384277 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:48.384258    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 19:44:48.384334 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:48.384259    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 19:44:48.385273 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:48.385255    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lg99k\""
Sep 18 19:44:48.390661 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:48.390634    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754066c2z"]
Sep 18 19:44:48.545800 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:48.545762    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km4sg\" (UniqueName: \"kubernetes.io/projected/1da6cd04-3316-4a50-aa5b-29985654056a-kube-api-access-km4sg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754066c2z\" (UID: \"1da6cd04-3316-4a50-aa5b-29985654056a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754066c2z"
Sep 18 19:44:48.545994 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:48.545803    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1da6cd04-3316-4a50-aa5b-29985654056a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754066c2z\" (UID: \"1da6cd04-3316-4a50-aa5b-29985654056a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754066c2z"
Sep 18 19:44:48.545994 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:48.545942    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1da6cd04-3316-4a50-aa5b-29985654056a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754066c2z\" (UID: \"1da6cd04-3316-4a50-aa5b-29985654056a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754066c2z"
Sep 18 19:44:48.647282 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:48.647179    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1da6cd04-3316-4a50-aa5b-29985654056a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754066c2z\" (UID: \"1da6cd04-3316-4a50-aa5b-29985654056a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754066c2z"
Sep 18 19:44:48.647282 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:48.647254    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km4sg\" (UniqueName: \"kubernetes.io/projected/1da6cd04-3316-4a50-aa5b-29985654056a-kube-api-access-km4sg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754066c2z\" (UID: \"1da6cd04-3316-4a50-aa5b-29985654056a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754066c2z"
Sep 18 19:44:48.647282 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:48.647277    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1da6cd04-3316-4a50-aa5b-29985654056a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754066c2z\" (UID: \"1da6cd04-3316-4a50-aa5b-29985654056a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754066c2z"
Sep 18 19:44:48.647641 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:48.647616    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1da6cd04-3316-4a50-aa5b-29985654056a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754066c2z\" (UID: \"1da6cd04-3316-4a50-aa5b-29985654056a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754066c2z"
Sep 18 19:44:48.647705 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:48.647627    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1da6cd04-3316-4a50-aa5b-29985654056a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754066c2z\" (UID: \"1da6cd04-3316-4a50-aa5b-29985654056a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754066c2z"
Sep 18 19:44:48.655711 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:48.655684    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km4sg\" (UniqueName: \"kubernetes.io/projected/1da6cd04-3316-4a50-aa5b-29985654056a-kube-api-access-km4sg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754066c2z\" (UID: \"1da6cd04-3316-4a50-aa5b-29985654056a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754066c2z"
Sep 18 19:44:48.691708 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:48.691676    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754066c2z"
Sep 18 19:44:48.959378 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:48.959354    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754066c2z"]
Sep 18 19:44:48.961644 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:44:48.961614    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da6cd04_3316_4a50_aa5b_29985654056a.slice/crio-4c3b9a6459361f86287e4a7fc2ef3cb5a513550052dac42bd5ece832202180cd WatchSource:0}: Error finding container 4c3b9a6459361f86287e4a7fc2ef3cb5a513550052dac42bd5ece832202180cd: Status 404 returned error can't find the container with id 4c3b9a6459361f86287e4a7fc2ef3cb5a513550052dac42bd5ece832202180cd
Sep 18 19:44:49.244450 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:49.244369    2581 generic.go:356] "Generic (PLEG): container finished" podID="1da6cd04-3316-4a50-aa5b-29985654056a" containerID="0d2341111321890c2e0f3e5e2a176ee0330b359202bd57fa9eb58dcebe550094" exitCode=0
Sep 18 19:44:49.244450 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:49.244442    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754066c2z" event={"ID":"1da6cd04-3316-4a50-aa5b-29985654056a","Type":"ContainerDied","Data":"0d2341111321890c2e0f3e5e2a176ee0330b359202bd57fa9eb58dcebe550094"}
Sep 18 19:44:49.244628 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:49.244479    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754066c2z" event={"ID":"1da6cd04-3316-4a50-aa5b-29985654056a","Type":"ContainerStarted","Data":"4c3b9a6459361f86287e4a7fc2ef3cb5a513550052dac42bd5ece832202180cd"}
Sep 18 19:44:51.255899 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:51.255864    2581 generic.go:356] "Generic (PLEG): container finished" podID="1da6cd04-3316-4a50-aa5b-29985654056a" containerID="bd1358398f6145daaebb0196a4784cc593b952c14499356a8cca2f70a84783d6" exitCode=0
Sep 18 19:44:51.256366 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:51.255917    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754066c2z" event={"ID":"1da6cd04-3316-4a50-aa5b-29985654056a","Type":"ContainerDied","Data":"bd1358398f6145daaebb0196a4784cc593b952c14499356a8cca2f70a84783d6"}
Sep 18 19:44:52.262186 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:52.262150    2581 generic.go:356] "Generic (PLEG): container finished" podID="1da6cd04-3316-4a50-aa5b-29985654056a" containerID="2c30b2a4a43988b1aacf01f413ba03cbd70d7418ef358f90769b4545496053ab" exitCode=0
Sep 18 19:44:52.262543 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:52.262236    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754066c2z" event={"ID":"1da6cd04-3316-4a50-aa5b-29985654056a","Type":"ContainerDied","Data":"2c30b2a4a43988b1aacf01f413ba03cbd70d7418ef358f90769b4545496053ab"}
Sep 18 19:44:53.393137 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:53.393114    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754066c2z"
Sep 18 19:44:53.487147 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:53.487122    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1da6cd04-3316-4a50-aa5b-29985654056a-util\") pod \"1da6cd04-3316-4a50-aa5b-29985654056a\" (UID: \"1da6cd04-3316-4a50-aa5b-29985654056a\") "
Sep 18 19:44:53.487278 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:53.487153    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1da6cd04-3316-4a50-aa5b-29985654056a-bundle\") pod \"1da6cd04-3316-4a50-aa5b-29985654056a\" (UID: \"1da6cd04-3316-4a50-aa5b-29985654056a\") "
Sep 18 19:44:53.487278 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:53.487195    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-km4sg\" (UniqueName: \"kubernetes.io/projected/1da6cd04-3316-4a50-aa5b-29985654056a-kube-api-access-km4sg\") pod \"1da6cd04-3316-4a50-aa5b-29985654056a\" (UID: \"1da6cd04-3316-4a50-aa5b-29985654056a\") "
Sep 18 19:44:53.487818 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:53.487788    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1da6cd04-3316-4a50-aa5b-29985654056a-bundle" (OuterVolumeSpecName: "bundle") pod "1da6cd04-3316-4a50-aa5b-29985654056a" (UID: "1da6cd04-3316-4a50-aa5b-29985654056a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:44:53.489289 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:53.489265    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da6cd04-3316-4a50-aa5b-29985654056a-kube-api-access-km4sg" (OuterVolumeSpecName: "kube-api-access-km4sg") pod "1da6cd04-3316-4a50-aa5b-29985654056a" (UID: "1da6cd04-3316-4a50-aa5b-29985654056a"). InnerVolumeSpecName "kube-api-access-km4sg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:44:53.492668 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:53.492648    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1da6cd04-3316-4a50-aa5b-29985654056a-util" (OuterVolumeSpecName: "util") pod "1da6cd04-3316-4a50-aa5b-29985654056a" (UID: "1da6cd04-3316-4a50-aa5b-29985654056a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:44:53.588575 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:53.588553    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1da6cd04-3316-4a50-aa5b-29985654056a-util\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:44:53.588575 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:53.588574    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1da6cd04-3316-4a50-aa5b-29985654056a-bundle\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:44:53.588698 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:53.588583    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km4sg\" (UniqueName: \"kubernetes.io/projected/1da6cd04-3316-4a50-aa5b-29985654056a-kube-api-access-km4sg\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:44:54.278334 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:54.278299    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754066c2z" event={"ID":"1da6cd04-3316-4a50-aa5b-29985654056a","Type":"ContainerDied","Data":"4c3b9a6459361f86287e4a7fc2ef3cb5a513550052dac42bd5ece832202180cd"}
Sep 18 19:44:54.278334 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:54.278319    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e6754066c2z"
Sep 18 19:44:54.278582 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:44:54.278366    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c3b9a6459361f86287e4a7fc2ef3cb5a513550052dac42bd5ece832202180cd"
Sep 18 19:45:01.494349 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:01.494316    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f6j9x"]
Sep 18 19:45:01.494723 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:01.494649    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1da6cd04-3316-4a50-aa5b-29985654056a" containerName="extract"
Sep 18 19:45:01.494723 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:01.494660    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1da6cd04-3316-4a50-aa5b-29985654056a" containerName="extract"
Sep 18 19:45:01.494723 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:01.494671    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1da6cd04-3316-4a50-aa5b-29985654056a" containerName="util"
Sep 18 19:45:01.494723 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:01.494676    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1da6cd04-3316-4a50-aa5b-29985654056a" containerName="util"
Sep 18 19:45:01.494723 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:01.494692    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1da6cd04-3316-4a50-aa5b-29985654056a" containerName="pull"
Sep 18 19:45:01.494723 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:01.494697    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1da6cd04-3316-4a50-aa5b-29985654056a" containerName="pull"
Sep 18 19:45:01.494972 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:01.494745    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1da6cd04-3316-4a50-aa5b-29985654056a" containerName="extract"
Sep 18 19:45:01.497802 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:01.497787    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f6j9x"
Sep 18 19:45:01.500431 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:01.500408    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-gf6p5\""
Sep 18 19:45:01.500554 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:01.500439    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:45:01.500554 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:01.500437    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 18 19:45:01.515005 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:01.514979    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f6j9x"]
Sep 18 19:45:01.654132 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:01.654101    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd5wp\" (UniqueName: \"kubernetes.io/projected/14ddcad5-caba-4456-8c2e-5f645912e6c2-kube-api-access-vd5wp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-f6j9x\" (UID: \"14ddcad5-caba-4456-8c2e-5f645912e6c2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f6j9x"
Sep 18 19:45:01.654290 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:01.654207    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14ddcad5-caba-4456-8c2e-5f645912e6c2-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-f6j9x\" (UID: \"14ddcad5-caba-4456-8c2e-5f645912e6c2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f6j9x"
Sep 18 19:45:01.755531 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:01.755444    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd5wp\" (UniqueName: \"kubernetes.io/projected/14ddcad5-caba-4456-8c2e-5f645912e6c2-kube-api-access-vd5wp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-f6j9x\" (UID: \"14ddcad5-caba-4456-8c2e-5f645912e6c2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f6j9x"
Sep 18 19:45:01.755688 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:01.755553    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14ddcad5-caba-4456-8c2e-5f645912e6c2-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-f6j9x\" (UID: \"14ddcad5-caba-4456-8c2e-5f645912e6c2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f6j9x"
Sep 18 19:45:01.755962 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:01.755942    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14ddcad5-caba-4456-8c2e-5f645912e6c2-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-f6j9x\" (UID: \"14ddcad5-caba-4456-8c2e-5f645912e6c2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f6j9x"
Sep 18 19:45:01.763506 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:01.763484    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd5wp\" (UniqueName: \"kubernetes.io/projected/14ddcad5-caba-4456-8c2e-5f645912e6c2-kube-api-access-vd5wp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-f6j9x\" (UID: \"14ddcad5-caba-4456-8c2e-5f645912e6c2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f6j9x"
Sep 18 19:45:01.806761 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:01.806738    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f6j9x"
Sep 18 19:45:02.081523 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:02.081501    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f6j9x"]
Sep 18 19:45:02.083648 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:45:02.083621    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14ddcad5_caba_4456_8c2e_5f645912e6c2.slice/crio-2068b793633485c133dc4ebbe60e78c79c9b0e9963fc457e70704ac43bdef6bc WatchSource:0}: Error finding container 2068b793633485c133dc4ebbe60e78c79c9b0e9963fc457e70704ac43bdef6bc: Status 404 returned error can't find the container with id 2068b793633485c133dc4ebbe60e78c79c9b0e9963fc457e70704ac43bdef6bc
Sep 18 19:45:02.312430 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:02.312393    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f6j9x" event={"ID":"14ddcad5-caba-4456-8c2e-5f645912e6c2","Type":"ContainerStarted","Data":"2068b793633485c133dc4ebbe60e78c79c9b0e9963fc457e70704ac43bdef6bc"}
Sep 18 19:45:04.324013 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:04.323919    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f6j9x" event={"ID":"14ddcad5-caba-4456-8c2e-5f645912e6c2","Type":"ContainerStarted","Data":"dc00cab6b1bfd2bc84f55573b88d6d2a9916b2cd7e29e465abe84a0f2da0a82e"}
Sep 18 19:45:04.348716 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:04.348673    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f6j9x" podStartSLOduration=1.393468821 podStartE2EDuration="3.3486582s" totalImagesPullingTime="1.955189379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:45:04.346183515 +0000 UTC m=+427.448471842" watchObservedRunningTime="2026-09-18 19:45:04.3486582 +0000 UTC m=+427.450946531"
Sep 18 19:45:07.256178 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.256147    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cc68c547f-kmlfx"]
Sep 18 19:45:07.259481 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.259467    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cc68c547f-kmlfx"
Sep 18 19:45:07.261848 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.261817    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 19:45:07.261972 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.261876    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 19:45:07.261972 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.261919    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 19:45:07.262115 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.261997    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 19:45:07.262174 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.262158    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 19:45:07.262973 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.262950    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hrjgw\""
Sep 18 19:45:07.262973 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.262970    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 19:45:07.263168 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.263023    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 19:45:07.266626 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.266605    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 18 19:45:07.269854 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.269819    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cc68c547f-kmlfx"]
Sep 18 19:45:07.401963 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.401933    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q47sq\" (UniqueName: \"kubernetes.io/projected/df95825b-4cda-4434-8962-457e21e7d633-kube-api-access-q47sq\") pod \"console-6cc68c547f-kmlfx\" (UID: \"df95825b-4cda-4434-8962-457e21e7d633\") " pod="openshift-console/console-6cc68c547f-kmlfx"
Sep 18 19:45:07.402120 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.401980    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df95825b-4cda-4434-8962-457e21e7d633-trusted-ca-bundle\") pod \"console-6cc68c547f-kmlfx\" (UID: \"df95825b-4cda-4434-8962-457e21e7d633\") " pod="openshift-console/console-6cc68c547f-kmlfx"
Sep 18 19:45:07.402120 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.402040    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df95825b-4cda-4434-8962-457e21e7d633-service-ca\") pod \"console-6cc68c547f-kmlfx\" (UID: \"df95825b-4cda-4434-8962-457e21e7d633\") " pod="openshift-console/console-6cc68c547f-kmlfx"
Sep 18 19:45:07.402120 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.402097    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df95825b-4cda-4434-8962-457e21e7d633-console-serving-cert\") pod \"console-6cc68c547f-kmlfx\" (UID: \"df95825b-4cda-4434-8962-457e21e7d633\") " pod="openshift-console/console-6cc68c547f-kmlfx"
Sep 18 19:45:07.402256 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.402132    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df95825b-4cda-4434-8962-457e21e7d633-oauth-serving-cert\") pod \"console-6cc68c547f-kmlfx\" (UID: \"df95825b-4cda-4434-8962-457e21e7d633\") " pod="openshift-console/console-6cc68c547f-kmlfx"
Sep 18 19:45:07.402256 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.402188    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df95825b-4cda-4434-8962-457e21e7d633-console-oauth-config\") pod \"console-6cc68c547f-kmlfx\" (UID: \"df95825b-4cda-4434-8962-457e21e7d633\") " pod="openshift-console/console-6cc68c547f-kmlfx"
Sep 18 19:45:07.402256 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.402209    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df95825b-4cda-4434-8962-457e21e7d633-console-config\") pod \"console-6cc68c547f-kmlfx\" (UID: \"df95825b-4cda-4434-8962-457e21e7d633\") " pod="openshift-console/console-6cc68c547f-kmlfx"
Sep 18 19:45:07.503219 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.503189    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q47sq\" (UniqueName: \"kubernetes.io/projected/df95825b-4cda-4434-8962-457e21e7d633-kube-api-access-q47sq\") pod \"console-6cc68c547f-kmlfx\" (UID: \"df95825b-4cda-4434-8962-457e21e7d633\") " pod="openshift-console/console-6cc68c547f-kmlfx"
Sep 18 19:45:07.503402 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.503248    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df95825b-4cda-4434-8962-457e21e7d633-trusted-ca-bundle\") pod \"console-6cc68c547f-kmlfx\" (UID: \"df95825b-4cda-4434-8962-457e21e7d633\") " pod="openshift-console/console-6cc68c547f-kmlfx"
Sep 18 19:45:07.503402 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.503295    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df95825b-4cda-4434-8962-457e21e7d633-service-ca\") pod \"console-6cc68c547f-kmlfx\" (UID: \"df95825b-4cda-4434-8962-457e21e7d633\") " pod="openshift-console/console-6cc68c547f-kmlfx"
Sep 18 19:45:07.503402 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.503347    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df95825b-4cda-4434-8962-457e21e7d633-console-serving-cert\") pod \"console-6cc68c547f-kmlfx\" (UID: \"df95825b-4cda-4434-8962-457e21e7d633\") " pod="openshift-console/console-6cc68c547f-kmlfx"
Sep 18 19:45:07.503402 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.503377    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df95825b-4cda-4434-8962-457e21e7d633-oauth-serving-cert\") pod \"console-6cc68c547f-kmlfx\" (UID: \"df95825b-4cda-4434-8962-457e21e7d633\") " pod="openshift-console/console-6cc68c547f-kmlfx"
Sep 18 19:45:07.503402 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.503395    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df95825b-4cda-4434-8962-457e21e7d633-console-oauth-config\") pod \"console-6cc68c547f-kmlfx\" (UID: \"df95825b-4cda-4434-8962-457e21e7d633\") " pod="openshift-console/console-6cc68c547f-kmlfx"
Sep 18 19:45:07.503630 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.503409    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df95825b-4cda-4434-8962-457e21e7d633-console-config\") pod \"console-6cc68c547f-kmlfx\" (UID: \"df95825b-4cda-4434-8962-457e21e7d633\") " pod="openshift-console/console-6cc68c547f-kmlfx"
Sep 18 19:45:07.504141 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.504121    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df95825b-4cda-4434-8962-457e21e7d633-service-ca\") pod \"console-6cc68c547f-kmlfx\" (UID: \"df95825b-4cda-4434-8962-457e21e7d633\") " pod="openshift-console/console-6cc68c547f-kmlfx"
Sep 18 19:45:07.504266 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.504169    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df95825b-4cda-4434-8962-457e21e7d633-console-config\") pod \"console-6cc68c547f-kmlfx\" (UID: \"df95825b-4cda-4434-8962-457e21e7d633\") " pod="openshift-console/console-6cc68c547f-kmlfx"
Sep 18 19:45:07.504266 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.504255    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df95825b-4cda-4434-8962-457e21e7d633-oauth-serving-cert\") pod \"console-6cc68c547f-kmlfx\" (UID: \"df95825b-4cda-4434-8962-457e21e7d633\") " pod="openshift-console/console-6cc68c547f-kmlfx"
Sep 18 19:45:07.504482 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.504454    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df95825b-4cda-4434-8962-457e21e7d633-trusted-ca-bundle\") pod \"console-6cc68c547f-kmlfx\" (UID: \"df95825b-4cda-4434-8962-457e21e7d633\") " pod="openshift-console/console-6cc68c547f-kmlfx"
Sep 18 19:45:07.505981 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.505964    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df95825b-4cda-4434-8962-457e21e7d633-console-oauth-config\") pod \"console-6cc68c547f-kmlfx\" (UID: \"df95825b-4cda-4434-8962-457e21e7d633\") " pod="openshift-console/console-6cc68c547f-kmlfx"
Sep 18 19:45:07.506145 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.506130    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df95825b-4cda-4434-8962-457e21e7d633-console-serving-cert\") pod \"console-6cc68c547f-kmlfx\" (UID: \"df95825b-4cda-4434-8962-457e21e7d633\") " pod="openshift-console/console-6cc68c547f-kmlfx"
Sep 18 19:45:07.510904 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.510851    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q47sq\" (UniqueName: \"kubernetes.io/projected/df95825b-4cda-4434-8962-457e21e7d633-kube-api-access-q47sq\") pod \"console-6cc68c547f-kmlfx\" (UID: \"df95825b-4cda-4434-8962-457e21e7d633\") " pod="openshift-console/console-6cc68c547f-kmlfx"
Sep 18 19:45:07.569284 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.569247    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cc68c547f-kmlfx"
Sep 18 19:45:07.841640 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:07.841617    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cc68c547f-kmlfx"]
Sep 18 19:45:07.844142 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:45:07.844114    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf95825b_4cda_4434_8962_457e21e7d633.slice/crio-cb12760a2e41441ad5ba269ca9f46d1d864fc0ab9276f5266045b10582afa3df WatchSource:0}: Error finding container cb12760a2e41441ad5ba269ca9f46d1d864fc0ab9276f5266045b10582afa3df: Status 404 returned error can't find the container with id cb12760a2e41441ad5ba269ca9f46d1d864fc0ab9276f5266045b10582afa3df
Sep 18 19:45:08.343708 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:08.343676    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cc68c547f-kmlfx" event={"ID":"df95825b-4cda-4434-8962-457e21e7d633","Type":"ContainerStarted","Data":"96b14af78bef930122b023edbc6a0dbabf1421d5a8293e06a36465504a938504"}
Sep 18 19:45:08.344084 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:08.343723    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cc68c547f-kmlfx" event={"ID":"df95825b-4cda-4434-8962-457e21e7d633","Type":"ContainerStarted","Data":"cb12760a2e41441ad5ba269ca9f46d1d864fc0ab9276f5266045b10582afa3df"}
Sep 18 19:45:08.365871 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:08.365806    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6cc68c547f-kmlfx" podStartSLOduration=1.36579125 podStartE2EDuration="1.36579125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:45:08.364924789 +0000 UTC m=+431.467213117" watchObservedRunningTime="2026-09-18 19:45:08.36579125 +0000 UTC m=+431.468079629"
Sep 18 19:45:10.698936 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:10.698904    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2wrcq"]
Sep 18 19:45:10.702468 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:10.702452    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2wrcq"
Sep 18 19:45:10.705085 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:10.705063    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 19:45:10.705989 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:10.705972    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 19:45:10.706095 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:10.705991    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lg99k\""
Sep 18 19:45:10.712309 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:10.712289    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2wrcq"]
Sep 18 19:45:10.829186 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:10.829157    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f9dafe8-d356-4b88-b834-aafbd3d02997-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2wrcq\" (UID: \"1f9dafe8-d356-4b88-b834-aafbd3d02997\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2wrcq"
Sep 18 19:45:10.829327 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:10.829207    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg2c6\" (UniqueName: \"kubernetes.io/projected/1f9dafe8-d356-4b88-b834-aafbd3d02997-kube-api-access-cg2c6\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2wrcq\" (UID: \"1f9dafe8-d356-4b88-b834-aafbd3d02997\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2wrcq"
Sep 18 19:45:10.829327 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:10.829277    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f9dafe8-d356-4b88-b834-aafbd3d02997-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2wrcq\" (UID: \"1f9dafe8-d356-4b88-b834-aafbd3d02997\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2wrcq"
Sep 18 19:45:10.930489 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:10.930458    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f9dafe8-d356-4b88-b834-aafbd3d02997-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2wrcq\" (UID: \"1f9dafe8-d356-4b88-b834-aafbd3d02997\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2wrcq"
Sep 18 19:45:10.930616 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:10.930505    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg2c6\" (UniqueName: \"kubernetes.io/projected/1f9dafe8-d356-4b88-b834-aafbd3d02997-kube-api-access-cg2c6\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2wrcq\" (UID: \"1f9dafe8-d356-4b88-b834-aafbd3d02997\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2wrcq"
Sep 18 19:45:10.930616 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:10.930550    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f9dafe8-d356-4b88-b834-aafbd3d02997-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2wrcq\" (UID: \"1f9dafe8-d356-4b88-b834-aafbd3d02997\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2wrcq"
Sep 18 19:45:10.930868 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:10.930826    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f9dafe8-d356-4b88-b834-aafbd3d02997-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2wrcq\" (UID: \"1f9dafe8-d356-4b88-b834-aafbd3d02997\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2wrcq"
Sep 18 19:45:10.930952 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:10.930880    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f9dafe8-d356-4b88-b834-aafbd3d02997-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2wrcq\" (UID: \"1f9dafe8-d356-4b88-b834-aafbd3d02997\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2wrcq"
Sep 18 19:45:10.938706 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:10.938685    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg2c6\" (UniqueName: \"kubernetes.io/projected/1f9dafe8-d356-4b88-b834-aafbd3d02997-kube-api-access-cg2c6\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2wrcq\" (UID: \"1f9dafe8-d356-4b88-b834-aafbd3d02997\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2wrcq"
Sep 18 19:45:11.011693 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:11.011635    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2wrcq"
Sep 18 19:45:11.280731 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:11.280705    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2wrcq"]
Sep 18 19:45:11.282397 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:45:11.282375    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f9dafe8_d356_4b88_b834_aafbd3d02997.slice/crio-ca49f56e2b13fe25564f53be9e7aa57b96990f5d4024317f4d3009c1433490fa WatchSource:0}: Error finding container ca49f56e2b13fe25564f53be9e7aa57b96990f5d4024317f4d3009c1433490fa: Status 404 returned error can't find the container with id ca49f56e2b13fe25564f53be9e7aa57b96990f5d4024317f4d3009c1433490fa
Sep 18 19:45:11.358633 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:11.358600    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2wrcq" event={"ID":"1f9dafe8-d356-4b88-b834-aafbd3d02997","Type":"ContainerStarted","Data":"fd550e919a1319ba252026af6be39585ef33a3969d1292d464e504ee32a3e1f0"}
Sep 18 19:45:11.358745 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:11.358663    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2wrcq" event={"ID":"1f9dafe8-d356-4b88-b834-aafbd3d02997","Type":"ContainerStarted","Data":"ca49f56e2b13fe25564f53be9e7aa57b96990f5d4024317f4d3009c1433490fa"}
Sep 18 19:45:12.364499 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:12.364462    2581 generic.go:356] "Generic (PLEG): container finished" podID="1f9dafe8-d356-4b88-b834-aafbd3d02997" containerID="fd550e919a1319ba252026af6be39585ef33a3969d1292d464e504ee32a3e1f0" exitCode=0
Sep 18 19:45:12.364915 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:12.364550    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2wrcq" event={"ID":"1f9dafe8-d356-4b88-b834-aafbd3d02997","Type":"ContainerDied","Data":"fd550e919a1319ba252026af6be39585ef33a3969d1292d464e504ee32a3e1f0"}
Sep 18 19:45:13.557160 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:13.557120    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ntx2d"]
Sep 18 19:45:13.563922 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:13.563899    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ntx2d"
Sep 18 19:45:13.566402 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:13.566380    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 18 19:45:13.566520 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:13.566436    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-94ttm\""
Sep 18 19:45:13.566520 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:13.566451    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 18 19:45:13.570377 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:13.570358    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ntx2d"]
Sep 18 19:45:13.650086 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:13.650053    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgp4f\" (UniqueName: \"kubernetes.io/projected/08d17bff-58b8-48a7-88c6-e46128c9425b-kube-api-access-sgp4f\") pod \"cert-manager-cainjector-559cdd8c87-ntx2d\" (UID: \"08d17bff-58b8-48a7-88c6-e46128c9425b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ntx2d"
Sep 18 19:45:13.650254 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:13.650142    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08d17bff-58b8-48a7-88c6-e46128c9425b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ntx2d\" (UID: \"08d17bff-58b8-48a7-88c6-e46128c9425b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ntx2d"
Sep 18 19:45:13.751273 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:13.751236    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08d17bff-58b8-48a7-88c6-e46128c9425b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ntx2d\" (UID: \"08d17bff-58b8-48a7-88c6-e46128c9425b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ntx2d"
Sep 18 19:45:13.751459 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:13.751326    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgp4f\" (UniqueName: \"kubernetes.io/projected/08d17bff-58b8-48a7-88c6-e46128c9425b-kube-api-access-sgp4f\") pod \"cert-manager-cainjector-559cdd8c87-ntx2d\" (UID: \"08d17bff-58b8-48a7-88c6-e46128c9425b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ntx2d"
Sep 18 19:45:13.760162 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:13.760118    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/08d17bff-58b8-48a7-88c6-e46128c9425b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ntx2d\" (UID: \"08d17bff-58b8-48a7-88c6-e46128c9425b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ntx2d"
Sep 18 19:45:13.760262 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:13.760172    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgp4f\" (UniqueName: \"kubernetes.io/projected/08d17bff-58b8-48a7-88c6-e46128c9425b-kube-api-access-sgp4f\") pod \"cert-manager-cainjector-559cdd8c87-ntx2d\" (UID: \"08d17bff-58b8-48a7-88c6-e46128c9425b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ntx2d"
Sep 18 19:45:13.875781 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:13.875701    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ntx2d"
Sep 18 19:45:14.377325 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:14.377294    2581 generic.go:356] "Generic (PLEG): container finished" podID="1f9dafe8-d356-4b88-b834-aafbd3d02997" containerID="8bf20aa66b969787a8a2567a47fd6048ad13786700d2b33ec60d2bb354b47445" exitCode=0
Sep 18 19:45:14.377455 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:14.377377    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2wrcq" event={"ID":"1f9dafe8-d356-4b88-b834-aafbd3d02997","Type":"ContainerDied","Data":"8bf20aa66b969787a8a2567a47fd6048ad13786700d2b33ec60d2bb354b47445"}
Sep 18 19:45:14.386360 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:14.386330    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ntx2d"]
Sep 18 19:45:15.386268 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:15.386227    2581 generic.go:356] "Generic (PLEG): container finished" podID="1f9dafe8-d356-4b88-b834-aafbd3d02997" containerID="a67088030cc9e7fc33e168d316c565b531e1628cd937cc8437e04720dd6b6620" exitCode=0
Sep 18 19:45:15.386687 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:15.386317    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2wrcq" event={"ID":"1f9dafe8-d356-4b88-b834-aafbd3d02997","Type":"ContainerDied","Data":"a67088030cc9e7fc33e168d316c565b531e1628cd937cc8437e04720dd6b6620"}
Sep 18 19:45:15.387687 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:15.387660    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ntx2d" event={"ID":"08d17bff-58b8-48a7-88c6-e46128c9425b","Type":"ContainerStarted","Data":"cfad65957d1c16d03d8f30728e90c814ad1a4875e5e661223899a68b71458fad"}
Sep 18 19:45:17.345934 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:17.345911    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2wrcq"
Sep 18 19:45:17.375079 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:17.375058    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cg2c6\" (UniqueName: \"kubernetes.io/projected/1f9dafe8-d356-4b88-b834-aafbd3d02997-kube-api-access-cg2c6\") pod \"1f9dafe8-d356-4b88-b834-aafbd3d02997\" (UID: \"1f9dafe8-d356-4b88-b834-aafbd3d02997\") "
Sep 18 19:45:17.375204 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:17.375101    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f9dafe8-d356-4b88-b834-aafbd3d02997-util\") pod \"1f9dafe8-d356-4b88-b834-aafbd3d02997\" (UID: \"1f9dafe8-d356-4b88-b834-aafbd3d02997\") "
Sep 18 19:45:17.375204 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:17.375136    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f9dafe8-d356-4b88-b834-aafbd3d02997-bundle\") pod \"1f9dafe8-d356-4b88-b834-aafbd3d02997\" (UID: \"1f9dafe8-d356-4b88-b834-aafbd3d02997\") "
Sep 18 19:45:17.375545 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:17.375522    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9dafe8-d356-4b88-b834-aafbd3d02997-bundle" (OuterVolumeSpecName: "bundle") pod "1f9dafe8-d356-4b88-b834-aafbd3d02997" (UID: "1f9dafe8-d356-4b88-b834-aafbd3d02997"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:45:17.377301 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:17.377277    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f9dafe8-d356-4b88-b834-aafbd3d02997-kube-api-access-cg2c6" (OuterVolumeSpecName: "kube-api-access-cg2c6") pod "1f9dafe8-d356-4b88-b834-aafbd3d02997" (UID: "1f9dafe8-d356-4b88-b834-aafbd3d02997"). InnerVolumeSpecName "kube-api-access-cg2c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:45:17.382675 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:17.382647    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9dafe8-d356-4b88-b834-aafbd3d02997-util" (OuterVolumeSpecName: "util") pod "1f9dafe8-d356-4b88-b834-aafbd3d02997" (UID: "1f9dafe8-d356-4b88-b834-aafbd3d02997"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:45:17.402454 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:17.402226    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2wrcq" event={"ID":"1f9dafe8-d356-4b88-b834-aafbd3d02997","Type":"ContainerDied","Data":"ca49f56e2b13fe25564f53be9e7aa57b96990f5d4024317f4d3009c1433490fa"}
Sep 18 19:45:17.402454 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:17.402273    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca49f56e2b13fe25564f53be9e7aa57b96990f5d4024317f4d3009c1433490fa"
Sep 18 19:45:17.402454 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:17.402292    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f2wrcq"
Sep 18 19:45:17.475760 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:17.475732    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f9dafe8-d356-4b88-b834-aafbd3d02997-util\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:45:17.475760 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:17.475757    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f9dafe8-d356-4b88-b834-aafbd3d02997-bundle\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:45:17.475931 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:17.475779    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg2c6\" (UniqueName: \"kubernetes.io/projected/1f9dafe8-d356-4b88-b834-aafbd3d02997-kube-api-access-cg2c6\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:45:17.569556 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:17.569525    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6cc68c547f-kmlfx"
Sep 18 19:45:17.569704 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:17.569576    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cc68c547f-kmlfx"
Sep 18 19:45:17.575893 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:17.575868    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6cc68c547f-kmlfx"
Sep 18 19:45:18.408257 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:18.408220    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ntx2d" event={"ID":"08d17bff-58b8-48a7-88c6-e46128c9425b","Type":"ContainerStarted","Data":"6c54a55bbe862fb3a500e74ae2cf6f6d35d50fded54f753e7e26bc2b63fa9f0f"}
Sep 18 19:45:18.412048 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:18.412027    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6cc68c547f-kmlfx"
Sep 18 19:45:18.426857 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:18.426783    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ntx2d" podStartSLOduration=2.432026242 podStartE2EDuration="5.426768852s" totalImagesPullingTime="2.99474261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:45:18.42560983 +0000 UTC m=+441.527898158" watchObservedRunningTime="2026-09-18 19:45:18.426768852 +0000 UTC m=+441.529057179"
Sep 18 19:45:23.964439 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:23.964399    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fjbfv"]
Sep 18 19:45:23.965256 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:23.964872    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f9dafe8-d356-4b88-b834-aafbd3d02997" containerName="pull"
Sep 18 19:45:23.965256 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:23.964890    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f9dafe8-d356-4b88-b834-aafbd3d02997" containerName="pull"
Sep 18 19:45:23.965256 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:23.964920    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f9dafe8-d356-4b88-b834-aafbd3d02997" containerName="extract"
Sep 18 19:45:23.965256 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:23.964928    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f9dafe8-d356-4b88-b834-aafbd3d02997" containerName="extract"
Sep 18 19:45:23.965256 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:23.964950    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f9dafe8-d356-4b88-b834-aafbd3d02997" containerName="util"
Sep 18 19:45:23.965256 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:23.964957    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f9dafe8-d356-4b88-b834-aafbd3d02997" containerName="util"
Sep 18 19:45:23.965256 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:23.965031    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f9dafe8-d356-4b88-b834-aafbd3d02997" containerName="extract"
Sep 18 19:45:23.969873 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:23.969852    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fjbfv"
Sep 18 19:45:23.972159 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:23.972140    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 18 19:45:23.973017 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:23.973002    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-xxrnb\""
Sep 18 19:45:23.973074 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:23.973006    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:45:23.976312 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:23.976291    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fjbfv"]
Sep 18 19:45:24.025241 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:24.025210    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ce1a023-ce9b-441d-ad6d-7c5790179b51-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fjbfv\" (UID: \"0ce1a023-ce9b-441d-ad6d-7c5790179b51\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fjbfv"
Sep 18 19:45:24.025388 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:24.025257    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9g84\" (UniqueName: \"kubernetes.io/projected/0ce1a023-ce9b-441d-ad6d-7c5790179b51-kube-api-access-r9g84\") pod \"openshift-lws-operator-fd8ccff4c-fjbfv\" (UID: \"0ce1a023-ce9b-441d-ad6d-7c5790179b51\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fjbfv"
Sep 18 19:45:24.126455 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:24.126427    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9g84\" (UniqueName: \"kubernetes.io/projected/0ce1a023-ce9b-441d-ad6d-7c5790179b51-kube-api-access-r9g84\") pod \"openshift-lws-operator-fd8ccff4c-fjbfv\" (UID: \"0ce1a023-ce9b-441d-ad6d-7c5790179b51\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fjbfv"
Sep 18 19:45:24.126577 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:24.126487    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ce1a023-ce9b-441d-ad6d-7c5790179b51-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fjbfv\" (UID: \"0ce1a023-ce9b-441d-ad6d-7c5790179b51\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fjbfv"
Sep 18 19:45:24.126866 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:24.126826    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ce1a023-ce9b-441d-ad6d-7c5790179b51-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fjbfv\" (UID: \"0ce1a023-ce9b-441d-ad6d-7c5790179b51\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fjbfv"
Sep 18 19:45:24.134148 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:24.134123    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9g84\" (UniqueName: \"kubernetes.io/projected/0ce1a023-ce9b-441d-ad6d-7c5790179b51-kube-api-access-r9g84\") pod \"openshift-lws-operator-fd8ccff4c-fjbfv\" (UID: \"0ce1a023-ce9b-441d-ad6d-7c5790179b51\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fjbfv"
Sep 18 19:45:24.280829 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:24.280810    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fjbfv"
Sep 18 19:45:24.548741 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:24.548718    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fjbfv"]
Sep 18 19:45:24.550904 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:45:24.550877    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ce1a023_ce9b_441d_ad6d_7c5790179b51.slice/crio-a05ea256a1d507907845b891711774d331d5ac964b66436fd44f3c948701a952 WatchSource:0}: Error finding container a05ea256a1d507907845b891711774d331d5ac964b66436fd44f3c948701a952: Status 404 returned error can't find the container with id a05ea256a1d507907845b891711774d331d5ac964b66436fd44f3c948701a952
Sep 18 19:45:25.441946 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:25.441915    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fjbfv" event={"ID":"0ce1a023-ce9b-441d-ad6d-7c5790179b51","Type":"ContainerStarted","Data":"a05ea256a1d507907845b891711774d331d5ac964b66436fd44f3c948701a952"}
Sep 18 19:45:28.458602 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:28.458570    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fjbfv" event={"ID":"0ce1a023-ce9b-441d-ad6d-7c5790179b51","Type":"ContainerStarted","Data":"1a7ff44fd37bfa81fc29edfb6357da058d321e8e78b33ede514a57ac6113b305"}
Sep 18 19:45:28.476621 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:28.476556    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fjbfv" podStartSLOduration=2.08911977 podStartE2EDuration="5.476542583s" totalImagesPullingTime="3.387422813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:45:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:45:28.475924729 +0000 UTC m=+451.578213060" watchObservedRunningTime="2026-09-18 19:45:28.476542583 +0000 UTC m=+451.578830910"
Sep 18 19:45:43.620197 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:43.620163    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-d69b774b8-h2kfq"]
Sep 18 19:45:43.622260 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:43.622244    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d69b774b8-h2kfq"
Sep 18 19:45:43.625768 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:43.625747    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 18 19:45:43.625910 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:43.625776    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 18 19:45:43.625910 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:43.625787    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 18 19:45:43.625910 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:43.625751    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-9xlpb\""
Sep 18 19:45:43.632930 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:43.632908    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d69b774b8-h2kfq"]
Sep 18 19:45:43.686194 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:43.686161    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc342203-c44d-4cb8-84bb-2b70c1f759ad-metrics-cert\") pod \"lws-controller-manager-d69b774b8-h2kfq\" (UID: \"bc342203-c44d-4cb8-84bb-2b70c1f759ad\") " pod="openshift-lws-operator/lws-controller-manager-d69b774b8-h2kfq"
Sep 18 19:45:43.686372 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:43.686272    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bc342203-c44d-4cb8-84bb-2b70c1f759ad-manager-config\") pod \"lws-controller-manager-d69b774b8-h2kfq\" (UID: \"bc342203-c44d-4cb8-84bb-2b70c1f759ad\") " pod="openshift-lws-operator/lws-controller-manager-d69b774b8-h2kfq"
Sep 18 19:45:43.686372 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:43.686303    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75b7m\" (UniqueName: \"kubernetes.io/projected/bc342203-c44d-4cb8-84bb-2b70c1f759ad-kube-api-access-75b7m\") pod \"lws-controller-manager-d69b774b8-h2kfq\" (UID: \"bc342203-c44d-4cb8-84bb-2b70c1f759ad\") " pod="openshift-lws-operator/lws-controller-manager-d69b774b8-h2kfq"
Sep 18 19:45:43.686372 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:43.686364    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc342203-c44d-4cb8-84bb-2b70c1f759ad-cert\") pod \"lws-controller-manager-d69b774b8-h2kfq\" (UID: \"bc342203-c44d-4cb8-84bb-2b70c1f759ad\") " pod="openshift-lws-operator/lws-controller-manager-d69b774b8-h2kfq"
Sep 18 19:45:43.786811 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:43.786779    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bc342203-c44d-4cb8-84bb-2b70c1f759ad-manager-config\") pod \"lws-controller-manager-d69b774b8-h2kfq\" (UID: \"bc342203-c44d-4cb8-84bb-2b70c1f759ad\") " pod="openshift-lws-operator/lws-controller-manager-d69b774b8-h2kfq"
Sep 18 19:45:43.786993 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:43.786820    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75b7m\" (UniqueName: \"kubernetes.io/projected/bc342203-c44d-4cb8-84bb-2b70c1f759ad-kube-api-access-75b7m\") pod \"lws-controller-manager-d69b774b8-h2kfq\" (UID: \"bc342203-c44d-4cb8-84bb-2b70c1f759ad\") " pod="openshift-lws-operator/lws-controller-manager-d69b774b8-h2kfq"
Sep 18 19:45:43.786993 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:43.786871    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc342203-c44d-4cb8-84bb-2b70c1f759ad-cert\") pod \"lws-controller-manager-d69b774b8-h2kfq\" (UID: \"bc342203-c44d-4cb8-84bb-2b70c1f759ad\") " pod="openshift-lws-operator/lws-controller-manager-d69b774b8-h2kfq"
Sep 18 19:45:43.786993 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:43.786913    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc342203-c44d-4cb8-84bb-2b70c1f759ad-metrics-cert\") pod \"lws-controller-manager-d69b774b8-h2kfq\" (UID: \"bc342203-c44d-4cb8-84bb-2b70c1f759ad\") " pod="openshift-lws-operator/lws-controller-manager-d69b774b8-h2kfq"
Sep 18 19:45:43.787461 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:43.787439    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bc342203-c44d-4cb8-84bb-2b70c1f759ad-manager-config\") pod \"lws-controller-manager-d69b774b8-h2kfq\" (UID: \"bc342203-c44d-4cb8-84bb-2b70c1f759ad\") " pod="openshift-lws-operator/lws-controller-manager-d69b774b8-h2kfq"
Sep 18 19:45:43.789432 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:43.789407    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc342203-c44d-4cb8-84bb-2b70c1f759ad-cert\") pod \"lws-controller-manager-d69b774b8-h2kfq\" (UID: \"bc342203-c44d-4cb8-84bb-2b70c1f759ad\") " pod="openshift-lws-operator/lws-controller-manager-d69b774b8-h2kfq"
Sep 18 19:45:43.789516 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:43.789455    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bc342203-c44d-4cb8-84bb-2b70c1f759ad-metrics-cert\") pod \"lws-controller-manager-d69b774b8-h2kfq\" (UID: \"bc342203-c44d-4cb8-84bb-2b70c1f759ad\") " pod="openshift-lws-operator/lws-controller-manager-d69b774b8-h2kfq"
Sep 18 19:45:43.800849 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:43.800820    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75b7m\" (UniqueName: \"kubernetes.io/projected/bc342203-c44d-4cb8-84bb-2b70c1f759ad-kube-api-access-75b7m\") pod \"lws-controller-manager-d69b774b8-h2kfq\" (UID: \"bc342203-c44d-4cb8-84bb-2b70c1f759ad\") " pod="openshift-lws-operator/lws-controller-manager-d69b774b8-h2kfq"
Sep 18 19:45:43.931575 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:43.931485    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d69b774b8-h2kfq"
Sep 18 19:45:44.206080 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:44.206052    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d69b774b8-h2kfq"]
Sep 18 19:45:44.208024 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:45:44.207992    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc342203_c44d_4cb8_84bb_2b70c1f759ad.slice/crio-f2fd1a741ae5d7ec58a4988b4d467c43c66a44509770e9cb992823f1d5eda025 WatchSource:0}: Error finding container f2fd1a741ae5d7ec58a4988b4d467c43c66a44509770e9cb992823f1d5eda025: Status 404 returned error can't find the container with id f2fd1a741ae5d7ec58a4988b4d467c43c66a44509770e9cb992823f1d5eda025
Sep 18 19:45:44.532906 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:44.532871    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d69b774b8-h2kfq" event={"ID":"bc342203-c44d-4cb8-84bb-2b70c1f759ad","Type":"ContainerStarted","Data":"f2fd1a741ae5d7ec58a4988b4d467c43c66a44509770e9cb992823f1d5eda025"}
Sep 18 19:45:47.549203 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:47.549168    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d69b774b8-h2kfq" event={"ID":"bc342203-c44d-4cb8-84bb-2b70c1f759ad","Type":"ContainerStarted","Data":"f46996a692ae26ee8b5763f823820d2de7e861482bed0586be71fe879d99c779"}
Sep 18 19:45:47.549599 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:47.549303    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-d69b774b8-h2kfq"
Sep 18 19:45:47.565670 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:47.565627    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-d69b774b8-h2kfq" podStartSLOduration=1.651294825 podStartE2EDuration="4.565612968s" totalImagesPullingTime="2.914318143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:45:47.564031689 +0000 UTC m=+470.666320018" watchObservedRunningTime="2026-09-18 19:45:47.565612968 +0000 UTC m=+470.667901349"
Sep 18 19:45:54.180130 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.180094    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcp28"]
Sep 18 19:45:54.183693 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.183674    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcp28"
Sep 18 19:45:54.186349 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.186320    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 19:45:54.186349 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.186340    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lg99k\""
Sep 18 19:45:54.186566 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.186331    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 19:45:54.192324 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.192296    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcp28"]
Sep 18 19:45:54.280137 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.280100    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cctlsp"]
Sep 18 19:45:54.285965 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.285932    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cctlsp"
Sep 18 19:45:54.287306 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.287278    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qssb\" (UniqueName: \"kubernetes.io/projected/20f23180-3073-4c3c-9bfe-08b159f4a2f3-kube-api-access-5qssb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcp28\" (UID: \"20f23180-3073-4c3c-9bfe-08b159f4a2f3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcp28"
Sep 18 19:45:54.287410 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.287328    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20f23180-3073-4c3c-9bfe-08b159f4a2f3-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcp28\" (UID: \"20f23180-3073-4c3c-9bfe-08b159f4a2f3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcp28"
Sep 18 19:45:54.287410 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.287354    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20f23180-3073-4c3c-9bfe-08b159f4a2f3-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcp28\" (UID: \"20f23180-3073-4c3c-9bfe-08b159f4a2f3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcp28"
Sep 18 19:45:54.291631 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.291604    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cctlsp"]
Sep 18 19:45:54.378055 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.378019    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77m28z"]
Sep 18 19:45:54.381516 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.381502    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77m28z"
Sep 18 19:45:54.388010 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.387985    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qssb\" (UniqueName: \"kubernetes.io/projected/20f23180-3073-4c3c-9bfe-08b159f4a2f3-kube-api-access-5qssb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcp28\" (UID: \"20f23180-3073-4c3c-9bfe-08b159f4a2f3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcp28"
Sep 18 19:45:54.388146 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.388129    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0389a73-9387-4217-b574-c103a81a5093-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cctlsp\" (UID: \"b0389a73-9387-4217-b574-c103a81a5093\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cctlsp"
Sep 18 19:45:54.388259 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.388243    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20f23180-3073-4c3c-9bfe-08b159f4a2f3-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcp28\" (UID: \"20f23180-3073-4c3c-9bfe-08b159f4a2f3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcp28"
Sep 18 19:45:54.388383 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.388357    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20f23180-3073-4c3c-9bfe-08b159f4a2f3-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcp28\" (UID: \"20f23180-3073-4c3c-9bfe-08b159f4a2f3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcp28"
Sep 18 19:45:54.388502 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.388404    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjczt\" (UniqueName: \"kubernetes.io/projected/b0389a73-9387-4217-b574-c103a81a5093-kube-api-access-zjczt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cctlsp\" (UID: \"b0389a73-9387-4217-b574-c103a81a5093\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cctlsp"
Sep 18 19:45:54.388502 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.388445    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0389a73-9387-4217-b574-c103a81a5093-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cctlsp\" (UID: \"b0389a73-9387-4217-b574-c103a81a5093\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cctlsp"
Sep 18 19:45:54.388692 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.388668    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20f23180-3073-4c3c-9bfe-08b159f4a2f3-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcp28\" (UID: \"20f23180-3073-4c3c-9bfe-08b159f4a2f3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcp28"
Sep 18 19:45:54.388771 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.388733    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20f23180-3073-4c3c-9bfe-08b159f4a2f3-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcp28\" (UID: \"20f23180-3073-4c3c-9bfe-08b159f4a2f3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcp28"
Sep 18 19:45:54.390549 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.390529    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77m28z"]
Sep 18 19:45:54.395473 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.395448    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qssb\" (UniqueName: \"kubernetes.io/projected/20f23180-3073-4c3c-9bfe-08b159f4a2f3-kube-api-access-5qssb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcp28\" (UID: \"20f23180-3073-4c3c-9bfe-08b159f4a2f3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcp28"
Sep 18 19:45:54.480205 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.480138    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62fxzh"]
Sep 18 19:45:54.483900 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.483886    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62fxzh"
Sep 18 19:45:54.489488 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.489417    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tflp\" (UniqueName: \"kubernetes.io/projected/793df892-fc17-4f4d-8358-a8d776c491c0-kube-api-access-8tflp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77m28z\" (UID: \"793df892-fc17-4f4d-8358-a8d776c491c0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77m28z"
Sep 18 19:45:54.489666 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.489624    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0389a73-9387-4217-b574-c103a81a5093-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cctlsp\" (UID: \"b0389a73-9387-4217-b574-c103a81a5093\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cctlsp"
Sep 18 19:45:54.489666 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.489669    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/793df892-fc17-4f4d-8358-a8d776c491c0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77m28z\" (UID: \"793df892-fc17-4f4d-8358-a8d776c491c0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77m28z"
Sep 18 19:45:54.489817 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.489710    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjczt\" (UniqueName: \"kubernetes.io/projected/b0389a73-9387-4217-b574-c103a81a5093-kube-api-access-zjczt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cctlsp\" (UID: \"b0389a73-9387-4217-b574-c103a81a5093\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cctlsp"
Sep 18 19:45:54.489817 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.489749    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0389a73-9387-4217-b574-c103a81a5093-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cctlsp\" (UID: \"b0389a73-9387-4217-b574-c103a81a5093\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cctlsp"
Sep 18 19:45:54.489817 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.489774    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/793df892-fc17-4f4d-8358-a8d776c491c0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77m28z\" (UID: \"793df892-fc17-4f4d-8358-a8d776c491c0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77m28z"
Sep 18 19:45:54.490093 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.490048    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0389a73-9387-4217-b574-c103a81a5093-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cctlsp\" (UID: \"b0389a73-9387-4217-b574-c103a81a5093\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cctlsp"
Sep 18 19:45:54.490093 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.490082    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0389a73-9387-4217-b574-c103a81a5093-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cctlsp\" (UID: \"b0389a73-9387-4217-b574-c103a81a5093\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cctlsp"
Sep 18 19:45:54.492573 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.492554    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62fxzh"]
Sep 18 19:45:54.494248 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.494225    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcp28"
Sep 18 19:45:54.497953 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.497935    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjczt\" (UniqueName: \"kubernetes.io/projected/b0389a73-9387-4217-b574-c103a81a5093-kube-api-access-zjczt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cctlsp\" (UID: \"b0389a73-9387-4217-b574-c103a81a5093\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cctlsp"
Sep 18 19:45:54.590710 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.590685    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/793df892-fc17-4f4d-8358-a8d776c491c0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77m28z\" (UID: \"793df892-fc17-4f4d-8358-a8d776c491c0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77m28z"
Sep 18 19:45:54.590893 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.590741    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ddd7ee4-15e9-4a70-beab-9e9de655c30f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62fxzh\" (UID: \"3ddd7ee4-15e9-4a70-beab-9e9de655c30f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62fxzh"
Sep 18 19:45:54.590893 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.590771    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zchd6\" (UniqueName: \"kubernetes.io/projected/3ddd7ee4-15e9-4a70-beab-9e9de655c30f-kube-api-access-zchd6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62fxzh\" (UID: \"3ddd7ee4-15e9-4a70-beab-9e9de655c30f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62fxzh"
Sep 18 19:45:54.590893 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.590797    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/793df892-fc17-4f4d-8358-a8d776c491c0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77m28z\" (UID: \"793df892-fc17-4f4d-8358-a8d776c491c0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77m28z"
Sep 18 19:45:54.590893 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.590861    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tflp\" (UniqueName: \"kubernetes.io/projected/793df892-fc17-4f4d-8358-a8d776c491c0-kube-api-access-8tflp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77m28z\" (UID: \"793df892-fc17-4f4d-8358-a8d776c491c0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77m28z"
Sep 18 19:45:54.591170 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.590958    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ddd7ee4-15e9-4a70-beab-9e9de655c30f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62fxzh\" (UID: \"3ddd7ee4-15e9-4a70-beab-9e9de655c30f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62fxzh"
Sep 18 19:45:54.591170 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.591123    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/793df892-fc17-4f4d-8358-a8d776c491c0-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77m28z\" (UID: \"793df892-fc17-4f4d-8358-a8d776c491c0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77m28z"
Sep 18 19:45:54.591268 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.591187    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/793df892-fc17-4f4d-8358-a8d776c491c0-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77m28z\" (UID: \"793df892-fc17-4f4d-8358-a8d776c491c0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77m28z"
Sep 18 19:45:54.596904 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.596880    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cctlsp"
Sep 18 19:45:54.599000 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.598980    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tflp\" (UniqueName: \"kubernetes.io/projected/793df892-fc17-4f4d-8358-a8d776c491c0-kube-api-access-8tflp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77m28z\" (UID: \"793df892-fc17-4f4d-8358-a8d776c491c0\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77m28z"
Sep 18 19:45:54.691054 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.691023    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77m28z"
Sep 18 19:45:54.692394 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.692369    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ddd7ee4-15e9-4a70-beab-9e9de655c30f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62fxzh\" (UID: \"3ddd7ee4-15e9-4a70-beab-9e9de655c30f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62fxzh"
Sep 18 19:45:54.692501 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.692406    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zchd6\" (UniqueName: \"kubernetes.io/projected/3ddd7ee4-15e9-4a70-beab-9e9de655c30f-kube-api-access-zchd6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62fxzh\" (UID: \"3ddd7ee4-15e9-4a70-beab-9e9de655c30f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62fxzh"
Sep 18 19:45:54.692501 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.692490    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ddd7ee4-15e9-4a70-beab-9e9de655c30f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62fxzh\" (UID: \"3ddd7ee4-15e9-4a70-beab-9e9de655c30f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62fxzh"
Sep 18 19:45:54.692818 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.692795    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ddd7ee4-15e9-4a70-beab-9e9de655c30f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62fxzh\" (UID: \"3ddd7ee4-15e9-4a70-beab-9e9de655c30f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62fxzh"
Sep 18 19:45:54.692888 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.692810    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ddd7ee4-15e9-4a70-beab-9e9de655c30f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62fxzh\" (UID: \"3ddd7ee4-15e9-4a70-beab-9e9de655c30f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62fxzh"
Sep 18 19:45:54.700960 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.700943    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zchd6\" (UniqueName: \"kubernetes.io/projected/3ddd7ee4-15e9-4a70-beab-9e9de655c30f-kube-api-access-zchd6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62fxzh\" (UID: \"3ddd7ee4-15e9-4a70-beab-9e9de655c30f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62fxzh"
Sep 18 19:45:54.768247 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.768212    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcp28"]
Sep 18 19:45:54.769734 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:45:54.769710    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f23180_3073_4c3c_9bfe_08b159f4a2f3.slice/crio-257c8911a797b0ddace3831e49710feb62a0cf392a8d5c22a33431e66283cd13 WatchSource:0}: Error finding container 257c8911a797b0ddace3831e49710feb62a0cf392a8d5c22a33431e66283cd13: Status 404 returned error can't find the container with id 257c8911a797b0ddace3831e49710feb62a0cf392a8d5c22a33431e66283cd13
Sep 18 19:45:54.793477 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.793450    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62fxzh"
Sep 18 19:45:54.872297 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.872213    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cctlsp"]
Sep 18 19:45:54.874332 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:45:54.874291    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0389a73_9387_4217_b574_c103a81a5093.slice/crio-5ffd6b03f7c2b272f258ee3629d0e69c06d428c29e6ec18731f444d1245b97ea WatchSource:0}: Error finding container 5ffd6b03f7c2b272f258ee3629d0e69c06d428c29e6ec18731f444d1245b97ea: Status 404 returned error can't find the container with id 5ffd6b03f7c2b272f258ee3629d0e69c06d428c29e6ec18731f444d1245b97ea
Sep 18 19:45:54.961758 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:54.961733    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77m28z"]
Sep 18 19:45:54.963488 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:45:54.963458    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod793df892_fc17_4f4d_8358_a8d776c491c0.slice/crio-16579022dcf0521c88cdfa6a4d998423aa9019719be87bbbc847a433811ef202 WatchSource:0}: Error finding container 16579022dcf0521c88cdfa6a4d998423aa9019719be87bbbc847a433811ef202: Status 404 returned error can't find the container with id 16579022dcf0521c88cdfa6a4d998423aa9019719be87bbbc847a433811ef202
Sep 18 19:45:55.071572 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:55.071546    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62fxzh"]
Sep 18 19:45:55.090624 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:45:55.090596    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ddd7ee4_15e9_4a70_beab_9e9de655c30f.slice/crio-cd8cc5c340321d3df29b1ff9c07069207c13204bda3a45082fdd8742546cb591 WatchSource:0}: Error finding container cd8cc5c340321d3df29b1ff9c07069207c13204bda3a45082fdd8742546cb591: Status 404 returned error can't find the container with id cd8cc5c340321d3df29b1ff9c07069207c13204bda3a45082fdd8742546cb591
Sep 18 19:45:55.587734 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:55.587700    2581 generic.go:356] "Generic (PLEG): container finished" podID="3ddd7ee4-15e9-4a70-beab-9e9de655c30f" containerID="3318c7ad2d1cdf85bd3ec221e52b1a5849af7eca3f381a4d6cf1fa7cce5deef8" exitCode=0
Sep 18 19:45:55.588200 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:55.587783    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62fxzh" event={"ID":"3ddd7ee4-15e9-4a70-beab-9e9de655c30f","Type":"ContainerDied","Data":"3318c7ad2d1cdf85bd3ec221e52b1a5849af7eca3f381a4d6cf1fa7cce5deef8"}
Sep 18 19:45:55.588200 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:55.587862    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62fxzh" event={"ID":"3ddd7ee4-15e9-4a70-beab-9e9de655c30f","Type":"ContainerStarted","Data":"cd8cc5c340321d3df29b1ff9c07069207c13204bda3a45082fdd8742546cb591"}
Sep 18 19:45:55.589475 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:55.589447    2581 generic.go:356] "Generic (PLEG): container finished" podID="b0389a73-9387-4217-b574-c103a81a5093" containerID="ac7f01156e4e02ed5a24048ec673a1ed98d6665f5128be12fc2d043a94810ff3" exitCode=0
Sep 18 19:45:55.589579 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:55.589496    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cctlsp" event={"ID":"b0389a73-9387-4217-b574-c103a81a5093","Type":"ContainerDied","Data":"ac7f01156e4e02ed5a24048ec673a1ed98d6665f5128be12fc2d043a94810ff3"}
Sep 18 19:45:55.589579 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:55.589538    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cctlsp" event={"ID":"b0389a73-9387-4217-b574-c103a81a5093","Type":"ContainerStarted","Data":"5ffd6b03f7c2b272f258ee3629d0e69c06d428c29e6ec18731f444d1245b97ea"}
Sep 18 19:45:55.591713 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:55.591623    2581 generic.go:356] "Generic (PLEG): container finished" podID="20f23180-3073-4c3c-9bfe-08b159f4a2f3" containerID="edda7030aef72156eb2c4d4f8c85aa4f1f333e4fcf85fddf86299afa9e305e69" exitCode=0
Sep 18 19:45:55.591713 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:55.591669    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcp28" event={"ID":"20f23180-3073-4c3c-9bfe-08b159f4a2f3","Type":"ContainerDied","Data":"edda7030aef72156eb2c4d4f8c85aa4f1f333e4fcf85fddf86299afa9e305e69"}
Sep 18 19:45:55.591811 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:55.591732    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcp28" event={"ID":"20f23180-3073-4c3c-9bfe-08b159f4a2f3","Type":"ContainerStarted","Data":"257c8911a797b0ddace3831e49710feb62a0cf392a8d5c22a33431e66283cd13"}
Sep 18 19:45:55.593385 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:55.593365    2581 generic.go:356] "Generic (PLEG): container finished" podID="793df892-fc17-4f4d-8358-a8d776c491c0" containerID="1c135fc4b018955c0a7d7f7ed3c088b3b91fb4059fa6182e0b66392ca81fd335" exitCode=0
Sep 18 19:45:55.593474 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:55.593389    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77m28z" event={"ID":"793df892-fc17-4f4d-8358-a8d776c491c0","Type":"ContainerDied","Data":"1c135fc4b018955c0a7d7f7ed3c088b3b91fb4059fa6182e0b66392ca81fd335"}
Sep 18 19:45:55.593474 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:55.593410    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77m28z" event={"ID":"793df892-fc17-4f4d-8358-a8d776c491c0","Type":"ContainerStarted","Data":"16579022dcf0521c88cdfa6a4d998423aa9019719be87bbbc847a433811ef202"}
Sep 18 19:45:56.601113 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:56.601084    2581 generic.go:356] "Generic (PLEG): container finished" podID="3ddd7ee4-15e9-4a70-beab-9e9de655c30f" containerID="692bbebf00043eae75e7beb0a8898c56dc9d50c207ee5f5faf8b525bb01171ef" exitCode=0
Sep 18 19:45:56.601549 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:56.601170    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62fxzh" event={"ID":"3ddd7ee4-15e9-4a70-beab-9e9de655c30f","Type":"ContainerDied","Data":"692bbebf00043eae75e7beb0a8898c56dc9d50c207ee5f5faf8b525bb01171ef"}
Sep 18 19:45:56.603235 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:56.603217    2581 generic.go:356] "Generic (PLEG): container finished" podID="b0389a73-9387-4217-b574-c103a81a5093" containerID="f9d826d412b7af2e439a56ac02dfc458538a5f2daed1bdf6c012bbedd425ab89" exitCode=0
Sep 18 19:45:56.603319 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:56.603279    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cctlsp" event={"ID":"b0389a73-9387-4217-b574-c103a81a5093","Type":"ContainerDied","Data":"f9d826d412b7af2e439a56ac02dfc458538a5f2daed1bdf6c012bbedd425ab89"}
Sep 18 19:45:56.612039 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:56.612012    2581 generic.go:356] "Generic (PLEG): container finished" podID="20f23180-3073-4c3c-9bfe-08b159f4a2f3" containerID="b14eb9db66c9f81f8052de277ba0581b22845c40ccd5d41610218504c857a6b5" exitCode=0
Sep 18 19:45:56.612156 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:56.612102    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcp28" event={"ID":"20f23180-3073-4c3c-9bfe-08b159f4a2f3","Type":"ContainerDied","Data":"b14eb9db66c9f81f8052de277ba0581b22845c40ccd5d41610218504c857a6b5"}
Sep 18 19:45:56.614059 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:56.614041    2581 generic.go:356] "Generic (PLEG): container finished" podID="793df892-fc17-4f4d-8358-a8d776c491c0" containerID="0c1bf5a45d27dbfeda814eb8e15a2284d60b8ef4aa233c03aa5a7fec5eff50af" exitCode=0
Sep 18 19:45:56.614154 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:56.614117    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77m28z" event={"ID":"793df892-fc17-4f4d-8358-a8d776c491c0","Type":"ContainerDied","Data":"0c1bf5a45d27dbfeda814eb8e15a2284d60b8ef4aa233c03aa5a7fec5eff50af"}
Sep 18 19:45:57.621431 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:57.621403    2581 generic.go:356] "Generic (PLEG): container finished" podID="3ddd7ee4-15e9-4a70-beab-9e9de655c30f" containerID="698762aac572b1e3202b043e33096a5931539aff970080462979c58d57210de6" exitCode=0
Sep 18 19:45:57.621922 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:57.621472    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62fxzh" event={"ID":"3ddd7ee4-15e9-4a70-beab-9e9de655c30f","Type":"ContainerDied","Data":"698762aac572b1e3202b043e33096a5931539aff970080462979c58d57210de6"}
Sep 18 19:45:57.623572 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:57.623550    2581 generic.go:356] "Generic (PLEG): container finished" podID="b0389a73-9387-4217-b574-c103a81a5093" containerID="4355b984da92afaef63b4de2a196bc7ada4d7942b5b05466a5b48821eccfe9fa" exitCode=0
Sep 18 19:45:57.623687 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:57.623601    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cctlsp" event={"ID":"b0389a73-9387-4217-b574-c103a81a5093","Type":"ContainerDied","Data":"4355b984da92afaef63b4de2a196bc7ada4d7942b5b05466a5b48821eccfe9fa"}
Sep 18 19:45:57.625554 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:57.625535    2581 generic.go:356] "Generic (PLEG): container finished" podID="20f23180-3073-4c3c-9bfe-08b159f4a2f3" containerID="f7f4d1a19c5eed4b47e403d8e3c3c7614518e655cd9228d9d9c9a556eac71664" exitCode=0
Sep 18 19:45:57.625640 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:57.625610    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcp28" event={"ID":"20f23180-3073-4c3c-9bfe-08b159f4a2f3","Type":"ContainerDied","Data":"f7f4d1a19c5eed4b47e403d8e3c3c7614518e655cd9228d9d9c9a556eac71664"}
Sep 18 19:45:57.627731 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:57.627712    2581 generic.go:356] "Generic (PLEG): container finished" podID="793df892-fc17-4f4d-8358-a8d776c491c0" containerID="53310126439cf7aee18b50215ac95f0efb5d8002aa116f4dbb6cdd143f677f97" exitCode=0
Sep 18 19:45:57.627814 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:57.627784    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77m28z" event={"ID":"793df892-fc17-4f4d-8358-a8d776c491c0","Type":"ContainerDied","Data":"53310126439cf7aee18b50215ac95f0efb5d8002aa116f4dbb6cdd143f677f97"}
Sep 18 19:45:58.556325 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:58.556297    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-d69b774b8-h2kfq"
Sep 18 19:45:58.791636 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:58.791459    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77m28z"
Sep 18 19:45:58.827471 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:58.827398    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62fxzh"
Sep 18 19:45:58.856557 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:58.856531    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cctlsp"
Sep 18 19:45:58.860152 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:58.860131    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcp28"
Sep 18 19:45:58.932562 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:58.932531    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ddd7ee4-15e9-4a70-beab-9e9de655c30f-bundle\") pod \"3ddd7ee4-15e9-4a70-beab-9e9de655c30f\" (UID: \"3ddd7ee4-15e9-4a70-beab-9e9de655c30f\") "
Sep 18 19:45:58.932801 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:58.932581    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/793df892-fc17-4f4d-8358-a8d776c491c0-bundle\") pod \"793df892-fc17-4f4d-8358-a8d776c491c0\" (UID: \"793df892-fc17-4f4d-8358-a8d776c491c0\") "
Sep 18 19:45:58.932801 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:58.932603    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/793df892-fc17-4f4d-8358-a8d776c491c0-util\") pod \"793df892-fc17-4f4d-8358-a8d776c491c0\" (UID: \"793df892-fc17-4f4d-8358-a8d776c491c0\") "
Sep 18 19:45:58.932801 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:58.932620    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tflp\" (UniqueName: \"kubernetes.io/projected/793df892-fc17-4f4d-8358-a8d776c491c0-kube-api-access-8tflp\") pod \"793df892-fc17-4f4d-8358-a8d776c491c0\" (UID: \"793df892-fc17-4f4d-8358-a8d776c491c0\") "
Sep 18 19:45:58.932801 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:58.932651    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zchd6\" (UniqueName: \"kubernetes.io/projected/3ddd7ee4-15e9-4a70-beab-9e9de655c30f-kube-api-access-zchd6\") pod \"3ddd7ee4-15e9-4a70-beab-9e9de655c30f\" (UID: \"3ddd7ee4-15e9-4a70-beab-9e9de655c30f\") "
Sep 18 19:45:58.932801 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:58.932687    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ddd7ee4-15e9-4a70-beab-9e9de655c30f-util\") pod \"3ddd7ee4-15e9-4a70-beab-9e9de655c30f\" (UID: \"3ddd7ee4-15e9-4a70-beab-9e9de655c30f\") "
Sep 18 19:45:58.933194 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:58.933169    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ddd7ee4-15e9-4a70-beab-9e9de655c30f-bundle" (OuterVolumeSpecName: "bundle") pod "3ddd7ee4-15e9-4a70-beab-9e9de655c30f" (UID: "3ddd7ee4-15e9-4a70-beab-9e9de655c30f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:45:58.933269 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:58.933206    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793df892-fc17-4f4d-8358-a8d776c491c0-bundle" (OuterVolumeSpecName: "bundle") pod "793df892-fc17-4f4d-8358-a8d776c491c0" (UID: "793df892-fc17-4f4d-8358-a8d776c491c0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:45:58.934899 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:58.934872    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ddd7ee4-15e9-4a70-beab-9e9de655c30f-kube-api-access-zchd6" (OuterVolumeSpecName: "kube-api-access-zchd6") pod "3ddd7ee4-15e9-4a70-beab-9e9de655c30f" (UID: "3ddd7ee4-15e9-4a70-beab-9e9de655c30f"). InnerVolumeSpecName "kube-api-access-zchd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:45:58.935172 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:58.935150    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/793df892-fc17-4f4d-8358-a8d776c491c0-kube-api-access-8tflp" (OuterVolumeSpecName: "kube-api-access-8tflp") pod "793df892-fc17-4f4d-8358-a8d776c491c0" (UID: "793df892-fc17-4f4d-8358-a8d776c491c0"). InnerVolumeSpecName "kube-api-access-8tflp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:45:58.938168 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:58.938151    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793df892-fc17-4f4d-8358-a8d776c491c0-util" (OuterVolumeSpecName: "util") pod "793df892-fc17-4f4d-8358-a8d776c491c0" (UID: "793df892-fc17-4f4d-8358-a8d776c491c0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:45:58.938902 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:58.938879    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ddd7ee4-15e9-4a70-beab-9e9de655c30f-util" (OuterVolumeSpecName: "util") pod "3ddd7ee4-15e9-4a70-beab-9e9de655c30f" (UID: "3ddd7ee4-15e9-4a70-beab-9e9de655c30f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:45:59.034200 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.034168    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20f23180-3073-4c3c-9bfe-08b159f4a2f3-bundle\") pod \"20f23180-3073-4c3c-9bfe-08b159f4a2f3\" (UID: \"20f23180-3073-4c3c-9bfe-08b159f4a2f3\") "
Sep 18 19:45:59.034354 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.034225    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qssb\" (UniqueName: \"kubernetes.io/projected/20f23180-3073-4c3c-9bfe-08b159f4a2f3-kube-api-access-5qssb\") pod \"20f23180-3073-4c3c-9bfe-08b159f4a2f3\" (UID: \"20f23180-3073-4c3c-9bfe-08b159f4a2f3\") "
Sep 18 19:45:59.034354 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.034263    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20f23180-3073-4c3c-9bfe-08b159f4a2f3-util\") pod \"20f23180-3073-4c3c-9bfe-08b159f4a2f3\" (UID: \"20f23180-3073-4c3c-9bfe-08b159f4a2f3\") "
Sep 18 19:45:59.034354 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.034285    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0389a73-9387-4217-b574-c103a81a5093-bundle\") pod \"b0389a73-9387-4217-b574-c103a81a5093\" (UID: \"b0389a73-9387-4217-b574-c103a81a5093\") "
Sep 18 19:45:59.034354 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.034323    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0389a73-9387-4217-b574-c103a81a5093-util\") pod \"b0389a73-9387-4217-b574-c103a81a5093\" (UID: \"b0389a73-9387-4217-b574-c103a81a5093\") "
Sep 18 19:45:59.034354 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.034339    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjczt\" (UniqueName: \"kubernetes.io/projected/b0389a73-9387-4217-b574-c103a81a5093-kube-api-access-zjczt\") pod \"b0389a73-9387-4217-b574-c103a81a5093\" (UID: \"b0389a73-9387-4217-b574-c103a81a5093\") "
Sep 18 19:45:59.034606 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.034532    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ddd7ee4-15e9-4a70-beab-9e9de655c30f-bundle\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:45:59.034606 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.034549    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/793df892-fc17-4f4d-8358-a8d776c491c0-bundle\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:45:59.034606 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.034562    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/793df892-fc17-4f4d-8358-a8d776c491c0-util\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:45:59.034606 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.034574    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tflp\" (UniqueName: \"kubernetes.io/projected/793df892-fc17-4f4d-8358-a8d776c491c0-kube-api-access-8tflp\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:45:59.034606 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.034591    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zchd6\" (UniqueName: \"kubernetes.io/projected/3ddd7ee4-15e9-4a70-beab-9e9de655c30f-kube-api-access-zchd6\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:45:59.034606 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.034603    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ddd7ee4-15e9-4a70-beab-9e9de655c30f-util\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:45:59.035099 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.035042    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f23180-3073-4c3c-9bfe-08b159f4a2f3-bundle" (OuterVolumeSpecName: "bundle") pod "20f23180-3073-4c3c-9bfe-08b159f4a2f3" (UID: "20f23180-3073-4c3c-9bfe-08b159f4a2f3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:45:59.035359 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.035332    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0389a73-9387-4217-b574-c103a81a5093-bundle" (OuterVolumeSpecName: "bundle") pod "b0389a73-9387-4217-b574-c103a81a5093" (UID: "b0389a73-9387-4217-b574-c103a81a5093"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:45:59.036649 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.036617    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20f23180-3073-4c3c-9bfe-08b159f4a2f3-kube-api-access-5qssb" (OuterVolumeSpecName: "kube-api-access-5qssb") pod "20f23180-3073-4c3c-9bfe-08b159f4a2f3" (UID: "20f23180-3073-4c3c-9bfe-08b159f4a2f3"). InnerVolumeSpecName "kube-api-access-5qssb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:45:59.036742 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.036644    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0389a73-9387-4217-b574-c103a81a5093-kube-api-access-zjczt" (OuterVolumeSpecName: "kube-api-access-zjczt") pod "b0389a73-9387-4217-b574-c103a81a5093" (UID: "b0389a73-9387-4217-b574-c103a81a5093"). InnerVolumeSpecName "kube-api-access-zjczt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:45:59.043449 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.043410    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0389a73-9387-4217-b574-c103a81a5093-util" (OuterVolumeSpecName: "util") pod "b0389a73-9387-4217-b574-c103a81a5093" (UID: "b0389a73-9387-4217-b574-c103a81a5093"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:45:59.043538 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.043522    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f23180-3073-4c3c-9bfe-08b159f4a2f3-util" (OuterVolumeSpecName: "util") pod "20f23180-3073-4c3c-9bfe-08b159f4a2f3" (UID: "20f23180-3073-4c3c-9bfe-08b159f4a2f3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:45:59.135198 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.135120    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20f23180-3073-4c3c-9bfe-08b159f4a2f3-bundle\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:45:59.135198 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.135147    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qssb\" (UniqueName: \"kubernetes.io/projected/20f23180-3073-4c3c-9bfe-08b159f4a2f3-kube-api-access-5qssb\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:45:59.135198 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.135159    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20f23180-3073-4c3c-9bfe-08b159f4a2f3-util\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:45:59.135198 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.135168    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0389a73-9387-4217-b574-c103a81a5093-bundle\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:45:59.135198 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.135176    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0389a73-9387-4217-b574-c103a81a5093-util\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:45:59.135198 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.135184    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjczt\" (UniqueName: \"kubernetes.io/projected/b0389a73-9387-4217-b574-c103a81a5093-kube-api-access-zjczt\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:45:59.640600 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.640572    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62fxzh"
Sep 18 19:45:59.640749 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.640570    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62fxzh" event={"ID":"3ddd7ee4-15e9-4a70-beab-9e9de655c30f","Type":"ContainerDied","Data":"cd8cc5c340321d3df29b1ff9c07069207c13204bda3a45082fdd8742546cb591"}
Sep 18 19:45:59.640749 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.640697    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd8cc5c340321d3df29b1ff9c07069207c13204bda3a45082fdd8742546cb591"
Sep 18 19:45:59.642588 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.642570    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cctlsp"
Sep 18 19:45:59.642875 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.642582    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cctlsp" event={"ID":"b0389a73-9387-4217-b574-c103a81a5093","Type":"ContainerDied","Data":"5ffd6b03f7c2b272f258ee3629d0e69c06d428c29e6ec18731f444d1245b97ea"}
Sep 18 19:45:59.642875 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.642623    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ffd6b03f7c2b272f258ee3629d0e69c06d428c29e6ec18731f444d1245b97ea"
Sep 18 19:45:59.644866 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.644828    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcp28"
Sep 18 19:45:59.644968 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.644861    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcp28" event={"ID":"20f23180-3073-4c3c-9bfe-08b159f4a2f3","Type":"ContainerDied","Data":"257c8911a797b0ddace3831e49710feb62a0cf392a8d5c22a33431e66283cd13"}
Sep 18 19:45:59.644968 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.644901    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="257c8911a797b0ddace3831e49710feb62a0cf392a8d5c22a33431e66283cd13"
Sep 18 19:45:59.646919 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.646878    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77m28z" event={"ID":"793df892-fc17-4f4d-8358-a8d776c491c0","Type":"ContainerDied","Data":"16579022dcf0521c88cdfa6a4d998423aa9019719be87bbbc847a433811ef202"}
Sep 18 19:45:59.646919 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.646916    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d77m28z"
Sep 18 19:45:59.647078 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:45:59.646917    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16579022dcf0521c88cdfa6a4d998423aa9019719be87bbbc847a433811ef202"
Sep 18 19:46:20.273866 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.273822    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-bq46m"]
Sep 18 19:46:20.274347 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.274157    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0389a73-9387-4217-b574-c103a81a5093" containerName="util"
Sep 18 19:46:20.274347 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.274169    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0389a73-9387-4217-b574-c103a81a5093" containerName="util"
Sep 18 19:46:20.274347 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.274183    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ddd7ee4-15e9-4a70-beab-9e9de655c30f" containerName="extract"
Sep 18 19:46:20.274347 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.274189    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ddd7ee4-15e9-4a70-beab-9e9de655c30f" containerName="extract"
Sep 18 19:46:20.274347 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.274198    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="793df892-fc17-4f4d-8358-a8d776c491c0" containerName="extract"
Sep 18 19:46:20.274347 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.274203    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="793df892-fc17-4f4d-8358-a8d776c491c0" containerName="extract"
Sep 18 19:46:20.274347 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.274212    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ddd7ee4-15e9-4a70-beab-9e9de655c30f" containerName="pull"
Sep 18 19:46:20.274347 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.274217    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ddd7ee4-15e9-4a70-beab-9e9de655c30f" containerName="pull"
Sep 18 19:46:20.274347 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.274221    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="793df892-fc17-4f4d-8358-a8d776c491c0" containerName="pull"
Sep 18 19:46:20.274347 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.274226    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="793df892-fc17-4f4d-8358-a8d776c491c0" containerName="pull"
Sep 18 19:46:20.274347 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.274231    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0389a73-9387-4217-b574-c103a81a5093" containerName="pull"
Sep 18 19:46:20.274347 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.274235    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0389a73-9387-4217-b574-c103a81a5093" containerName="pull"
Sep 18 19:46:20.274347 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.274242    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ddd7ee4-15e9-4a70-beab-9e9de655c30f" containerName="util"
Sep 18 19:46:20.274347 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.274247    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ddd7ee4-15e9-4a70-beab-9e9de655c30f" containerName="util"
Sep 18 19:46:20.274347 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.274256    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0389a73-9387-4217-b574-c103a81a5093" containerName="extract"
Sep 18 19:46:20.274347 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.274260    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0389a73-9387-4217-b574-c103a81a5093" containerName="extract"
Sep 18 19:46:20.274347 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.274266    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20f23180-3073-4c3c-9bfe-08b159f4a2f3" containerName="extract"
Sep 18 19:46:20.274347 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.274271    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="20f23180-3073-4c3c-9bfe-08b159f4a2f3" containerName="extract"
Sep 18 19:46:20.274347 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.274279    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20f23180-3073-4c3c-9bfe-08b159f4a2f3" containerName="pull"
Sep 18 19:46:20.274347 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.274284    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="20f23180-3073-4c3c-9bfe-08b159f4a2f3" containerName="pull"
Sep 18 19:46:20.274347 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.274290    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="793df892-fc17-4f4d-8358-a8d776c491c0" containerName="util"
Sep 18 19:46:20.274347 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.274294    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="793df892-fc17-4f4d-8358-a8d776c491c0" containerName="util"
Sep 18 19:46:20.274347 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.274302    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20f23180-3073-4c3c-9bfe-08b159f4a2f3" containerName="util"
Sep 18 19:46:20.274347 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.274307    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="20f23180-3073-4c3c-9bfe-08b159f4a2f3" containerName="util"
Sep 18 19:46:20.274347 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.274357    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0389a73-9387-4217-b574-c103a81a5093" containerName="extract"
Sep 18 19:46:20.275144 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.274366    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="20f23180-3073-4c3c-9bfe-08b159f4a2f3" containerName="extract"
Sep 18 19:46:20.275144 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.274374    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="793df892-fc17-4f4d-8358-a8d776c491c0" containerName="extract"
Sep 18 19:46:20.275144 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.274381    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ddd7ee4-15e9-4a70-beab-9e9de655c30f" containerName="extract"
Sep 18 19:46:20.277004 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.276990    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bq46m"
Sep 18 19:46:20.279234 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.279214    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 18 19:46:20.279557 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.279531    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 19:46:20.280417 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.280396    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jkpnw\""
Sep 18 19:46:20.280501 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.280401    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 19:46:20.280557 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.280401    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 18 19:46:20.286680 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.286663    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-bq46m"]
Sep 18 19:46:20.305763 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.305739    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwccm\" (UniqueName: \"kubernetes.io/projected/d8340c00-5917-4482-9af1-3085b2315c7d-kube-api-access-wwccm\") pod \"kuadrant-console-plugin-fbdd4f445-bq46m\" (UID: \"d8340c00-5917-4482-9af1-3085b2315c7d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bq46m"
Sep 18 19:46:20.305913 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.305789    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8340c00-5917-4482-9af1-3085b2315c7d-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-bq46m\" (UID: \"d8340c00-5917-4482-9af1-3085b2315c7d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bq46m"
Sep 18 19:46:20.305913 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.305811    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d8340c00-5917-4482-9af1-3085b2315c7d-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-bq46m\" (UID: \"d8340c00-5917-4482-9af1-3085b2315c7d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bq46m"
Sep 18 19:46:20.406422 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.406393    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwccm\" (UniqueName: \"kubernetes.io/projected/d8340c00-5917-4482-9af1-3085b2315c7d-kube-api-access-wwccm\") pod \"kuadrant-console-plugin-fbdd4f445-bq46m\" (UID: \"d8340c00-5917-4482-9af1-3085b2315c7d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bq46m"
Sep 18 19:46:20.406603 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.406443    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8340c00-5917-4482-9af1-3085b2315c7d-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-bq46m\" (UID: \"d8340c00-5917-4482-9af1-3085b2315c7d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bq46m"
Sep 18 19:46:20.406603 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.406468    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d8340c00-5917-4482-9af1-3085b2315c7d-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-bq46m\" (UID: \"d8340c00-5917-4482-9af1-3085b2315c7d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bq46m"
Sep 18 19:46:20.407236 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.407215    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d8340c00-5917-4482-9af1-3085b2315c7d-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-bq46m\" (UID: \"d8340c00-5917-4482-9af1-3085b2315c7d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bq46m"
Sep 18 19:46:20.409084 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.409055    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8340c00-5917-4482-9af1-3085b2315c7d-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-bq46m\" (UID: \"d8340c00-5917-4482-9af1-3085b2315c7d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bq46m"
Sep 18 19:46:20.422819 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.422790    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwccm\" (UniqueName: \"kubernetes.io/projected/d8340c00-5917-4482-9af1-3085b2315c7d-kube-api-access-wwccm\") pod \"kuadrant-console-plugin-fbdd4f445-bq46m\" (UID: \"d8340c00-5917-4482-9af1-3085b2315c7d\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bq46m"
Sep 18 19:46:20.587387 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.587288    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bq46m"
Sep 18 19:46:20.868714 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:20.868688    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-bq46m"]
Sep 18 19:46:21.759812 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:21.759777    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bq46m" event={"ID":"d8340c00-5917-4482-9af1-3085b2315c7d","Type":"ContainerStarted","Data":"91c20fc16e34c182801523598f4e178b349232ccef2a4040db6d69d5bff5436d"}
Sep 18 19:46:25.785950 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:25.785920    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bq46m" event={"ID":"d8340c00-5917-4482-9af1-3085b2315c7d","Type":"ContainerStarted","Data":"0e317971d93bf579fbc0e648104ba29f06e98cb07469643f88c8980fcd967361"}
Sep 18 19:46:25.800458 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:46:25.800418    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bq46m" podStartSLOduration=1.033784234 podStartE2EDuration="5.800405397s" totalImagesPullingTime="4.766621163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:46:25.799938515 +0000 UTC m=+508.902226843" watchObservedRunningTime="2026-09-18 19:46:25.800405397 +0000 UTC m=+508.902693727"
Sep 18 19:47:04.386443 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:04.386403    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-xqcjv"]
Sep 18 19:47:04.406900 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:04.406870    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-xqcjv"]
Sep 18 19:47:04.407060 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:04.406931    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-xqcjv"
Sep 18 19:47:04.409376 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:04.409349    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9twvg\""
Sep 18 19:47:04.582188 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:04.582152    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv6bn\" (UniqueName: \"kubernetes.io/projected/9a05b09b-15ae-4a1b-ba6f-6bb9ba20eec9-kube-api-access-jv6bn\") pod \"authorino-7bcfd8f9cd-xqcjv\" (UID: \"9a05b09b-15ae-4a1b-ba6f-6bb9ba20eec9\") " pod="kuadrant-system/authorino-7bcfd8f9cd-xqcjv"
Sep 18 19:47:04.683684 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:04.683604    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv6bn\" (UniqueName: \"kubernetes.io/projected/9a05b09b-15ae-4a1b-ba6f-6bb9ba20eec9-kube-api-access-jv6bn\") pod \"authorino-7bcfd8f9cd-xqcjv\" (UID: \"9a05b09b-15ae-4a1b-ba6f-6bb9ba20eec9\") " pod="kuadrant-system/authorino-7bcfd8f9cd-xqcjv"
Sep 18 19:47:04.696562 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:04.696530    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv6bn\" (UniqueName: \"kubernetes.io/projected/9a05b09b-15ae-4a1b-ba6f-6bb9ba20eec9-kube-api-access-jv6bn\") pod \"authorino-7bcfd8f9cd-xqcjv\" (UID: \"9a05b09b-15ae-4a1b-ba6f-6bb9ba20eec9\") " pod="kuadrant-system/authorino-7bcfd8f9cd-xqcjv"
Sep 18 19:47:04.715827 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:04.715798    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-xqcjv"
Sep 18 19:47:04.982804 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:04.982777    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-xqcjv"]
Sep 18 19:47:04.984230 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:47:04.984165    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a05b09b_15ae_4a1b_ba6f_6bb9ba20eec9.slice/crio-9e4d08ce4c2b88270ee28bc189af052d58b33ebff4978628302c3203b3547256 WatchSource:0}: Error finding container 9e4d08ce4c2b88270ee28bc189af052d58b33ebff4978628302c3203b3547256: Status 404 returned error can't find the container with id 9e4d08ce4c2b88270ee28bc189af052d58b33ebff4978628302c3203b3547256
Sep 18 19:47:05.993975 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:05.993929    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-xqcjv" event={"ID":"9a05b09b-15ae-4a1b-ba6f-6bb9ba20eec9","Type":"ContainerStarted","Data":"9e4d08ce4c2b88270ee28bc189af052d58b33ebff4978628302c3203b3547256"}
Sep 18 19:47:08.005747 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:08.005711    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-xqcjv" event={"ID":"9a05b09b-15ae-4a1b-ba6f-6bb9ba20eec9","Type":"ContainerStarted","Data":"a52f4558fddf7b19dc12516f57e6c27ef2dcf17e169a75d9add95bc60c1f9e72"}
Sep 18 19:47:08.020376 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:08.020329    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-xqcjv" podStartSLOduration=1.992912337 podStartE2EDuration="4.020314827s" totalImagesPullingTime="2.02740249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:47:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:47:08.019099213 +0000 UTC m=+551.121387541" watchObservedRunningTime="2026-09-18 19:47:08.020314827 +0000 UTC m=+551.122603154"
Sep 18 19:47:10.053184 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:10.053148    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-xqcjv"]
Sep 18 19:47:10.053713 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:10.053291    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-xqcjv" podUID="9a05b09b-15ae-4a1b-ba6f-6bb9ba20eec9" containerName="authorino" containerID="cri-o://a52f4558fddf7b19dc12516f57e6c27ef2dcf17e169a75d9add95bc60c1f9e72" gracePeriod=30
Sep 18 19:47:10.289598 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:10.289567    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-xqcjv"
Sep 18 19:47:10.324268 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:10.324204    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jv6bn\" (UniqueName: \"kubernetes.io/projected/9a05b09b-15ae-4a1b-ba6f-6bb9ba20eec9-kube-api-access-jv6bn\") pod \"9a05b09b-15ae-4a1b-ba6f-6bb9ba20eec9\" (UID: \"9a05b09b-15ae-4a1b-ba6f-6bb9ba20eec9\") "
Sep 18 19:47:10.326449 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:10.326424    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a05b09b-15ae-4a1b-ba6f-6bb9ba20eec9-kube-api-access-jv6bn" (OuterVolumeSpecName: "kube-api-access-jv6bn") pod "9a05b09b-15ae-4a1b-ba6f-6bb9ba20eec9" (UID: "9a05b09b-15ae-4a1b-ba6f-6bb9ba20eec9"). InnerVolumeSpecName "kube-api-access-jv6bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:47:10.425737 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:10.425712    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv6bn\" (UniqueName: \"kubernetes.io/projected/9a05b09b-15ae-4a1b-ba6f-6bb9ba20eec9-kube-api-access-jv6bn\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:47:11.028502 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:11.028469    2581 generic.go:356] "Generic (PLEG): container finished" podID="9a05b09b-15ae-4a1b-ba6f-6bb9ba20eec9" containerID="a52f4558fddf7b19dc12516f57e6c27ef2dcf17e169a75d9add95bc60c1f9e72" exitCode=0
Sep 18 19:47:11.028674 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:11.028508    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-xqcjv"
Sep 18 19:47:11.028674 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:11.028523    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-xqcjv" event={"ID":"9a05b09b-15ae-4a1b-ba6f-6bb9ba20eec9","Type":"ContainerDied","Data":"a52f4558fddf7b19dc12516f57e6c27ef2dcf17e169a75d9add95bc60c1f9e72"}
Sep 18 19:47:11.028674 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:11.028555    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-xqcjv" event={"ID":"9a05b09b-15ae-4a1b-ba6f-6bb9ba20eec9","Type":"ContainerDied","Data":"9e4d08ce4c2b88270ee28bc189af052d58b33ebff4978628302c3203b3547256"}
Sep 18 19:47:11.028674 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:11.028570    2581 scope.go:117] "RemoveContainer" containerID="a52f4558fddf7b19dc12516f57e6c27ef2dcf17e169a75d9add95bc60c1f9e72"
Sep 18 19:47:11.039760 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:11.039731    2581 scope.go:117] "RemoveContainer" containerID="a52f4558fddf7b19dc12516f57e6c27ef2dcf17e169a75d9add95bc60c1f9e72"
Sep 18 19:47:11.040318 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:47:11.040293    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a52f4558fddf7b19dc12516f57e6c27ef2dcf17e169a75d9add95bc60c1f9e72\": container with ID starting with a52f4558fddf7b19dc12516f57e6c27ef2dcf17e169a75d9add95bc60c1f9e72 not found: ID does not exist" containerID="a52f4558fddf7b19dc12516f57e6c27ef2dcf17e169a75d9add95bc60c1f9e72"
Sep 18 19:47:11.040408 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:11.040328    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a52f4558fddf7b19dc12516f57e6c27ef2dcf17e169a75d9add95bc60c1f9e72"} err="failed to get container status \"a52f4558fddf7b19dc12516f57e6c27ef2dcf17e169a75d9add95bc60c1f9e72\": rpc error: code = NotFound desc = could not find container \"a52f4558fddf7b19dc12516f57e6c27ef2dcf17e169a75d9add95bc60c1f9e72\": container with ID starting with a52f4558fddf7b19dc12516f57e6c27ef2dcf17e169a75d9add95bc60c1f9e72 not found: ID does not exist"
Sep 18 19:47:11.049847 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:11.049810    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-xqcjv"]
Sep 18 19:47:11.052854 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:11.052817    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-xqcjv"]
Sep 18 19:47:11.432805 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:11.432715    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a05b09b-15ae-4a1b-ba6f-6bb9ba20eec9" path="/var/lib/kubelet/pods/9a05b09b-15ae-4a1b-ba6f-6bb9ba20eec9/volumes"
Sep 18 19:47:25.279728 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:25.279693    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d9898bd99-gfm4b"]
Sep 18 19:47:25.280135 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:25.280046    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a05b09b-15ae-4a1b-ba6f-6bb9ba20eec9" containerName="authorino"
Sep 18 19:47:25.280135 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:25.280058    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a05b09b-15ae-4a1b-ba6f-6bb9ba20eec9" containerName="authorino"
Sep 18 19:47:25.280135 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:25.280130    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a05b09b-15ae-4a1b-ba6f-6bb9ba20eec9" containerName="authorino"
Sep 18 19:47:25.282951 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:25.282935    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-gfm4b"
Sep 18 19:47:25.286276 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:25.286247    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9twvg\""
Sep 18 19:47:25.286568 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:25.286545    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 18 19:47:25.288078 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:25.288056    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-gfm4b"]
Sep 18 19:47:25.335392 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:25.335364    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5a7f347f-5fd5-4b95-a61d-fac88872ac60-tls-cert\") pod \"authorino-6d9898bd99-gfm4b\" (UID: \"5a7f347f-5fd5-4b95-a61d-fac88872ac60\") " pod="kuadrant-system/authorino-6d9898bd99-gfm4b"
Sep 18 19:47:25.335537 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:25.335435    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6g28\" (UniqueName: \"kubernetes.io/projected/5a7f347f-5fd5-4b95-a61d-fac88872ac60-kube-api-access-j6g28\") pod \"authorino-6d9898bd99-gfm4b\" (UID: \"5a7f347f-5fd5-4b95-a61d-fac88872ac60\") " pod="kuadrant-system/authorino-6d9898bd99-gfm4b"
Sep 18 19:47:25.436574 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:25.436547    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6g28\" (UniqueName: \"kubernetes.io/projected/5a7f347f-5fd5-4b95-a61d-fac88872ac60-kube-api-access-j6g28\") pod \"authorino-6d9898bd99-gfm4b\" (UID: \"5a7f347f-5fd5-4b95-a61d-fac88872ac60\") " pod="kuadrant-system/authorino-6d9898bd99-gfm4b"
Sep 18 19:47:25.436732 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:25.436589    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5a7f347f-5fd5-4b95-a61d-fac88872ac60-tls-cert\") pod \"authorino-6d9898bd99-gfm4b\" (UID: \"5a7f347f-5fd5-4b95-a61d-fac88872ac60\") " pod="kuadrant-system/authorino-6d9898bd99-gfm4b"
Sep 18 19:47:25.439120 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:25.439098    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5a7f347f-5fd5-4b95-a61d-fac88872ac60-tls-cert\") pod \"authorino-6d9898bd99-gfm4b\" (UID: \"5a7f347f-5fd5-4b95-a61d-fac88872ac60\") " pod="kuadrant-system/authorino-6d9898bd99-gfm4b"
Sep 18 19:47:25.444583 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:25.444558    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6g28\" (UniqueName: \"kubernetes.io/projected/5a7f347f-5fd5-4b95-a61d-fac88872ac60-kube-api-access-j6g28\") pod \"authorino-6d9898bd99-gfm4b\" (UID: \"5a7f347f-5fd5-4b95-a61d-fac88872ac60\") " pod="kuadrant-system/authorino-6d9898bd99-gfm4b"
Sep 18 19:47:25.592857 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:25.592743    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-gfm4b"
Sep 18 19:47:25.862879 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:25.862855    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-gfm4b"]
Sep 18 19:47:25.863934 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:47:25.863908    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a7f347f_5fd5_4b95_a61d_fac88872ac60.slice/crio-33a701f5be93149005b77f14aa20fc8d5e12799d1d5b30924ee3c68813d93c95 WatchSource:0}: Error finding container 33a701f5be93149005b77f14aa20fc8d5e12799d1d5b30924ee3c68813d93c95: Status 404 returned error can't find the container with id 33a701f5be93149005b77f14aa20fc8d5e12799d1d5b30924ee3c68813d93c95
Sep 18 19:47:26.106632 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:26.106594    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-gfm4b" event={"ID":"5a7f347f-5fd5-4b95-a61d-fac88872ac60","Type":"ContainerStarted","Data":"33a701f5be93149005b77f14aa20fc8d5e12799d1d5b30924ee3c68813d93c95"}
Sep 18 19:47:27.112798 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:27.112769    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-gfm4b" event={"ID":"5a7f347f-5fd5-4b95-a61d-fac88872ac60","Type":"ContainerStarted","Data":"5b3373c9e3cccebb1bc9f273380bfe481464d661541cf3d1bcba63888ee6ba1e"}
Sep 18 19:47:27.128042 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:27.127996    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d9898bd99-gfm4b" podStartSLOduration=1.573963775 podStartE2EDuration="2.127979276s" totalImagesPullingTime="554.015501ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:47:27.126236657 +0000 UTC m=+570.228524984" watchObservedRunningTime="2026-09-18 19:47:27.127979276 +0000 UTC m=+570.230267603"
Sep 18 19:47:55.249985 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:55.249949    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-t2pj2"]
Sep 18 19:47:55.253399 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:55.253378    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-t2pj2"
Sep 18 19:47:55.255740 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:55.255720    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 19:47:55.255740 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:55.255738    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 18 19:47:55.255940 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:55.255725    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-n7prf\""
Sep 18 19:47:55.256640 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:55.256627    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 19:47:55.262656 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:55.262637    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-t2pj2"]
Sep 18 19:47:55.376876 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:55.376820    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/46b58988-c6e2-4517-94dd-231064e5bf82-data\") pod \"seaweedfs-64568bcd49-t2pj2\" (UID: \"46b58988-c6e2-4517-94dd-231064e5bf82\") " pod="kserve/seaweedfs-64568bcd49-t2pj2"
Sep 18 19:47:55.377037 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:55.376932    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lmkm\" (UniqueName: \"kubernetes.io/projected/46b58988-c6e2-4517-94dd-231064e5bf82-kube-api-access-7lmkm\") pod \"seaweedfs-64568bcd49-t2pj2\" (UID: \"46b58988-c6e2-4517-94dd-231064e5bf82\") " pod="kserve/seaweedfs-64568bcd49-t2pj2"
Sep 18 19:47:55.477706 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:55.477675    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lmkm\" (UniqueName: \"kubernetes.io/projected/46b58988-c6e2-4517-94dd-231064e5bf82-kube-api-access-7lmkm\") pod \"seaweedfs-64568bcd49-t2pj2\" (UID: \"46b58988-c6e2-4517-94dd-231064e5bf82\") " pod="kserve/seaweedfs-64568bcd49-t2pj2"
Sep 18 19:47:55.477918 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:55.477731    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/46b58988-c6e2-4517-94dd-231064e5bf82-data\") pod \"seaweedfs-64568bcd49-t2pj2\" (UID: \"46b58988-c6e2-4517-94dd-231064e5bf82\") " pod="kserve/seaweedfs-64568bcd49-t2pj2"
Sep 18 19:47:55.478059 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:55.478041    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/46b58988-c6e2-4517-94dd-231064e5bf82-data\") pod \"seaweedfs-64568bcd49-t2pj2\" (UID: \"46b58988-c6e2-4517-94dd-231064e5bf82\") " pod="kserve/seaweedfs-64568bcd49-t2pj2"
Sep 18 19:47:55.485943 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:55.485923    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lmkm\" (UniqueName: \"kubernetes.io/projected/46b58988-c6e2-4517-94dd-231064e5bf82-kube-api-access-7lmkm\") pod \"seaweedfs-64568bcd49-t2pj2\" (UID: \"46b58988-c6e2-4517-94dd-231064e5bf82\") " pod="kserve/seaweedfs-64568bcd49-t2pj2"
Sep 18 19:47:55.562932 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:55.562914    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-t2pj2"
Sep 18 19:47:55.843622 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:47:55.843588    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b58988_c6e2_4517_94dd_231064e5bf82.slice/crio-5621279a4a877b6f0ca20c4edf103bb1ce0c70670518e6efdb598270814807d0 WatchSource:0}: Error finding container 5621279a4a877b6f0ca20c4edf103bb1ce0c70670518e6efdb598270814807d0: Status 404 returned error can't find the container with id 5621279a4a877b6f0ca20c4edf103bb1ce0c70670518e6efdb598270814807d0
Sep 18 19:47:55.844284 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:55.844263    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-t2pj2"]
Sep 18 19:47:56.261008 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:56.260940    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-t2pj2" event={"ID":"46b58988-c6e2-4517-94dd-231064e5bf82","Type":"ContainerStarted","Data":"5621279a4a877b6f0ca20c4edf103bb1ce0c70670518e6efdb598270814807d0"}
Sep 18 19:47:58.600411 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:58.600389    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 18 19:47:59.278937 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:59.278904    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-t2pj2" event={"ID":"46b58988-c6e2-4517-94dd-231064e5bf82","Type":"ContainerStarted","Data":"39dd6282802c715cba4c32e5a5b657482b5dba1b571873eab4bb993e0ffd81df"}
Sep 18 19:47:59.279090 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:59.279033    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-t2pj2"
Sep 18 19:47:59.295865 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:47:59.295803    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-t2pj2" podStartSLOduration=1.542691063 podStartE2EDuration="4.295790414s" totalImagesPullingTime="2.753099351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:47:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:47:59.293274323 +0000 UTC m=+602.395562652" watchObservedRunningTime="2026-09-18 19:47:59.295790414 +0000 UTC m=+602.398078769"
Sep 18 19:48:05.286521 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:48:05.286490    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-t2pj2"
Sep 18 19:49:11.277862 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:49:11.277800    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-f2j8z"]
Sep 18 19:49:11.281195 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:49:11.281175    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-f2j8z"
Sep 18 19:49:11.283763 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:49:11.283744    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 18 19:49:11.283895 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:49:11.283788    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-7d5c9\""
Sep 18 19:49:11.291347 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:49:11.291324    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-f2j8z"]
Sep 18 19:49:11.317640 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:49:11.317618    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aa87bc1-647b-4681-b4d2-3e221d8d2967-tls-certs\") pod \"model-serving-api-7b954ffc95-f2j8z\" (UID: \"2aa87bc1-647b-4681-b4d2-3e221d8d2967\") " pod="kserve/model-serving-api-7b954ffc95-f2j8z"
Sep 18 19:49:11.317733 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:49:11.317669    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6lss\" (UniqueName: \"kubernetes.io/projected/2aa87bc1-647b-4681-b4d2-3e221d8d2967-kube-api-access-n6lss\") pod \"model-serving-api-7b954ffc95-f2j8z\" (UID: \"2aa87bc1-647b-4681-b4d2-3e221d8d2967\") " pod="kserve/model-serving-api-7b954ffc95-f2j8z"
Sep 18 19:49:11.418699 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:49:11.418656    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6lss\" (UniqueName: \"kubernetes.io/projected/2aa87bc1-647b-4681-b4d2-3e221d8d2967-kube-api-access-n6lss\") pod \"model-serving-api-7b954ffc95-f2j8z\" (UID: \"2aa87bc1-647b-4681-b4d2-3e221d8d2967\") " pod="kserve/model-serving-api-7b954ffc95-f2j8z"
Sep 18 19:49:11.418809 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:49:11.418727    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aa87bc1-647b-4681-b4d2-3e221d8d2967-tls-certs\") pod \"model-serving-api-7b954ffc95-f2j8z\" (UID: \"2aa87bc1-647b-4681-b4d2-3e221d8d2967\") " pod="kserve/model-serving-api-7b954ffc95-f2j8z"
Sep 18 19:49:11.418881 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:49:11.418814    2581 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 18 19:49:11.418918 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:49:11.418893    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aa87bc1-647b-4681-b4d2-3e221d8d2967-tls-certs podName:2aa87bc1-647b-4681-b4d2-3e221d8d2967 nodeName:}" failed. No retries permitted until 2026-09-18 19:49:11.9188776 +0000 UTC m=+675.021165906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2aa87bc1-647b-4681-b4d2-3e221d8d2967-tls-certs") pod "model-serving-api-7b954ffc95-f2j8z" (UID: "2aa87bc1-647b-4681-b4d2-3e221d8d2967") : secret "model-serving-api-tls" not found
Sep 18 19:49:11.429204 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:49:11.429184    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6lss\" (UniqueName: \"kubernetes.io/projected/2aa87bc1-647b-4681-b4d2-3e221d8d2967-kube-api-access-n6lss\") pod \"model-serving-api-7b954ffc95-f2j8z\" (UID: \"2aa87bc1-647b-4681-b4d2-3e221d8d2967\") " pod="kserve/model-serving-api-7b954ffc95-f2j8z"
Sep 18 19:49:11.923563 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:49:11.923524    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aa87bc1-647b-4681-b4d2-3e221d8d2967-tls-certs\") pod \"model-serving-api-7b954ffc95-f2j8z\" (UID: \"2aa87bc1-647b-4681-b4d2-3e221d8d2967\") " pod="kserve/model-serving-api-7b954ffc95-f2j8z"
Sep 18 19:49:11.926203 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:49:11.926177    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aa87bc1-647b-4681-b4d2-3e221d8d2967-tls-certs\") pod \"model-serving-api-7b954ffc95-f2j8z\" (UID: \"2aa87bc1-647b-4681-b4d2-3e221d8d2967\") " pod="kserve/model-serving-api-7b954ffc95-f2j8z"
Sep 18 19:49:12.192371 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:49:12.192294    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-f2j8z"
Sep 18 19:49:12.463867 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:49:12.463827    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-f2j8z"]
Sep 18 19:49:12.466305 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:49:12.466274    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aa87bc1_647b_4681_b4d2_3e221d8d2967.slice/crio-09d0965d43f508292b07daffcb0d6b29fabc93ebcaf4aa7edb3a39c709b85a7f WatchSource:0}: Error finding container 09d0965d43f508292b07daffcb0d6b29fabc93ebcaf4aa7edb3a39c709b85a7f: Status 404 returned error can't find the container with id 09d0965d43f508292b07daffcb0d6b29fabc93ebcaf4aa7edb3a39c709b85a7f
Sep 18 19:49:12.468219 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:49:12.468199    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:49:12.658950 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:49:12.658914    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-f2j8z" event={"ID":"2aa87bc1-647b-4681-b4d2-3e221d8d2967","Type":"ContainerStarted","Data":"09d0965d43f508292b07daffcb0d6b29fabc93ebcaf4aa7edb3a39c709b85a7f"}
Sep 18 19:49:15.680987 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:49:15.680949    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-f2j8z" event={"ID":"2aa87bc1-647b-4681-b4d2-3e221d8d2967","Type":"ContainerStarted","Data":"95d8ee3cfc6a93d2914748df9acf7fa104aa19b28602914d696971e227e08789"}
Sep 18 19:49:15.681402 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:49:15.681019    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-f2j8z"
Sep 18 19:49:15.697559 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:49:15.697516    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-f2j8z" podStartSLOduration=2.053954095 podStartE2EDuration="4.697501127s" totalImagesPullingTime="2.643547032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:49:15.696383087 +0000 UTC m=+678.798671415" watchObservedRunningTime="2026-09-18 19:49:15.697501127 +0000 UTC m=+678.799789446"
Sep 18 19:49:26.690696 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:49:26.690664    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-f2j8z"
Sep 18 19:51:30.511900 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:30.511871    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx"]
Sep 18 19:51:30.513957 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:30.513936    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx"
Sep 18 19:51:30.517449 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:30.517423    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 18 19:51:30.517573 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:30.517478    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 18 19:51:30.517573 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:30.517423    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 18 19:51:30.517573 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:30.517536    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-8h2g5\""
Sep 18 19:51:30.526000 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:30.525979    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx"]
Sep 18 19:51:30.611375 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:30.611348    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9479f11f-d9ba-4591-b9a9-22d232df00fe-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-wp2vx\" (UID: \"9479f11f-d9ba-4591-b9a9-22d232df00fe\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx"
Sep 18 19:51:30.611574 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:30.611388    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9479f11f-d9ba-4591-b9a9-22d232df00fe-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-wp2vx\" (UID: \"9479f11f-d9ba-4591-b9a9-22d232df00fe\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx"
Sep 18 19:51:30.611574 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:30.611423    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdntf\" (UniqueName: \"kubernetes.io/projected/9479f11f-d9ba-4591-b9a9-22d232df00fe-kube-api-access-fdntf\") pod \"fc-smoke-test-kserve-7f4687897-wp2vx\" (UID: \"9479f11f-d9ba-4591-b9a9-22d232df00fe\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx"
Sep 18 19:51:30.611574 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:30.611444    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9479f11f-d9ba-4591-b9a9-22d232df00fe-home\") pod \"fc-smoke-test-kserve-7f4687897-wp2vx\" (UID: \"9479f11f-d9ba-4591-b9a9-22d232df00fe\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx"
Sep 18 19:51:30.611574 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:30.611483    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9479f11f-d9ba-4591-b9a9-22d232df00fe-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-wp2vx\" (UID: \"9479f11f-d9ba-4591-b9a9-22d232df00fe\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx"
Sep 18 19:51:30.611574 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:30.611506    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9479f11f-d9ba-4591-b9a9-22d232df00fe-dshm\") pod \"fc-smoke-test-kserve-7f4687897-wp2vx\" (UID: \"9479f11f-d9ba-4591-b9a9-22d232df00fe\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx"
Sep 18 19:51:30.712223 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:30.712192    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9479f11f-d9ba-4591-b9a9-22d232df00fe-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-wp2vx\" (UID: \"9479f11f-d9ba-4591-b9a9-22d232df00fe\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx"
Sep 18 19:51:30.712223 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:30.712223    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9479f11f-d9ba-4591-b9a9-22d232df00fe-dshm\") pod \"fc-smoke-test-kserve-7f4687897-wp2vx\" (UID: \"9479f11f-d9ba-4591-b9a9-22d232df00fe\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx"
Sep 18 19:51:30.712420 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:30.712283    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9479f11f-d9ba-4591-b9a9-22d232df00fe-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-wp2vx\" (UID: \"9479f11f-d9ba-4591-b9a9-22d232df00fe\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx"
Sep 18 19:51:30.712420 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:30.712303    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9479f11f-d9ba-4591-b9a9-22d232df00fe-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-wp2vx\" (UID: \"9479f11f-d9ba-4591-b9a9-22d232df00fe\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx"
Sep 18 19:51:30.712420 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:30.712324    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdntf\" (UniqueName: \"kubernetes.io/projected/9479f11f-d9ba-4591-b9a9-22d232df00fe-kube-api-access-fdntf\") pod \"fc-smoke-test-kserve-7f4687897-wp2vx\" (UID: \"9479f11f-d9ba-4591-b9a9-22d232df00fe\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx"
Sep 18 19:51:30.712420 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:30.712338    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9479f11f-d9ba-4591-b9a9-22d232df00fe-home\") pod \"fc-smoke-test-kserve-7f4687897-wp2vx\" (UID: \"9479f11f-d9ba-4591-b9a9-22d232df00fe\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx"
Sep 18 19:51:30.712664 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:30.712640    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9479f11f-d9ba-4591-b9a9-22d232df00fe-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-wp2vx\" (UID: \"9479f11f-d9ba-4591-b9a9-22d232df00fe\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx"
Sep 18 19:51:30.712722 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:30.712652    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9479f11f-d9ba-4591-b9a9-22d232df00fe-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-wp2vx\" (UID: \"9479f11f-d9ba-4591-b9a9-22d232df00fe\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx"
Sep 18 19:51:30.712760 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:30.712732    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9479f11f-d9ba-4591-b9a9-22d232df00fe-home\") pod \"fc-smoke-test-kserve-7f4687897-wp2vx\" (UID: \"9479f11f-d9ba-4591-b9a9-22d232df00fe\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx"
Sep 18 19:51:30.714606 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:30.714584    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9479f11f-d9ba-4591-b9a9-22d232df00fe-dshm\") pod \"fc-smoke-test-kserve-7f4687897-wp2vx\" (UID: \"9479f11f-d9ba-4591-b9a9-22d232df00fe\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx"
Sep 18 19:51:30.715001 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:30.714981    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9479f11f-d9ba-4591-b9a9-22d232df00fe-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-wp2vx\" (UID: \"9479f11f-d9ba-4591-b9a9-22d232df00fe\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx"
Sep 18 19:51:30.721948 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:30.721924    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdntf\" (UniqueName: \"kubernetes.io/projected/9479f11f-d9ba-4591-b9a9-22d232df00fe-kube-api-access-fdntf\") pod \"fc-smoke-test-kserve-7f4687897-wp2vx\" (UID: \"9479f11f-d9ba-4591-b9a9-22d232df00fe\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx"
Sep 18 19:51:30.824064 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:30.824033    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx"
Sep 18 19:51:31.096949 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:31.096926    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx"]
Sep 18 19:51:31.098582 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:51:31.098554    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9479f11f_d9ba_4591_b9a9_22d232df00fe.slice/crio-ef7aeeba18ca311121615297ec119f522102abda02fec4811c9a3f794aa725d2 WatchSource:0}: Error finding container ef7aeeba18ca311121615297ec119f522102abda02fec4811c9a3f794aa725d2: Status 404 returned error can't find the container with id ef7aeeba18ca311121615297ec119f522102abda02fec4811c9a3f794aa725d2
Sep 18 19:51:31.383493 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:31.383415    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx" event={"ID":"9479f11f-d9ba-4591-b9a9-22d232df00fe","Type":"ContainerStarted","Data":"ef7aeeba18ca311121615297ec119f522102abda02fec4811c9a3f794aa725d2"}
Sep 18 19:51:33.400662 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:33.400625    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx" event={"ID":"9479f11f-d9ba-4591-b9a9-22d232df00fe","Type":"ContainerStarted","Data":"d15605f34dbf3ee27e5e7b2d78a7f9bd771fb69d127701e704b87409cbc12bd1"}
Sep 18 19:51:33.418588 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:33.418542    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx" podStartSLOduration=1.997656742 podStartE2EDuration="3.418529188s" totalImagesPullingTime="1.420872446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:51:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:51:33.417817897 +0000 UTC m=+816.520106218" watchObservedRunningTime="2026-09-18 19:51:33.418529188 +0000 UTC m=+816.520817516"
Sep 18 19:51:33.825103 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:33.825072    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx"
Sep 18 19:51:40.825067 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:40.825033    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx"
Sep 18 19:51:40.837552 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:40.837523    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx"
Sep 18 19:51:41.453434 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:51:41.453407    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx"
Sep 18 19:52:11.391018 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:52:11.390987    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx"]
Sep 18 19:52:11.400230 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:52:11.400193    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 19:52:11.400230 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:52:11.400230 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 19:52:11.400230 ip-10-0-143-134 kubenswrapper[2581]:  > containerID="d15605f34dbf3ee27e5e7b2d78a7f9bd771fb69d127701e704b87409cbc12bd1" cmd=["/bin/sleep","15"]
Sep 18 19:52:11.400506 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:52:11.400247    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx" message="" err=<
Sep 18 19:52:11.400506 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:52:11.400506 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 19:52:11.400506 ip-10-0-143-134 kubenswrapper[2581]:  >
Sep 18 19:52:11.400506 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:52:11.400259    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 19:52:11.400506 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:52:11.400506 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 19:52:11.400506 ip-10-0-143-134 kubenswrapper[2581]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx" podUID="9479f11f-d9ba-4591-b9a9-22d232df00fe" containerName="main" containerID="cri-o://d15605f34dbf3ee27e5e7b2d78a7f9bd771fb69d127701e704b87409cbc12bd1"
Sep 18 19:52:11.400506 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:52:11.400302    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx" podUID="9479f11f-d9ba-4591-b9a9-22d232df00fe" containerName="main" containerID="cri-o://d15605f34dbf3ee27e5e7b2d78a7f9bd771fb69d127701e704b87409cbc12bd1" gracePeriod=60
Sep 18 19:52:11.443051 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:52:11.443014    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx" podUID="9479f11f-d9ba-4591-b9a9-22d232df00fe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 18 19:52:11.598506 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:52:11.598474    2581 generic.go:356] "Generic (PLEG): container finished" podID="9479f11f-d9ba-4591-b9a9-22d232df00fe" containerID="d15605f34dbf3ee27e5e7b2d78a7f9bd771fb69d127701e704b87409cbc12bd1" exitCode=0
Sep 18 19:52:11.598652 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:52:11.598500    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx" event={"ID":"9479f11f-d9ba-4591-b9a9-22d232df00fe","Type":"ContainerDied","Data":"d15605f34dbf3ee27e5e7b2d78a7f9bd771fb69d127701e704b87409cbc12bd1"}
Sep 18 19:52:12.148508 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:52:12.148488    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx"
Sep 18 19:52:12.275460 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:52:12.275440    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9479f11f-d9ba-4591-b9a9-22d232df00fe-dshm\") pod \"9479f11f-d9ba-4591-b9a9-22d232df00fe\" (UID: \"9479f11f-d9ba-4591-b9a9-22d232df00fe\") "
Sep 18 19:52:12.275621 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:52:12.275475    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9479f11f-d9ba-4591-b9a9-22d232df00fe-home\") pod \"9479f11f-d9ba-4591-b9a9-22d232df00fe\" (UID: \"9479f11f-d9ba-4591-b9a9-22d232df00fe\") "
Sep 18 19:52:12.275621 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:52:12.275538    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdntf\" (UniqueName: \"kubernetes.io/projected/9479f11f-d9ba-4591-b9a9-22d232df00fe-kube-api-access-fdntf\") pod \"9479f11f-d9ba-4591-b9a9-22d232df00fe\" (UID: \"9479f11f-d9ba-4591-b9a9-22d232df00fe\") "
Sep 18 19:52:12.275621 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:52:12.275586    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9479f11f-d9ba-4591-b9a9-22d232df00fe-tls-certs\") pod \"9479f11f-d9ba-4591-b9a9-22d232df00fe\" (UID: \"9479f11f-d9ba-4591-b9a9-22d232df00fe\") "
Sep 18 19:52:12.275621 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:52:12.275602    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9479f11f-d9ba-4591-b9a9-22d232df00fe-tmp-dir\") pod \"9479f11f-d9ba-4591-b9a9-22d232df00fe\" (UID: \"9479f11f-d9ba-4591-b9a9-22d232df00fe\") "
Sep 18 19:52:12.275829 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:52:12.275636    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9479f11f-d9ba-4591-b9a9-22d232df00fe-model-cache\") pod \"9479f11f-d9ba-4591-b9a9-22d232df00fe\" (UID: \"9479f11f-d9ba-4591-b9a9-22d232df00fe\") "
Sep 18 19:52:12.275829 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:52:12.275797    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9479f11f-d9ba-4591-b9a9-22d232df00fe-home" (OuterVolumeSpecName: "home") pod "9479f11f-d9ba-4591-b9a9-22d232df00fe" (UID: "9479f11f-d9ba-4591-b9a9-22d232df00fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:52:12.275983 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:52:12.275926    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9479f11f-d9ba-4591-b9a9-22d232df00fe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9479f11f-d9ba-4591-b9a9-22d232df00fe" (UID: "9479f11f-d9ba-4591-b9a9-22d232df00fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:52:12.276044 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:52:12.276007    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9479f11f-d9ba-4591-b9a9-22d232df00fe-model-cache" (OuterVolumeSpecName: "model-cache") pod "9479f11f-d9ba-4591-b9a9-22d232df00fe" (UID: "9479f11f-d9ba-4591-b9a9-22d232df00fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:52:12.276112 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:52:12.276099    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9479f11f-d9ba-4591-b9a9-22d232df00fe-tmp-dir\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:52:12.276163 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:52:12.276118    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9479f11f-d9ba-4591-b9a9-22d232df00fe-home\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:52:12.277926 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:52:12.277904    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9479f11f-d9ba-4591-b9a9-22d232df00fe-dshm" (OuterVolumeSpecName: "dshm") pod "9479f11f-d9ba-4591-b9a9-22d232df00fe" (UID: "9479f11f-d9ba-4591-b9a9-22d232df00fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:52:12.278038 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:52:12.277949    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9479f11f-d9ba-4591-b9a9-22d232df00fe-kube-api-access-fdntf" (OuterVolumeSpecName: "kube-api-access-fdntf") pod "9479f11f-d9ba-4591-b9a9-22d232df00fe" (UID: "9479f11f-d9ba-4591-b9a9-22d232df00fe"). InnerVolumeSpecName "kube-api-access-fdntf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:52:12.278038 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:52:12.277956    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9479f11f-d9ba-4591-b9a9-22d232df00fe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9479f11f-d9ba-4591-b9a9-22d232df00fe" (UID: "9479f11f-d9ba-4591-b9a9-22d232df00fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:52:12.377320 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:52:12.377281    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdntf\" (UniqueName: \"kubernetes.io/projected/9479f11f-d9ba-4591-b9a9-22d232df00fe-kube-api-access-fdntf\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:52:12.377320 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:52:12.377314    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9479f11f-d9ba-4591-b9a9-22d232df00fe-tls-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:52:12.377320 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:52:12.377324    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9479f11f-d9ba-4591-b9a9-22d232df00fe-model-cache\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:52:12.377320 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:52:12.377331    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9479f11f-d9ba-4591-b9a9-22d232df00fe-dshm\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:52:12.605256 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:52:12.605180    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx"
Sep 18 19:52:12.605697 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:52:12.605180    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx" event={"ID":"9479f11f-d9ba-4591-b9a9-22d232df00fe","Type":"ContainerDied","Data":"ef7aeeba18ca311121615297ec119f522102abda02fec4811c9a3f794aa725d2"}
Sep 18 19:52:12.605697 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:52:12.605313    2581 scope.go:117] "RemoveContainer" containerID="d15605f34dbf3ee27e5e7b2d78a7f9bd771fb69d127701e704b87409cbc12bd1"
Sep 18 19:52:12.626801 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:52:12.626775    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx"]
Sep 18 19:52:12.631894 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:52:12.631873    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wp2vx"]
Sep 18 19:52:13.433699 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:52:13.433657    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9479f11f-d9ba-4591-b9a9-22d232df00fe" path="/var/lib/kubelet/pods/9479f11f-d9ba-4591-b9a9-22d232df00fe/volumes"
Sep 18 19:53:07.006435 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.006399    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8"]
Sep 18 19:53:07.009073 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.006826    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9479f11f-d9ba-4591-b9a9-22d232df00fe" containerName="main"
Sep 18 19:53:07.009073 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.006854    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9479f11f-d9ba-4591-b9a9-22d232df00fe" containerName="main"
Sep 18 19:53:07.009073 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.006924    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9479f11f-d9ba-4591-b9a9-22d232df00fe" containerName="main"
Sep 18 19:53:07.009895 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.009869    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8"
Sep 18 19:53:07.013496 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.013471    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 18 19:53:07.013496 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.013487    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 18 19:53:07.013682 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.013505    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-xm29g\""
Sep 18 19:53:07.013682 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.013525    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 18 19:53:07.018571 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.018548    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8"]
Sep 18 19:53:07.116912 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.116885    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/401e1b33-e0b6-4186-9455-8103b8655877-home\") pod \"fc-concurrency-test-kserve-b5d44467c-lxbn8\" (UID: \"401e1b33-e0b6-4186-9455-8103b8655877\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8"
Sep 18 19:53:07.117087 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.116925    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/401e1b33-e0b6-4186-9455-8103b8655877-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-lxbn8\" (UID: \"401e1b33-e0b6-4186-9455-8103b8655877\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8"
Sep 18 19:53:07.117087 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.116958    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/401e1b33-e0b6-4186-9455-8103b8655877-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-lxbn8\" (UID: \"401e1b33-e0b6-4186-9455-8103b8655877\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8"
Sep 18 19:53:07.117087 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.117032    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncn6h\" (UniqueName: \"kubernetes.io/projected/401e1b33-e0b6-4186-9455-8103b8655877-kube-api-access-ncn6h\") pod \"fc-concurrency-test-kserve-b5d44467c-lxbn8\" (UID: \"401e1b33-e0b6-4186-9455-8103b8655877\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8"
Sep 18 19:53:07.117087 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.117060    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/401e1b33-e0b6-4186-9455-8103b8655877-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-lxbn8\" (UID: \"401e1b33-e0b6-4186-9455-8103b8655877\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8"
Sep 18 19:53:07.117246 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.117091    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/401e1b33-e0b6-4186-9455-8103b8655877-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-lxbn8\" (UID: \"401e1b33-e0b6-4186-9455-8103b8655877\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8"
Sep 18 19:53:07.217620 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.217589    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncn6h\" (UniqueName: \"kubernetes.io/projected/401e1b33-e0b6-4186-9455-8103b8655877-kube-api-access-ncn6h\") pod \"fc-concurrency-test-kserve-b5d44467c-lxbn8\" (UID: \"401e1b33-e0b6-4186-9455-8103b8655877\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8"
Sep 18 19:53:07.217789 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.217624    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/401e1b33-e0b6-4186-9455-8103b8655877-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-lxbn8\" (UID: \"401e1b33-e0b6-4186-9455-8103b8655877\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8"
Sep 18 19:53:07.217789 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.217641    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/401e1b33-e0b6-4186-9455-8103b8655877-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-lxbn8\" (UID: \"401e1b33-e0b6-4186-9455-8103b8655877\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8"
Sep 18 19:53:07.217789 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.217678    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/401e1b33-e0b6-4186-9455-8103b8655877-home\") pod \"fc-concurrency-test-kserve-b5d44467c-lxbn8\" (UID: \"401e1b33-e0b6-4186-9455-8103b8655877\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8"
Sep 18 19:53:07.217789 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.217703    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/401e1b33-e0b6-4186-9455-8103b8655877-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-lxbn8\" (UID: \"401e1b33-e0b6-4186-9455-8103b8655877\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8"
Sep 18 19:53:07.217789 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.217730    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/401e1b33-e0b6-4186-9455-8103b8655877-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-lxbn8\" (UID: \"401e1b33-e0b6-4186-9455-8103b8655877\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8"
Sep 18 19:53:07.218160 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.218138    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/401e1b33-e0b6-4186-9455-8103b8655877-home\") pod \"fc-concurrency-test-kserve-b5d44467c-lxbn8\" (UID: \"401e1b33-e0b6-4186-9455-8103b8655877\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8"
Sep 18 19:53:07.218225 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.218203    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/401e1b33-e0b6-4186-9455-8103b8655877-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-lxbn8\" (UID: \"401e1b33-e0b6-4186-9455-8103b8655877\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8"
Sep 18 19:53:07.218261 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.218245    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/401e1b33-e0b6-4186-9455-8103b8655877-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-lxbn8\" (UID: \"401e1b33-e0b6-4186-9455-8103b8655877\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8"
Sep 18 19:53:07.220124 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.220106    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/401e1b33-e0b6-4186-9455-8103b8655877-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-lxbn8\" (UID: \"401e1b33-e0b6-4186-9455-8103b8655877\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8"
Sep 18 19:53:07.220303 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.220286    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/401e1b33-e0b6-4186-9455-8103b8655877-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-lxbn8\" (UID: \"401e1b33-e0b6-4186-9455-8103b8655877\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8"
Sep 18 19:53:07.225199 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.225178    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncn6h\" (UniqueName: \"kubernetes.io/projected/401e1b33-e0b6-4186-9455-8103b8655877-kube-api-access-ncn6h\") pod \"fc-concurrency-test-kserve-b5d44467c-lxbn8\" (UID: \"401e1b33-e0b6-4186-9455-8103b8655877\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8"
Sep 18 19:53:07.322208 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.322175    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8"
Sep 18 19:53:07.594202 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.594177    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8"]
Sep 18 19:53:07.595699 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:53:07.595671    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod401e1b33_e0b6_4186_9455_8103b8655877.slice/crio-7afc83cbf25f372bdb7b6957344d6f6e7f7fd10bcab5b82247163b5e23df25e4 WatchSource:0}: Error finding container 7afc83cbf25f372bdb7b6957344d6f6e7f7fd10bcab5b82247163b5e23df25e4: Status 404 returned error can't find the container with id 7afc83cbf25f372bdb7b6957344d6f6e7f7fd10bcab5b82247163b5e23df25e4
Sep 18 19:53:07.902299 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.902222    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8" event={"ID":"401e1b33-e0b6-4186-9455-8103b8655877","Type":"ContainerStarted","Data":"714fb6c2f464257c21121a8b1a9ce5666d72a0b5375c50f5a3895d5fb576e01b"}
Sep 18 19:53:07.902299 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.902285    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8" event={"ID":"401e1b33-e0b6-4186-9455-8103b8655877","Type":"ContainerStarted","Data":"7afc83cbf25f372bdb7b6957344d6f6e7f7fd10bcab5b82247163b5e23df25e4"}
Sep 18 19:53:07.962142 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:07.962084    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8" podStartSLOduration=1.962063896 podStartE2EDuration="1.962063896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:53:07.958169572 +0000 UTC m=+911.060457905" watchObservedRunningTime="2026-09-18 19:53:07.962063896 +0000 UTC m=+911.064352225"
Sep 18 19:53:08.322575 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:08.322542    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8"
Sep 18 19:53:17.323341 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:17.323308    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8"
Sep 18 19:53:17.335518 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:17.335495    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8"
Sep 18 19:53:17.966327 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:17.966305    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8"
Sep 18 19:53:45.952281 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:45.952243    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8"]
Sep 18 19:53:45.962890 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:53:45.962823    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 19:53:45.962890 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:53:45.962890 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 19:53:45.962890 ip-10-0-143-134 kubenswrapper[2581]:  > containerID="714fb6c2f464257c21121a8b1a9ce5666d72a0b5375c50f5a3895d5fb576e01b" cmd=["/bin/sleep","15"]
Sep 18 19:53:45.963205 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:45.962915    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8" message="" err=<
Sep 18 19:53:45.963205 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:53:45.963205 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 19:53:45.963205 ip-10-0-143-134 kubenswrapper[2581]:  >
Sep 18 19:53:45.963205 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:53:45.962932    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 19:53:45.963205 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 19:53:45.963205 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 19:53:45.963205 ip-10-0-143-134 kubenswrapper[2581]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8" podUID="401e1b33-e0b6-4186-9455-8103b8655877" containerName="main" containerID="cri-o://714fb6c2f464257c21121a8b1a9ce5666d72a0b5375c50f5a3895d5fb576e01b"
Sep 18 19:53:45.963205 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:45.962993    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8" podUID="401e1b33-e0b6-4186-9455-8103b8655877" containerName="main" containerID="cri-o://714fb6c2f464257c21121a8b1a9ce5666d72a0b5375c50f5a3895d5fb576e01b" gracePeriod=60
Sep 18 19:53:46.324972 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:46.324948    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8"
Sep 18 19:53:46.452074 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:46.452039    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncn6h\" (UniqueName: \"kubernetes.io/projected/401e1b33-e0b6-4186-9455-8103b8655877-kube-api-access-ncn6h\") pod \"401e1b33-e0b6-4186-9455-8103b8655877\" (UID: \"401e1b33-e0b6-4186-9455-8103b8655877\") "
Sep 18 19:53:46.452250 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:46.452087    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/401e1b33-e0b6-4186-9455-8103b8655877-model-cache\") pod \"401e1b33-e0b6-4186-9455-8103b8655877\" (UID: \"401e1b33-e0b6-4186-9455-8103b8655877\") "
Sep 18 19:53:46.452250 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:46.452124    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/401e1b33-e0b6-4186-9455-8103b8655877-tls-certs\") pod \"401e1b33-e0b6-4186-9455-8103b8655877\" (UID: \"401e1b33-e0b6-4186-9455-8103b8655877\") "
Sep 18 19:53:46.452250 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:46.452150    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/401e1b33-e0b6-4186-9455-8103b8655877-dshm\") pod \"401e1b33-e0b6-4186-9455-8103b8655877\" (UID: \"401e1b33-e0b6-4186-9455-8103b8655877\") "
Sep 18 19:53:46.452250 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:46.452229    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/401e1b33-e0b6-4186-9455-8103b8655877-tmp-dir\") pod \"401e1b33-e0b6-4186-9455-8103b8655877\" (UID: \"401e1b33-e0b6-4186-9455-8103b8655877\") "
Sep 18 19:53:46.452475 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:46.452264    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/401e1b33-e0b6-4186-9455-8103b8655877-home\") pod \"401e1b33-e0b6-4186-9455-8103b8655877\" (UID: \"401e1b33-e0b6-4186-9455-8103b8655877\") "
Sep 18 19:53:46.452539 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:46.452470    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/401e1b33-e0b6-4186-9455-8103b8655877-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "401e1b33-e0b6-4186-9455-8103b8655877" (UID: "401e1b33-e0b6-4186-9455-8103b8655877"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:53:46.452590 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:46.452537    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/401e1b33-e0b6-4186-9455-8103b8655877-home" (OuterVolumeSpecName: "home") pod "401e1b33-e0b6-4186-9455-8103b8655877" (UID: "401e1b33-e0b6-4186-9455-8103b8655877"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:53:46.452590 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:46.452350    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/401e1b33-e0b6-4186-9455-8103b8655877-model-cache" (OuterVolumeSpecName: "model-cache") pod "401e1b33-e0b6-4186-9455-8103b8655877" (UID: "401e1b33-e0b6-4186-9455-8103b8655877"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:53:46.455464 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:46.454642    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/401e1b33-e0b6-4186-9455-8103b8655877-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "401e1b33-e0b6-4186-9455-8103b8655877" (UID: "401e1b33-e0b6-4186-9455-8103b8655877"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:53:46.455464 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:46.455092    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/401e1b33-e0b6-4186-9455-8103b8655877-kube-api-access-ncn6h" (OuterVolumeSpecName: "kube-api-access-ncn6h") pod "401e1b33-e0b6-4186-9455-8103b8655877" (UID: "401e1b33-e0b6-4186-9455-8103b8655877"). InnerVolumeSpecName "kube-api-access-ncn6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:53:46.455464 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:46.455409    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/401e1b33-e0b6-4186-9455-8103b8655877-dshm" (OuterVolumeSpecName: "dshm") pod "401e1b33-e0b6-4186-9455-8103b8655877" (UID: "401e1b33-e0b6-4186-9455-8103b8655877"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:53:46.553573 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:46.553542    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/401e1b33-e0b6-4186-9455-8103b8655877-tmp-dir\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:53:46.553573 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:46.553565    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/401e1b33-e0b6-4186-9455-8103b8655877-home\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:53:46.553573 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:46.553575    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncn6h\" (UniqueName: \"kubernetes.io/projected/401e1b33-e0b6-4186-9455-8103b8655877-kube-api-access-ncn6h\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:53:46.553780 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:46.553584    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/401e1b33-e0b6-4186-9455-8103b8655877-model-cache\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:53:46.553780 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:46.553592    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/401e1b33-e0b6-4186-9455-8103b8655877-tls-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:53:46.553780 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:46.553599    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/401e1b33-e0b6-4186-9455-8103b8655877-dshm\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:53:47.108465 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:47.108429    2581 generic.go:356] "Generic (PLEG): container finished" podID="401e1b33-e0b6-4186-9455-8103b8655877" containerID="714fb6c2f464257c21121a8b1a9ce5666d72a0b5375c50f5a3895d5fb576e01b" exitCode=0
Sep 18 19:53:47.108913 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:47.108480    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8"
Sep 18 19:53:47.108913 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:47.108516    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8" event={"ID":"401e1b33-e0b6-4186-9455-8103b8655877","Type":"ContainerDied","Data":"714fb6c2f464257c21121a8b1a9ce5666d72a0b5375c50f5a3895d5fb576e01b"}
Sep 18 19:53:47.108913 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:47.108568    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8" event={"ID":"401e1b33-e0b6-4186-9455-8103b8655877","Type":"ContainerDied","Data":"7afc83cbf25f372bdb7b6957344d6f6e7f7fd10bcab5b82247163b5e23df25e4"}
Sep 18 19:53:47.108913 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:47.108591    2581 scope.go:117] "RemoveContainer" containerID="714fb6c2f464257c21121a8b1a9ce5666d72a0b5375c50f5a3895d5fb576e01b"
Sep 18 19:53:47.118740 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:47.118716    2581 scope.go:117] "RemoveContainer" containerID="714fb6c2f464257c21121a8b1a9ce5666d72a0b5375c50f5a3895d5fb576e01b"
Sep 18 19:53:47.119038 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:53:47.119012    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"714fb6c2f464257c21121a8b1a9ce5666d72a0b5375c50f5a3895d5fb576e01b\": container with ID starting with 714fb6c2f464257c21121a8b1a9ce5666d72a0b5375c50f5a3895d5fb576e01b not found: ID does not exist" containerID="714fb6c2f464257c21121a8b1a9ce5666d72a0b5375c50f5a3895d5fb576e01b"
Sep 18 19:53:47.119098 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:47.119050    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"714fb6c2f464257c21121a8b1a9ce5666d72a0b5375c50f5a3895d5fb576e01b"} err="failed to get container status \"714fb6c2f464257c21121a8b1a9ce5666d72a0b5375c50f5a3895d5fb576e01b\": rpc error: code = NotFound desc = could not find container \"714fb6c2f464257c21121a8b1a9ce5666d72a0b5375c50f5a3895d5fb576e01b\": container with ID starting with 714fb6c2f464257c21121a8b1a9ce5666d72a0b5375c50f5a3895d5fb576e01b not found: ID does not exist"
Sep 18 19:53:47.130859 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:47.130823    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8"]
Sep 18 19:53:47.134193 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:47.134174    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-lxbn8"]
Sep 18 19:53:47.434412 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:53:47.434335    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="401e1b33-e0b6-4186-9455-8103b8655877" path="/var/lib/kubelet/pods/401e1b33-e0b6-4186-9455-8103b8655877/volumes"
Sep 18 19:54:09.190650 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.190618    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 18 19:54:09.191189 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.191173    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="401e1b33-e0b6-4186-9455-8103b8655877" containerName="main"
Sep 18 19:54:09.191189 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.191191    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="401e1b33-e0b6-4186-9455-8103b8655877" containerName="main"
Sep 18 19:54:09.191298 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.191272    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="401e1b33-e0b6-4186-9455-8103b8655877" containerName="main"
Sep 18 19:54:09.194743 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.194728    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 19:54:09.197500 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.197476    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 18 19:54:09.200032 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.200016    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-68qj6\""
Sep 18 19:54:09.200103 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.200016    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-nlqwt\""
Sep 18 19:54:09.201209 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.201192    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 18 19:54:09.202720 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.202696    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:54:09.202994 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.202706    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 18 19:54:09.207980 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.207957    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 18 19:54:09.219271 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.219245    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 18 19:54:09.357242 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.357211    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79c2bc8d-19c2-463e-a801-e2392e400b8a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"79c2bc8d-19c2-463e-a801-e2392e400b8a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 19:54:09.357446 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.357279    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79c2bc8d-19c2-463e-a801-e2392e400b8a-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"79c2bc8d-19c2-463e-a801-e2392e400b8a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 19:54:09.357446 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.357298    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79c2bc8d-19c2-463e-a801-e2392e400b8a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"79c2bc8d-19c2-463e-a801-e2392e400b8a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 19:54:09.357446 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.357389    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79c2bc8d-19c2-463e-a801-e2392e400b8a-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"79c2bc8d-19c2-463e-a801-e2392e400b8a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 19:54:09.357446 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.357425    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79c2bc8d-19c2-463e-a801-e2392e400b8a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"79c2bc8d-19c2-463e-a801-e2392e400b8a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 19:54:09.357644 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.357463    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvbv8\" (UniqueName: \"kubernetes.io/projected/79c2bc8d-19c2-463e-a801-e2392e400b8a-kube-api-access-xvbv8\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"79c2bc8d-19c2-463e-a801-e2392e400b8a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 19:54:09.357644 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.357513    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79c2bc8d-19c2-463e-a801-e2392e400b8a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"79c2bc8d-19c2-463e-a801-e2392e400b8a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 19:54:09.357644 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.357543    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79c2bc8d-19c2-463e-a801-e2392e400b8a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"79c2bc8d-19c2-463e-a801-e2392e400b8a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 19:54:09.458441 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.458369    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79c2bc8d-19c2-463e-a801-e2392e400b8a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"79c2bc8d-19c2-463e-a801-e2392e400b8a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 19:54:09.458441 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.458398    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79c2bc8d-19c2-463e-a801-e2392e400b8a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"79c2bc8d-19c2-463e-a801-e2392e400b8a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 19:54:09.458441 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.458419    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79c2bc8d-19c2-463e-a801-e2392e400b8a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"79c2bc8d-19c2-463e-a801-e2392e400b8a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 19:54:09.458742 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.458477    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79c2bc8d-19c2-463e-a801-e2392e400b8a-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"79c2bc8d-19c2-463e-a801-e2392e400b8a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 19:54:09.458742 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.458503    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79c2bc8d-19c2-463e-a801-e2392e400b8a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"79c2bc8d-19c2-463e-a801-e2392e400b8a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 19:54:09.458742 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.458546    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79c2bc8d-19c2-463e-a801-e2392e400b8a-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"79c2bc8d-19c2-463e-a801-e2392e400b8a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 19:54:09.458742 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.458566    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79c2bc8d-19c2-463e-a801-e2392e400b8a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"79c2bc8d-19c2-463e-a801-e2392e400b8a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 19:54:09.458742 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.458597    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvbv8\" (UniqueName: \"kubernetes.io/projected/79c2bc8d-19c2-463e-a801-e2392e400b8a-kube-api-access-xvbv8\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"79c2bc8d-19c2-463e-a801-e2392e400b8a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 19:54:09.458742 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.458734    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79c2bc8d-19c2-463e-a801-e2392e400b8a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"79c2bc8d-19c2-463e-a801-e2392e400b8a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 19:54:09.459075 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.458860    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79c2bc8d-19c2-463e-a801-e2392e400b8a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"79c2bc8d-19c2-463e-a801-e2392e400b8a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 19:54:09.459075 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.458919    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79c2bc8d-19c2-463e-a801-e2392e400b8a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"79c2bc8d-19c2-463e-a801-e2392e400b8a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 19:54:09.459185 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.459110    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79c2bc8d-19c2-463e-a801-e2392e400b8a-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"79c2bc8d-19c2-463e-a801-e2392e400b8a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 19:54:09.459185 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.459161    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79c2bc8d-19c2-463e-a801-e2392e400b8a-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"79c2bc8d-19c2-463e-a801-e2392e400b8a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 19:54:09.460903 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.460884    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79c2bc8d-19c2-463e-a801-e2392e400b8a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"79c2bc8d-19c2-463e-a801-e2392e400b8a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 19:54:09.461011 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.460978    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79c2bc8d-19c2-463e-a801-e2392e400b8a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"79c2bc8d-19c2-463e-a801-e2392e400b8a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 19:54:09.467672 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.467651    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvbv8\" (UniqueName: \"kubernetes.io/projected/79c2bc8d-19c2-463e-a801-e2392e400b8a-kube-api-access-xvbv8\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"79c2bc8d-19c2-463e-a801-e2392e400b8a\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 19:54:09.507689 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.507661    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 19:54:09.789215 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:54:09.789185    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c2bc8d_19c2_463e_a801_e2392e400b8a.slice/crio-be803d8280055ee2d5286c9b983000a3e43729288b63de7fc7e8c42b438efef6 WatchSource:0}: Error finding container be803d8280055ee2d5286c9b983000a3e43729288b63de7fc7e8c42b438efef6: Status 404 returned error can't find the container with id be803d8280055ee2d5286c9b983000a3e43729288b63de7fc7e8c42b438efef6
Sep 18 19:54:09.789468 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:09.789437    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 18 19:54:10.229795 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:10.229710    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"79c2bc8d-19c2-463e-a801-e2392e400b8a","Type":"ContainerStarted","Data":"be803d8280055ee2d5286c9b983000a3e43729288b63de7fc7e8c42b438efef6"}
Sep 18 19:54:13.249538 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:13.249435    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"79c2bc8d-19c2-463e-a801-e2392e400b8a","Type":"ContainerStarted","Data":"83acce648423d0cddc50a039ab23e7ab7a102e213b4fb0cd32578cdbed3a82f2"}
Sep 18 19:54:15.261884 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:15.261825    2581 generic.go:356] "Generic (PLEG): container finished" podID="79c2bc8d-19c2-463e-a801-e2392e400b8a" containerID="83acce648423d0cddc50a039ab23e7ab7a102e213b4fb0cd32578cdbed3a82f2" exitCode=0
Sep 18 19:54:15.262269 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:15.261908    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"79c2bc8d-19c2-463e-a801-e2392e400b8a","Type":"ContainerDied","Data":"83acce648423d0cddc50a039ab23e7ab7a102e213b4fb0cd32578cdbed3a82f2"}
Sep 18 19:54:15.263400 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:15.263380    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:54:43.455010 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:43.454977    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"79c2bc8d-19c2-463e-a801-e2392e400b8a","Type":"ContainerStarted","Data":"c61223b35ce1439e8eb458a06555d8856591913c1656f329396ec7d0f524821a"}
Sep 18 19:54:43.475597 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:54:43.475546    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.033611846 podStartE2EDuration="34.47553037s" totalImagesPullingTime="30.389300622s" totalInitContainerRuntime="2.052617902s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:54:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:54:43.47232991 +0000 UTC m=+1006.574618237" watchObservedRunningTime="2026-09-18 19:54:43.47553037 +0000 UTC m=+1006.577818698"
Sep 18 19:55:19.690870 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.690822    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"]
Sep 18 19:55:19.717469 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.717430    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"]
Sep 18 19:55:19.717814 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.717571    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:19.720562 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.720536    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 18 19:55:19.720706 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.720672    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Sep 18 19:55:19.720806 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.720783    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-dbpcr\""
Sep 18 19:55:19.721202 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.720792    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 18 19:55:19.789501 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.789467    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb4950f-050d-42b5-a10e-d232b60f871a-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-sklk9\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:19.789671 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.789511    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb4950f-050d-42b5-a10e-d232b60f871a-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-sklk9\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:19.789671 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.789559    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mgvg\" (UniqueName: \"kubernetes.io/projected/3bb4950f-050d-42b5-a10e-d232b60f871a-kube-api-access-9mgvg\") pod \"router-gateway-1-openshift-default-66bbccc5fb-sklk9\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:19.789671 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.789595    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb4950f-050d-42b5-a10e-d232b60f871a-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-sklk9\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:19.789671 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.789616    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb4950f-050d-42b5-a10e-d232b60f871a-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-sklk9\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:19.789671 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.789656    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb4950f-050d-42b5-a10e-d232b60f871a-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-sklk9\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:19.789868 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.789680    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb4950f-050d-42b5-a10e-d232b60f871a-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-sklk9\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:19.789868 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.789704    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb4950f-050d-42b5-a10e-d232b60f871a-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-sklk9\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:19.789868 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.789725    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb4950f-050d-42b5-a10e-d232b60f871a-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-sklk9\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:19.891188 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.891154    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb4950f-050d-42b5-a10e-d232b60f871a-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-sklk9\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:19.891365 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.891207    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb4950f-050d-42b5-a10e-d232b60f871a-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-sklk9\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:19.891365 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.891259    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb4950f-050d-42b5-a10e-d232b60f871a-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-sklk9\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:19.891365 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.891292    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb4950f-050d-42b5-a10e-d232b60f871a-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-sklk9\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:19.891365 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.891322    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mgvg\" (UniqueName: \"kubernetes.io/projected/3bb4950f-050d-42b5-a10e-d232b60f871a-kube-api-access-9mgvg\") pod \"router-gateway-1-openshift-default-66bbccc5fb-sklk9\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:19.891577 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.891367    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb4950f-050d-42b5-a10e-d232b60f871a-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-sklk9\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:19.891577 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.891392    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb4950f-050d-42b5-a10e-d232b60f871a-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-sklk9\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:19.891577 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.891441    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb4950f-050d-42b5-a10e-d232b60f871a-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-sklk9\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:19.891577 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.891481    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb4950f-050d-42b5-a10e-d232b60f871a-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-sklk9\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:19.891812 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.891619    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb4950f-050d-42b5-a10e-d232b60f871a-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-sklk9\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:19.891894 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.891869    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb4950f-050d-42b5-a10e-d232b60f871a-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-sklk9\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:19.891947 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.891899    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb4950f-050d-42b5-a10e-d232b60f871a-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-sklk9\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:19.891947 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.891934    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb4950f-050d-42b5-a10e-d232b60f871a-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-sklk9\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:19.892209 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.892021    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb4950f-050d-42b5-a10e-d232b60f871a-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-sklk9\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:19.893735 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.893712    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb4950f-050d-42b5-a10e-d232b60f871a-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-sklk9\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:19.894240 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.894220    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb4950f-050d-42b5-a10e-d232b60f871a-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-sklk9\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:19.916296 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.916272    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mgvg\" (UniqueName: \"kubernetes.io/projected/3bb4950f-050d-42b5-a10e-d232b60f871a-kube-api-access-9mgvg\") pod \"router-gateway-1-openshift-default-66bbccc5fb-sklk9\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:19.917383 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:19.917361    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb4950f-050d-42b5-a10e-d232b60f871a-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-sklk9\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:20.030277 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:20.030248    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:20.314137 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:20.314113    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"]
Sep 18 19:55:20.315896 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:55:20.315866    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb4950f_050d_42b5_a10e_d232b60f871a.slice/crio-1f43feda97dd7441eb30917c4c3d3baea3a9f064ac6493f18f55e030904fefe6 WatchSource:0}: Error finding container 1f43feda97dd7441eb30917c4c3d3baea3a9f064ac6493f18f55e030904fefe6: Status 404 returned error can't find the container with id 1f43feda97dd7441eb30917c4c3d3baea3a9f064ac6493f18f55e030904fefe6
Sep 18 19:55:20.669315 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:20.669232    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9" event={"ID":"3bb4950f-050d-42b5-a10e-d232b60f871a","Type":"ContainerStarted","Data":"1f43feda97dd7441eb30917c4c3d3baea3a9f064ac6493f18f55e030904fefe6"}
Sep 18 19:55:23.098235 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:23.098197    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 18 19:55:23.098494 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:23.098274    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 18 19:55:23.098494 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:23.098299    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 18 19:55:23.689660 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:23.689623    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9" event={"ID":"3bb4950f-050d-42b5-a10e-d232b60f871a","Type":"ContainerStarted","Data":"65b62d61a7cfdcc21ec2b25facd9ebfbff1e1aee5052a54971cc91067a6563e5"}
Sep 18 19:55:23.711612 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:23.711565    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9" podStartSLOduration=1.931509003 podStartE2EDuration="4.711550824s" totalImagesPullingTime="2.780041821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:55:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:55:23.708824699 +0000 UTC m=+1046.811113027" watchObservedRunningTime="2026-09-18 19:55:23.711550824 +0000 UTC m=+1046.813839151"
Sep 18 19:55:24.031236 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:24.031199    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:24.036126 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:24.036104    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:24.694994 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:24.694960    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:24.696085 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:24.696068    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:41.519431 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:41.519390    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"]
Sep 18 19:55:41.519901 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:41.519617    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9" podUID="3bb4950f-050d-42b5-a10e-d232b60f871a" containerName="istio-proxy" containerID="cri-o://65b62d61a7cfdcc21ec2b25facd9ebfbff1e1aee5052a54971cc91067a6563e5" gracePeriod=30
Sep 18 19:55:44.364458 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.364435    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:44.429428 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.429399    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb4950f-050d-42b5-a10e-d232b60f871a-istio-token\") pod \"3bb4950f-050d-42b5-a10e-d232b60f871a\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") "
Sep 18 19:55:44.429588 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.429440    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb4950f-050d-42b5-a10e-d232b60f871a-workload-certs\") pod \"3bb4950f-050d-42b5-a10e-d232b60f871a\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") "
Sep 18 19:55:44.429588 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.429484    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb4950f-050d-42b5-a10e-d232b60f871a-credential-socket\") pod \"3bb4950f-050d-42b5-a10e-d232b60f871a\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") "
Sep 18 19:55:44.429588 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.429512    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb4950f-050d-42b5-a10e-d232b60f871a-istio-data\") pod \"3bb4950f-050d-42b5-a10e-d232b60f871a\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") "
Sep 18 19:55:44.429588 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.429539    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mgvg\" (UniqueName: \"kubernetes.io/projected/3bb4950f-050d-42b5-a10e-d232b60f871a-kube-api-access-9mgvg\") pod \"3bb4950f-050d-42b5-a10e-d232b60f871a\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") "
Sep 18 19:55:44.429588 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.429564    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb4950f-050d-42b5-a10e-d232b60f871a-istio-podinfo\") pod \"3bb4950f-050d-42b5-a10e-d232b60f871a\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") "
Sep 18 19:55:44.429588 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.429588    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb4950f-050d-42b5-a10e-d232b60f871a-istio-envoy\") pod \"3bb4950f-050d-42b5-a10e-d232b60f871a\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") "
Sep 18 19:55:44.429931 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.429651    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb4950f-050d-42b5-a10e-d232b60f871a-workload-socket\") pod \"3bb4950f-050d-42b5-a10e-d232b60f871a\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") "
Sep 18 19:55:44.429931 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.429708    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb4950f-050d-42b5-a10e-d232b60f871a-istiod-ca-cert\") pod \"3bb4950f-050d-42b5-a10e-d232b60f871a\" (UID: \"3bb4950f-050d-42b5-a10e-d232b60f871a\") "
Sep 18 19:55:44.429931 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.429802    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb4950f-050d-42b5-a10e-d232b60f871a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3bb4950f-050d-42b5-a10e-d232b60f871a" (UID: "3bb4950f-050d-42b5-a10e-d232b60f871a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:55:44.429931 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.429817    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb4950f-050d-42b5-a10e-d232b60f871a-istio-data" (OuterVolumeSpecName: "istio-data") pod "3bb4950f-050d-42b5-a10e-d232b60f871a" (UID: "3bb4950f-050d-42b5-a10e-d232b60f871a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:55:44.430210 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.430196    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bb4950f-050d-42b5-a10e-d232b60f871a-istio-data\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:55:44.430278 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.430216    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bb4950f-050d-42b5-a10e-d232b60f871a-workload-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:55:44.430339 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.430283    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb4950f-050d-42b5-a10e-d232b60f871a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3bb4950f-050d-42b5-a10e-d232b60f871a" (UID: "3bb4950f-050d-42b5-a10e-d232b60f871a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:55:44.431172 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.431129    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb4950f-050d-42b5-a10e-d232b60f871a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3bb4950f-050d-42b5-a10e-d232b60f871a" (UID: "3bb4950f-050d-42b5-a10e-d232b60f871a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:55:44.431293 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.431170    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb4950f-050d-42b5-a10e-d232b60f871a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3bb4950f-050d-42b5-a10e-d232b60f871a" (UID: "3bb4950f-050d-42b5-a10e-d232b60f871a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:55:44.432596 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.432570    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb4950f-050d-42b5-a10e-d232b60f871a-istio-token" (OuterVolumeSpecName: "istio-token") pod "3bb4950f-050d-42b5-a10e-d232b60f871a" (UID: "3bb4950f-050d-42b5-a10e-d232b60f871a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:55:44.432805 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.432777    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bb4950f-050d-42b5-a10e-d232b60f871a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3bb4950f-050d-42b5-a10e-d232b60f871a" (UID: "3bb4950f-050d-42b5-a10e-d232b60f871a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 19:55:44.433089 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.433064    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb4950f-050d-42b5-a10e-d232b60f871a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3bb4950f-050d-42b5-a10e-d232b60f871a" (UID: "3bb4950f-050d-42b5-a10e-d232b60f871a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:55:44.433230 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.433210    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb4950f-050d-42b5-a10e-d232b60f871a-kube-api-access-9mgvg" (OuterVolumeSpecName: "kube-api-access-9mgvg") pod "3bb4950f-050d-42b5-a10e-d232b60f871a" (UID: "3bb4950f-050d-42b5-a10e-d232b60f871a"). InnerVolumeSpecName "kube-api-access-9mgvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:55:44.530702 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.530664    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mgvg\" (UniqueName: \"kubernetes.io/projected/3bb4950f-050d-42b5-a10e-d232b60f871a-kube-api-access-9mgvg\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:55:44.530702 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.530703    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bb4950f-050d-42b5-a10e-d232b60f871a-istio-podinfo\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:55:44.530914 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.530719    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bb4950f-050d-42b5-a10e-d232b60f871a-istio-envoy\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:55:44.530914 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.530734    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb4950f-050d-42b5-a10e-d232b60f871a-workload-socket\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:55:44.530914 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.530748    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bb4950f-050d-42b5-a10e-d232b60f871a-istiod-ca-cert\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:55:44.530914 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.530761    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bb4950f-050d-42b5-a10e-d232b60f871a-istio-token\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:55:44.530914 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.530774    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bb4950f-050d-42b5-a10e-d232b60f871a-credential-socket\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:55:44.821196 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.821154    2581 generic.go:356] "Generic (PLEG): container finished" podID="3bb4950f-050d-42b5-a10e-d232b60f871a" containerID="65b62d61a7cfdcc21ec2b25facd9ebfbff1e1aee5052a54971cc91067a6563e5" exitCode=0
Sep 18 19:55:44.821372 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.821200    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"
Sep 18 19:55:44.821372 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.821243    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9" event={"ID":"3bb4950f-050d-42b5-a10e-d232b60f871a","Type":"ContainerDied","Data":"65b62d61a7cfdcc21ec2b25facd9ebfbff1e1aee5052a54971cc91067a6563e5"}
Sep 18 19:55:44.821372 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.821303    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9" event={"ID":"3bb4950f-050d-42b5-a10e-d232b60f871a","Type":"ContainerDied","Data":"1f43feda97dd7441eb30917c4c3d3baea3a9f064ac6493f18f55e030904fefe6"}
Sep 18 19:55:44.821372 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.821332    2581 scope.go:117] "RemoveContainer" containerID="65b62d61a7cfdcc21ec2b25facd9ebfbff1e1aee5052a54971cc91067a6563e5"
Sep 18 19:55:44.831756 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.831732    2581 scope.go:117] "RemoveContainer" containerID="65b62d61a7cfdcc21ec2b25facd9ebfbff1e1aee5052a54971cc91067a6563e5"
Sep 18 19:55:44.832115 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:55:44.832096    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65b62d61a7cfdcc21ec2b25facd9ebfbff1e1aee5052a54971cc91067a6563e5\": container with ID starting with 65b62d61a7cfdcc21ec2b25facd9ebfbff1e1aee5052a54971cc91067a6563e5 not found: ID does not exist" containerID="65b62d61a7cfdcc21ec2b25facd9ebfbff1e1aee5052a54971cc91067a6563e5"
Sep 18 19:55:44.832193 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.832123    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65b62d61a7cfdcc21ec2b25facd9ebfbff1e1aee5052a54971cc91067a6563e5"} err="failed to get container status \"65b62d61a7cfdcc21ec2b25facd9ebfbff1e1aee5052a54971cc91067a6563e5\": rpc error: code = NotFound desc = could not find container \"65b62d61a7cfdcc21ec2b25facd9ebfbff1e1aee5052a54971cc91067a6563e5\": container with ID starting with 65b62d61a7cfdcc21ec2b25facd9ebfbff1e1aee5052a54971cc91067a6563e5 not found: ID does not exist"
Sep 18 19:55:44.842846 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.842807    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"]
Sep 18 19:55:44.853766 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:44.848240    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-sklk9"]
Sep 18 19:55:45.433329 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:45.433287    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bb4950f-050d-42b5-a10e-d232b60f871a" path="/var/lib/kubelet/pods/3bb4950f-050d-42b5-a10e-d232b60f871a/volumes"
Sep 18 19:55:48.217604 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.217569    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"]
Sep 18 19:55:48.218126 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.218107    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bb4950f-050d-42b5-a10e-d232b60f871a" containerName="istio-proxy"
Sep 18 19:55:48.218205 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.218130    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bb4950f-050d-42b5-a10e-d232b60f871a" containerName="istio-proxy"
Sep 18 19:55:48.218270 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.218256    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bb4950f-050d-42b5-a10e-d232b60f871a" containerName="istio-proxy"
Sep 18 19:55:48.224213 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.224192    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"
Sep 18 19:55:48.226809 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.226781    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 18 19:55:48.226809 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.226801    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:55:48.227012 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.226781    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 18 19:55:48.227012 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.226854    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Sep 18 19:55:48.227012 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.226802    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-wgnmt\""
Sep 18 19:55:48.227200 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.227172    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 18 19:55:48.232934 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.232914    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"]
Sep 18 19:55:48.267051 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.267024    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57068a42-1966-4cbc-bf6b-74f51dc7e96c-model-cache\") pod \"auth-enabled-test-kserve-764896fcbf-ft6tz\" (UID: \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"
Sep 18 19:55:48.267210 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.267060    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57068a42-1966-4cbc-bf6b-74f51dc7e96c-tls-certs\") pod \"auth-enabled-test-kserve-764896fcbf-ft6tz\" (UID: \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"
Sep 18 19:55:48.267210 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.267145    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57068a42-1966-4cbc-bf6b-74f51dc7e96c-dshm\") pod \"auth-enabled-test-kserve-764896fcbf-ft6tz\" (UID: \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"
Sep 18 19:55:48.267210 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.267169    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57068a42-1966-4cbc-bf6b-74f51dc7e96c-kserve-provision-location\") pod \"auth-enabled-test-kserve-764896fcbf-ft6tz\" (UID: \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"
Sep 18 19:55:48.267210 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.267208    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57068a42-1966-4cbc-bf6b-74f51dc7e96c-home\") pod \"auth-enabled-test-kserve-764896fcbf-ft6tz\" (UID: \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"
Sep 18 19:55:48.267391 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.267245    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57068a42-1966-4cbc-bf6b-74f51dc7e96c-tmp-dir\") pod \"auth-enabled-test-kserve-764896fcbf-ft6tz\" (UID: \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"
Sep 18 19:55:48.267391 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.267260    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57068a42-1966-4cbc-bf6b-74f51dc7e96c-cabundle-cert\") pod \"auth-enabled-test-kserve-764896fcbf-ft6tz\" (UID: \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"
Sep 18 19:55:48.267391 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.267319    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8tzt\" (UniqueName: \"kubernetes.io/projected/57068a42-1966-4cbc-bf6b-74f51dc7e96c-kube-api-access-p8tzt\") pod \"auth-enabled-test-kserve-764896fcbf-ft6tz\" (UID: \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"
Sep 18 19:55:48.368210 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.368173    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57068a42-1966-4cbc-bf6b-74f51dc7e96c-tmp-dir\") pod \"auth-enabled-test-kserve-764896fcbf-ft6tz\" (UID: \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"
Sep 18 19:55:48.368210 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.368208    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57068a42-1966-4cbc-bf6b-74f51dc7e96c-cabundle-cert\") pod \"auth-enabled-test-kserve-764896fcbf-ft6tz\" (UID: \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"
Sep 18 19:55:48.368473 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.368258    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8tzt\" (UniqueName: \"kubernetes.io/projected/57068a42-1966-4cbc-bf6b-74f51dc7e96c-kube-api-access-p8tzt\") pod \"auth-enabled-test-kserve-764896fcbf-ft6tz\" (UID: \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"
Sep 18 19:55:48.368473 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.368281    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57068a42-1966-4cbc-bf6b-74f51dc7e96c-model-cache\") pod \"auth-enabled-test-kserve-764896fcbf-ft6tz\" (UID: \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"
Sep 18 19:55:48.368473 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.368298    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57068a42-1966-4cbc-bf6b-74f51dc7e96c-tls-certs\") pod \"auth-enabled-test-kserve-764896fcbf-ft6tz\" (UID: \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"
Sep 18 19:55:48.368473 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.368337    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57068a42-1966-4cbc-bf6b-74f51dc7e96c-dshm\") pod \"auth-enabled-test-kserve-764896fcbf-ft6tz\" (UID: \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"
Sep 18 19:55:48.368473 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.368354    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57068a42-1966-4cbc-bf6b-74f51dc7e96c-kserve-provision-location\") pod \"auth-enabled-test-kserve-764896fcbf-ft6tz\" (UID: \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"
Sep 18 19:55:48.368473 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.368388    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57068a42-1966-4cbc-bf6b-74f51dc7e96c-home\") pod \"auth-enabled-test-kserve-764896fcbf-ft6tz\" (UID: \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"
Sep 18 19:55:48.368767 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.368671    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57068a42-1966-4cbc-bf6b-74f51dc7e96c-tmp-dir\") pod \"auth-enabled-test-kserve-764896fcbf-ft6tz\" (UID: \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"
Sep 18 19:55:48.368767 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.368711    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57068a42-1966-4cbc-bf6b-74f51dc7e96c-model-cache\") pod \"auth-enabled-test-kserve-764896fcbf-ft6tz\" (UID: \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"
Sep 18 19:55:48.369080 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.369054    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57068a42-1966-4cbc-bf6b-74f51dc7e96c-kserve-provision-location\") pod \"auth-enabled-test-kserve-764896fcbf-ft6tz\" (UID: \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"
Sep 18 19:55:48.369359 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.369067    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57068a42-1966-4cbc-bf6b-74f51dc7e96c-home\") pod \"auth-enabled-test-kserve-764896fcbf-ft6tz\" (UID: \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"
Sep 18 19:55:48.369359 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.369155    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57068a42-1966-4cbc-bf6b-74f51dc7e96c-cabundle-cert\") pod \"auth-enabled-test-kserve-764896fcbf-ft6tz\" (UID: \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"
Sep 18 19:55:48.370866 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.370813    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57068a42-1966-4cbc-bf6b-74f51dc7e96c-dshm\") pod \"auth-enabled-test-kserve-764896fcbf-ft6tz\" (UID: \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"
Sep 18 19:55:48.371108 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.371093    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57068a42-1966-4cbc-bf6b-74f51dc7e96c-tls-certs\") pod \"auth-enabled-test-kserve-764896fcbf-ft6tz\" (UID: \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"
Sep 18 19:55:48.376898 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.376876    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8tzt\" (UniqueName: \"kubernetes.io/projected/57068a42-1966-4cbc-bf6b-74f51dc7e96c-kube-api-access-p8tzt\") pod \"auth-enabled-test-kserve-764896fcbf-ft6tz\" (UID: \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"
Sep 18 19:55:48.536555 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.536522    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"
Sep 18 19:55:48.816462 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.816428    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"]
Sep 18 19:55:48.816681 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:55:48.816658    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57068a42_1966_4cbc_bf6b_74f51dc7e96c.slice/crio-9da238a569131068617e09bad23f6b87e82fe0f96797c36965c9796c8b60fa58 WatchSource:0}: Error finding container 9da238a569131068617e09bad23f6b87e82fe0f96797c36965c9796c8b60fa58: Status 404 returned error can't find the container with id 9da238a569131068617e09bad23f6b87e82fe0f96797c36965c9796c8b60fa58
Sep 18 19:55:48.846039 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:48.846009    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz" event={"ID":"57068a42-1966-4cbc-bf6b-74f51dc7e96c","Type":"ContainerStarted","Data":"9da238a569131068617e09bad23f6b87e82fe0f96797c36965c9796c8b60fa58"}
Sep 18 19:55:49.852579 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:49.852546    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz" event={"ID":"57068a42-1966-4cbc-bf6b-74f51dc7e96c","Type":"ContainerStarted","Data":"df34f5bb8fdc6b60400fd36520df846a2428070b209cf5290284047cc16c1926"}
Sep 18 19:55:51.866264 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:51.866225    2581 generic.go:356] "Generic (PLEG): container finished" podID="57068a42-1966-4cbc-bf6b-74f51dc7e96c" containerID="df34f5bb8fdc6b60400fd36520df846a2428070b209cf5290284047cc16c1926" exitCode=0
Sep 18 19:55:51.866712 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:51.866265    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz" event={"ID":"57068a42-1966-4cbc-bf6b-74f51dc7e96c","Type":"ContainerDied","Data":"df34f5bb8fdc6b60400fd36520df846a2428070b209cf5290284047cc16c1926"}
Sep 18 19:55:52.875028 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:52.874993    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz" event={"ID":"57068a42-1966-4cbc-bf6b-74f51dc7e96c","Type":"ContainerStarted","Data":"629fe114b387d8d84938d813039f532892cb806bfc44eb34d6d9129e325db009"}
Sep 18 19:55:52.895442 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:52.895379    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz" podStartSLOduration=2.534164919 podStartE2EDuration="4.895355427s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.361190508s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:55:52.894258641 +0000 UTC m=+1075.996546978" watchObservedRunningTime="2026-09-18 19:55:52.895355427 +0000 UTC m=+1075.997643764"
Sep 18 19:55:53.536987 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:53.536952    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"
Sep 18 19:55:58.536865 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:58.536808    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"
Sep 18 19:55:58.538170 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:55:58.538137    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz" podUID="57068a42-1966-4cbc-bf6b-74f51dc7e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 18 19:56:08.537915 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:08.537864    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz" podUID="57068a42-1966-4cbc-bf6b-74f51dc7e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 18 19:56:13.427639 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:13.427595    2581 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" secret="" err="secret \"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-nlqwt\" not found"
Sep 18 19:56:13.507916 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:56:13.507879    2581 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 18 19:56:13.508140 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:56:13.507960    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79c2bc8d-19c2-463e-a801-e2392e400b8a-tls-certs podName:79c2bc8d-19c2-463e-a801-e2392e400b8a nodeName:}" failed. No retries permitted until 2026-09-18 19:56:14.007939499 +0000 UTC m=+1097.110227811 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/79c2bc8d-19c2-463e-a801-e2392e400b8a-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "79c2bc8d-19c2-463e-a801-e2392e400b8a") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 18 19:56:14.011652 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:56:14.011612    2581 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 18 19:56:14.011927 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:56:14.011689    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79c2bc8d-19c2-463e-a801-e2392e400b8a-tls-certs podName:79c2bc8d-19c2-463e-a801-e2392e400b8a nodeName:}" failed. No retries permitted until 2026-09-18 19:56:15.011674474 +0000 UTC m=+1098.113962780 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/79c2bc8d-19c2-463e-a801-e2392e400b8a-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "79c2bc8d-19c2-463e-a801-e2392e400b8a") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 18 19:56:15.020561 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:56:15.020520    2581 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 18 19:56:15.020990 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:56:15.020616    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79c2bc8d-19c2-463e-a801-e2392e400b8a-tls-certs podName:79c2bc8d-19c2-463e-a801-e2392e400b8a nodeName:}" failed. No retries permitted until 2026-09-18 19:56:17.020593706 +0000 UTC m=+1100.122882015 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/79c2bc8d-19c2-463e-a801-e2392e400b8a-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "79c2bc8d-19c2-463e-a801-e2392e400b8a") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 18 19:56:17.041130 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:56:17.041099    2581 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 18 19:56:17.041511 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:56:17.041163    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79c2bc8d-19c2-463e-a801-e2392e400b8a-tls-certs podName:79c2bc8d-19c2-463e-a801-e2392e400b8a nodeName:}" failed. No retries permitted until 2026-09-18 19:56:21.041149141 +0000 UTC m=+1104.143437447 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/79c2bc8d-19c2-463e-a801-e2392e400b8a-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "79c2bc8d-19c2-463e-a801-e2392e400b8a") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 18 19:56:18.537963 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:18.537913    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz" podUID="57068a42-1966-4cbc-bf6b-74f51dc7e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 18 19:56:19.348472 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:19.348432    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 18 19:56:21.085373 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:56:21.085337    2581 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 18 19:56:21.085816 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:56:21.085413    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79c2bc8d-19c2-463e-a801-e2392e400b8a-tls-certs podName:79c2bc8d-19c2-463e-a801-e2392e400b8a nodeName:}" failed. No retries permitted until 2026-09-18 19:56:29.085398348 +0000 UTC m=+1112.187686653 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/79c2bc8d-19c2-463e-a801-e2392e400b8a-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "79c2bc8d-19c2-463e-a801-e2392e400b8a") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 18 19:56:28.537707 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:28.537667    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz" podUID="57068a42-1966-4cbc-bf6b-74f51dc7e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 18 19:56:29.153211 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:56:29.153178    2581 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 18 19:56:29.153466 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:56:29.153242    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79c2bc8d-19c2-463e-a801-e2392e400b8a-tls-certs podName:79c2bc8d-19c2-463e-a801-e2392e400b8a nodeName:}" failed. No retries permitted until 2026-09-18 19:56:45.153228936 +0000 UTC m=+1128.255517241 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/79c2bc8d-19c2-463e-a801-e2392e400b8a-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "79c2bc8d-19c2-463e-a801-e2392e400b8a") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 18 19:56:34.371706 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:34.371652    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="79c2bc8d-19c2-463e-a801-e2392e400b8a" containerName="main" containerID="cri-o://c61223b35ce1439e8eb458a06555d8856591913c1656f329396ec7d0f524821a" gracePeriod=45
Sep 18 19:56:35.117793 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.117761    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 19:56:35.148273 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.148193    2581 generic.go:356] "Generic (PLEG): container finished" podID="79c2bc8d-19c2-463e-a801-e2392e400b8a" containerID="c61223b35ce1439e8eb458a06555d8856591913c1656f329396ec7d0f524821a" exitCode=0
Sep 18 19:56:35.148273 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.148252    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"79c2bc8d-19c2-463e-a801-e2392e400b8a","Type":"ContainerDied","Data":"c61223b35ce1439e8eb458a06555d8856591913c1656f329396ec7d0f524821a"}
Sep 18 19:56:35.148498 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.148311    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"79c2bc8d-19c2-463e-a801-e2392e400b8a","Type":"ContainerDied","Data":"be803d8280055ee2d5286c9b983000a3e43729288b63de7fc7e8c42b438efef6"}
Sep 18 19:56:35.148498 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.148329    2581 scope.go:117] "RemoveContainer" containerID="c61223b35ce1439e8eb458a06555d8856591913c1656f329396ec7d0f524821a"
Sep 18 19:56:35.148498 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.148314    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 19:56:35.158764 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.158737    2581 scope.go:117] "RemoveContainer" containerID="83acce648423d0cddc50a039ab23e7ab7a102e213b4fb0cd32578cdbed3a82f2"
Sep 18 19:56:35.167410 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.167393    2581 scope.go:117] "RemoveContainer" containerID="c61223b35ce1439e8eb458a06555d8856591913c1656f329396ec7d0f524821a"
Sep 18 19:56:35.167708 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:56:35.167686    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c61223b35ce1439e8eb458a06555d8856591913c1656f329396ec7d0f524821a\": container with ID starting with c61223b35ce1439e8eb458a06555d8856591913c1656f329396ec7d0f524821a not found: ID does not exist" containerID="c61223b35ce1439e8eb458a06555d8856591913c1656f329396ec7d0f524821a"
Sep 18 19:56:35.167795 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.167727    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c61223b35ce1439e8eb458a06555d8856591913c1656f329396ec7d0f524821a"} err="failed to get container status \"c61223b35ce1439e8eb458a06555d8856591913c1656f329396ec7d0f524821a\": rpc error: code = NotFound desc = could not find container \"c61223b35ce1439e8eb458a06555d8856591913c1656f329396ec7d0f524821a\": container with ID starting with c61223b35ce1439e8eb458a06555d8856591913c1656f329396ec7d0f524821a not found: ID does not exist"
Sep 18 19:56:35.167795 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.167751    2581 scope.go:117] "RemoveContainer" containerID="83acce648423d0cddc50a039ab23e7ab7a102e213b4fb0cd32578cdbed3a82f2"
Sep 18 19:56:35.168044 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:56:35.168025    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83acce648423d0cddc50a039ab23e7ab7a102e213b4fb0cd32578cdbed3a82f2\": container with ID starting with 83acce648423d0cddc50a039ab23e7ab7a102e213b4fb0cd32578cdbed3a82f2 not found: ID does not exist" containerID="83acce648423d0cddc50a039ab23e7ab7a102e213b4fb0cd32578cdbed3a82f2"
Sep 18 19:56:35.168118 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.168052    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83acce648423d0cddc50a039ab23e7ab7a102e213b4fb0cd32578cdbed3a82f2"} err="failed to get container status \"83acce648423d0cddc50a039ab23e7ab7a102e213b4fb0cd32578cdbed3a82f2\": rpc error: code = NotFound desc = could not find container \"83acce648423d0cddc50a039ab23e7ab7a102e213b4fb0cd32578cdbed3a82f2\": container with ID starting with 83acce648423d0cddc50a039ab23e7ab7a102e213b4fb0cd32578cdbed3a82f2 not found: ID does not exist"
Sep 18 19:56:35.207516 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.207493    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79c2bc8d-19c2-463e-a801-e2392e400b8a-cabundle-cert\") pod \"79c2bc8d-19c2-463e-a801-e2392e400b8a\" (UID: \"79c2bc8d-19c2-463e-a801-e2392e400b8a\") "
Sep 18 19:56:35.207634 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.207530    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79c2bc8d-19c2-463e-a801-e2392e400b8a-home\") pod \"79c2bc8d-19c2-463e-a801-e2392e400b8a\" (UID: \"79c2bc8d-19c2-463e-a801-e2392e400b8a\") "
Sep 18 19:56:35.207634 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.207581    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79c2bc8d-19c2-463e-a801-e2392e400b8a-kserve-provision-location\") pod \"79c2bc8d-19c2-463e-a801-e2392e400b8a\" (UID: \"79c2bc8d-19c2-463e-a801-e2392e400b8a\") "
Sep 18 19:56:35.207634 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.207608    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79c2bc8d-19c2-463e-a801-e2392e400b8a-tmp-dir\") pod \"79c2bc8d-19c2-463e-a801-e2392e400b8a\" (UID: \"79c2bc8d-19c2-463e-a801-e2392e400b8a\") "
Sep 18 19:56:35.207634 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.207622    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79c2bc8d-19c2-463e-a801-e2392e400b8a-dshm\") pod \"79c2bc8d-19c2-463e-a801-e2392e400b8a\" (UID: \"79c2bc8d-19c2-463e-a801-e2392e400b8a\") "
Sep 18 19:56:35.207871 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.207641    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79c2bc8d-19c2-463e-a801-e2392e400b8a-model-cache\") pod \"79c2bc8d-19c2-463e-a801-e2392e400b8a\" (UID: \"79c2bc8d-19c2-463e-a801-e2392e400b8a\") "
Sep 18 19:56:35.207871 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.207667    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvbv8\" (UniqueName: \"kubernetes.io/projected/79c2bc8d-19c2-463e-a801-e2392e400b8a-kube-api-access-xvbv8\") pod \"79c2bc8d-19c2-463e-a801-e2392e400b8a\" (UID: \"79c2bc8d-19c2-463e-a801-e2392e400b8a\") "
Sep 18 19:56:35.207871 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.207740    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79c2bc8d-19c2-463e-a801-e2392e400b8a-tls-certs\") pod \"79c2bc8d-19c2-463e-a801-e2392e400b8a\" (UID: \"79c2bc8d-19c2-463e-a801-e2392e400b8a\") "
Sep 18 19:56:35.208035 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.207917    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c2bc8d-19c2-463e-a801-e2392e400b8a-model-cache" (OuterVolumeSpecName: "model-cache") pod "79c2bc8d-19c2-463e-a801-e2392e400b8a" (UID: "79c2bc8d-19c2-463e-a801-e2392e400b8a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:56:35.208035 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.207950    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c2bc8d-19c2-463e-a801-e2392e400b8a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "79c2bc8d-19c2-463e-a801-e2392e400b8a" (UID: "79c2bc8d-19c2-463e-a801-e2392e400b8a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:56:35.208141 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.208067    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79c2bc8d-19c2-463e-a801-e2392e400b8a-model-cache\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:56:35.208141 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.208084    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79c2bc8d-19c2-463e-a801-e2392e400b8a-cabundle-cert\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:56:35.208609 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.208583    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c2bc8d-19c2-463e-a801-e2392e400b8a-home" (OuterVolumeSpecName: "home") pod "79c2bc8d-19c2-463e-a801-e2392e400b8a" (UID: "79c2bc8d-19c2-463e-a801-e2392e400b8a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:56:35.210875 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.210817    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c2bc8d-19c2-463e-a801-e2392e400b8a-kube-api-access-xvbv8" (OuterVolumeSpecName: "kube-api-access-xvbv8") pod "79c2bc8d-19c2-463e-a801-e2392e400b8a" (UID: "79c2bc8d-19c2-463e-a801-e2392e400b8a"). InnerVolumeSpecName "kube-api-access-xvbv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:56:35.210875 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.210863    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79c2bc8d-19c2-463e-a801-e2392e400b8a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "79c2bc8d-19c2-463e-a801-e2392e400b8a" (UID: "79c2bc8d-19c2-463e-a801-e2392e400b8a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:56:35.211027 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.210863    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c2bc8d-19c2-463e-a801-e2392e400b8a-dshm" (OuterVolumeSpecName: "dshm") pod "79c2bc8d-19c2-463e-a801-e2392e400b8a" (UID: "79c2bc8d-19c2-463e-a801-e2392e400b8a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:56:35.219517 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.219494    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c2bc8d-19c2-463e-a801-e2392e400b8a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "79c2bc8d-19c2-463e-a801-e2392e400b8a" (UID: "79c2bc8d-19c2-463e-a801-e2392e400b8a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:56:35.239274 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.239237    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c2bc8d-19c2-463e-a801-e2392e400b8a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "79c2bc8d-19c2-463e-a801-e2392e400b8a" (UID: "79c2bc8d-19c2-463e-a801-e2392e400b8a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:56:35.309416 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.309372    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79c2bc8d-19c2-463e-a801-e2392e400b8a-kserve-provision-location\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:56:35.309416 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.309410    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79c2bc8d-19c2-463e-a801-e2392e400b8a-tmp-dir\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:56:35.309416 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.309420    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79c2bc8d-19c2-463e-a801-e2392e400b8a-dshm\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:56:35.309416 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.309429    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvbv8\" (UniqueName: \"kubernetes.io/projected/79c2bc8d-19c2-463e-a801-e2392e400b8a-kube-api-access-xvbv8\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:56:35.309667 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.309438    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79c2bc8d-19c2-463e-a801-e2392e400b8a-tls-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:56:35.309667 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.309447    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79c2bc8d-19c2-463e-a801-e2392e400b8a-home\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:56:35.464893 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.464860    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 18 19:56:35.469849 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:35.469812    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 18 19:56:37.433920 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:37.433887    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79c2bc8d-19c2-463e-a801-e2392e400b8a" path="/var/lib/kubelet/pods/79c2bc8d-19c2-463e-a801-e2392e400b8a/volumes"
Sep 18 19:56:38.537923 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:38.537878    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz" podUID="57068a42-1966-4cbc-bf6b-74f51dc7e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 18 19:56:48.537985 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:48.537938    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz" podUID="57068a42-1966-4cbc-bf6b-74f51dc7e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 18 19:56:58.537676 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:56:58.537634    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz" podUID="57068a42-1966-4cbc-bf6b-74f51dc7e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 18 19:57:08.537450 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:57:08.537407    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz" podUID="57068a42-1966-4cbc-bf6b-74f51dc7e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 18 19:57:18.537637 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:57:18.537589    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz" podUID="57068a42-1966-4cbc-bf6b-74f51dc7e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 18 19:57:28.536992 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:57:28.536947    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz" podUID="57068a42-1966-4cbc-bf6b-74f51dc7e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 18 19:57:38.537968 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:57:38.537912    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz" podUID="57068a42-1966-4cbc-bf6b-74f51dc7e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 18 19:57:48.546645 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:57:48.546615    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"
Sep 18 19:57:48.554984 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:57:48.554956    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"
Sep 18 19:57:53.838554 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:57:53.838516    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"]
Sep 18 19:58:08.861528 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:08.861467    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz" podUID="57068a42-1966-4cbc-bf6b-74f51dc7e96c" containerName="main" containerID="cri-o://629fe114b387d8d84938d813039f532892cb806bfc44eb34d6d9129e325db009" gracePeriod=45
Sep 18 19:58:09.608677 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.608650    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"
Sep 18 19:58:09.666690 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.666600    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57068a42-1966-4cbc-bf6b-74f51dc7e96c-tls-certs\") pod \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\" (UID: \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\") "
Sep 18 19:58:09.666690 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.666650    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57068a42-1966-4cbc-bf6b-74f51dc7e96c-cabundle-cert\") pod \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\" (UID: \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\") "
Sep 18 19:58:09.666690 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.666672    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57068a42-1966-4cbc-bf6b-74f51dc7e96c-dshm\") pod \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\" (UID: \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\") "
Sep 18 19:58:09.666690 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.666689    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8tzt\" (UniqueName: \"kubernetes.io/projected/57068a42-1966-4cbc-bf6b-74f51dc7e96c-kube-api-access-p8tzt\") pod \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\" (UID: \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\") "
Sep 18 19:58:09.667054 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.666870    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57068a42-1966-4cbc-bf6b-74f51dc7e96c-tmp-dir\") pod \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\" (UID: \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\") "
Sep 18 19:58:09.667054 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.666957    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57068a42-1966-4cbc-bf6b-74f51dc7e96c-model-cache\") pod \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\" (UID: \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\") "
Sep 18 19:58:09.667054 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.667040    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57068a42-1966-4cbc-bf6b-74f51dc7e96c-home\") pod \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\" (UID: \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\") "
Sep 18 19:58:09.667215 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.667081    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57068a42-1966-4cbc-bf6b-74f51dc7e96c-kserve-provision-location\") pod \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\" (UID: \"57068a42-1966-4cbc-bf6b-74f51dc7e96c\") "
Sep 18 19:58:09.667215 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.667098    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57068a42-1966-4cbc-bf6b-74f51dc7e96c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "57068a42-1966-4cbc-bf6b-74f51dc7e96c" (UID: "57068a42-1966-4cbc-bf6b-74f51dc7e96c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:58:09.667942 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.667907    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57068a42-1966-4cbc-bf6b-74f51dc7e96c-cabundle-cert\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:58:09.669165 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.668541    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57068a42-1966-4cbc-bf6b-74f51dc7e96c-home" (OuterVolumeSpecName: "home") pod "57068a42-1966-4cbc-bf6b-74f51dc7e96c" (UID: "57068a42-1966-4cbc-bf6b-74f51dc7e96c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:58:09.669362 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.669332    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57068a42-1966-4cbc-bf6b-74f51dc7e96c-model-cache" (OuterVolumeSpecName: "model-cache") pod "57068a42-1966-4cbc-bf6b-74f51dc7e96c" (UID: "57068a42-1966-4cbc-bf6b-74f51dc7e96c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:58:09.673879 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.672778    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57068a42-1966-4cbc-bf6b-74f51dc7e96c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "57068a42-1966-4cbc-bf6b-74f51dc7e96c" (UID: "57068a42-1966-4cbc-bf6b-74f51dc7e96c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:58:09.673879 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.672791    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57068a42-1966-4cbc-bf6b-74f51dc7e96c-dshm" (OuterVolumeSpecName: "dshm") pod "57068a42-1966-4cbc-bf6b-74f51dc7e96c" (UID: "57068a42-1966-4cbc-bf6b-74f51dc7e96c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:58:09.681490 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.681280    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57068a42-1966-4cbc-bf6b-74f51dc7e96c-kube-api-access-p8tzt" (OuterVolumeSpecName: "kube-api-access-p8tzt") pod "57068a42-1966-4cbc-bf6b-74f51dc7e96c" (UID: "57068a42-1966-4cbc-bf6b-74f51dc7e96c"). InnerVolumeSpecName "kube-api-access-p8tzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:58:09.684370 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.684343    2581 generic.go:356] "Generic (PLEG): container finished" podID="57068a42-1966-4cbc-bf6b-74f51dc7e96c" containerID="629fe114b387d8d84938d813039f532892cb806bfc44eb34d6d9129e325db009" exitCode=0
Sep 18 19:58:09.684485 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.684460    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz" event={"ID":"57068a42-1966-4cbc-bf6b-74f51dc7e96c","Type":"ContainerDied","Data":"629fe114b387d8d84938d813039f532892cb806bfc44eb34d6d9129e325db009"}
Sep 18 19:58:09.684566 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.684550    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz" event={"ID":"57068a42-1966-4cbc-bf6b-74f51dc7e96c","Type":"ContainerDied","Data":"9da238a569131068617e09bad23f6b87e82fe0f96797c36965c9796c8b60fa58"}
Sep 18 19:58:09.684635 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.684590    2581 scope.go:117] "RemoveContainer" containerID="629fe114b387d8d84938d813039f532892cb806bfc44eb34d6d9129e325db009"
Sep 18 19:58:09.684778 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.684762    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"
Sep 18 19:58:09.693451 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.693417    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57068a42-1966-4cbc-bf6b-74f51dc7e96c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "57068a42-1966-4cbc-bf6b-74f51dc7e96c" (UID: "57068a42-1966-4cbc-bf6b-74f51dc7e96c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:58:09.695038 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.695021    2581 scope.go:117] "RemoveContainer" containerID="df34f5bb8fdc6b60400fd36520df846a2428070b209cf5290284047cc16c1926"
Sep 18 19:58:09.697447 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.697422    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57068a42-1966-4cbc-bf6b-74f51dc7e96c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "57068a42-1966-4cbc-bf6b-74f51dc7e96c" (UID: "57068a42-1966-4cbc-bf6b-74f51dc7e96c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:58:09.703788 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.703769    2581 scope.go:117] "RemoveContainer" containerID="629fe114b387d8d84938d813039f532892cb806bfc44eb34d6d9129e325db009"
Sep 18 19:58:09.704119 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:58:09.704090    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"629fe114b387d8d84938d813039f532892cb806bfc44eb34d6d9129e325db009\": container with ID starting with 629fe114b387d8d84938d813039f532892cb806bfc44eb34d6d9129e325db009 not found: ID does not exist" containerID="629fe114b387d8d84938d813039f532892cb806bfc44eb34d6d9129e325db009"
Sep 18 19:58:09.704207 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.704117    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"629fe114b387d8d84938d813039f532892cb806bfc44eb34d6d9129e325db009"} err="failed to get container status \"629fe114b387d8d84938d813039f532892cb806bfc44eb34d6d9129e325db009\": rpc error: code = NotFound desc = could not find container \"629fe114b387d8d84938d813039f532892cb806bfc44eb34d6d9129e325db009\": container with ID starting with 629fe114b387d8d84938d813039f532892cb806bfc44eb34d6d9129e325db009 not found: ID does not exist"
Sep 18 19:58:09.704207 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.704135    2581 scope.go:117] "RemoveContainer" containerID="df34f5bb8fdc6b60400fd36520df846a2428070b209cf5290284047cc16c1926"
Sep 18 19:58:09.704398 ip-10-0-143-134 kubenswrapper[2581]: E0918 19:58:09.704379    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df34f5bb8fdc6b60400fd36520df846a2428070b209cf5290284047cc16c1926\": container with ID starting with df34f5bb8fdc6b60400fd36520df846a2428070b209cf5290284047cc16c1926 not found: ID does not exist" containerID="df34f5bb8fdc6b60400fd36520df846a2428070b209cf5290284047cc16c1926"
Sep 18 19:58:09.704461 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.704402    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df34f5bb8fdc6b60400fd36520df846a2428070b209cf5290284047cc16c1926"} err="failed to get container status \"df34f5bb8fdc6b60400fd36520df846a2428070b209cf5290284047cc16c1926\": rpc error: code = NotFound desc = could not find container \"df34f5bb8fdc6b60400fd36520df846a2428070b209cf5290284047cc16c1926\": container with ID starting with df34f5bb8fdc6b60400fd36520df846a2428070b209cf5290284047cc16c1926 not found: ID does not exist"
Sep 18 19:58:09.768688 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.768652    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57068a42-1966-4cbc-bf6b-74f51dc7e96c-dshm\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:58:09.768688 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.768682    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8tzt\" (UniqueName: \"kubernetes.io/projected/57068a42-1966-4cbc-bf6b-74f51dc7e96c-kube-api-access-p8tzt\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:58:09.768918 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.768716    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57068a42-1966-4cbc-bf6b-74f51dc7e96c-tmp-dir\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:58:09.768918 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.768725    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57068a42-1966-4cbc-bf6b-74f51dc7e96c-model-cache\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:58:09.768918 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.768733    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57068a42-1966-4cbc-bf6b-74f51dc7e96c-home\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:58:09.768918 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.768740    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57068a42-1966-4cbc-bf6b-74f51dc7e96c-kserve-provision-location\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:58:09.768918 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:09.768748    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57068a42-1966-4cbc-bf6b-74f51dc7e96c-tls-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 19:58:10.010642 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:10.010612    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"]
Sep 18 19:58:10.014246 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:10.014218    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz"]
Sep 18 19:58:10.548830 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:10.548794    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-764896fcbf-ft6tz" podUID="57068a42-1966-4cbc-bf6b-74f51dc7e96c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": context deadline exceeded"
Sep 18 19:58:11.432947 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:58:11.432916    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57068a42-1966-4cbc-bf6b-74f51dc7e96c" path="/var/lib/kubelet/pods/57068a42-1966-4cbc-bf6b-74f51dc7e96c/volumes"
Sep 18 19:59:01.017185 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.017157    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"]
Sep 18 19:59:01.017606 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.017514    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57068a42-1966-4cbc-bf6b-74f51dc7e96c" containerName="main"
Sep 18 19:59:01.017606 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.017525    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="57068a42-1966-4cbc-bf6b-74f51dc7e96c" containerName="main"
Sep 18 19:59:01.017606 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.017543    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79c2bc8d-19c2-463e-a801-e2392e400b8a" containerName="main"
Sep 18 19:59:01.017606 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.017548    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="79c2bc8d-19c2-463e-a801-e2392e400b8a" containerName="main"
Sep 18 19:59:01.017606 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.017558    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57068a42-1966-4cbc-bf6b-74f51dc7e96c" containerName="storage-initializer"
Sep 18 19:59:01.017606 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.017563    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="57068a42-1966-4cbc-bf6b-74f51dc7e96c" containerName="storage-initializer"
Sep 18 19:59:01.017606 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.017574    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79c2bc8d-19c2-463e-a801-e2392e400b8a" containerName="storage-initializer"
Sep 18 19:59:01.017606 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.017579    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="79c2bc8d-19c2-463e-a801-e2392e400b8a" containerName="storage-initializer"
Sep 18 19:59:01.017890 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.017645    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="57068a42-1966-4cbc-bf6b-74f51dc7e96c" containerName="main"
Sep 18 19:59:01.017890 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.017656    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="79c2bc8d-19c2-463e-a801-e2392e400b8a" containerName="main"
Sep 18 19:59:01.021185 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.021168    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"
Sep 18 19:59:01.023478 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.023456    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 18 19:59:01.023624 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.023539    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 18 19:59:01.023624 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.023606    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:59:01.023752 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.023640    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Sep 18 19:59:01.023752 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.023692    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-49p28\""
Sep 18 19:59:01.023994 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.023980    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 18 19:59:01.030700 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.030679    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"]
Sep 18 19:59:01.094340 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.094314    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1faf5ed6-eee9-41e1-918b-86deb144c0ba-home\") pod \"auth-invalid-token-test-kserve-855d876c96-4g6ks\" (UID: \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"
Sep 18 19:59:01.094507 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.094350    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1faf5ed6-eee9-41e1-918b-86deb144c0ba-model-cache\") pod \"auth-invalid-token-test-kserve-855d876c96-4g6ks\" (UID: \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"
Sep 18 19:59:01.094507 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.094381    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1faf5ed6-eee9-41e1-918b-86deb144c0ba-tls-certs\") pod \"auth-invalid-token-test-kserve-855d876c96-4g6ks\" (UID: \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"
Sep 18 19:59:01.094507 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.094446    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1faf5ed6-eee9-41e1-918b-86deb144c0ba-cabundle-cert\") pod \"auth-invalid-token-test-kserve-855d876c96-4g6ks\" (UID: \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"
Sep 18 19:59:01.094507 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.094484    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pglht\" (UniqueName: \"kubernetes.io/projected/1faf5ed6-eee9-41e1-918b-86deb144c0ba-kube-api-access-pglht\") pod \"auth-invalid-token-test-kserve-855d876c96-4g6ks\" (UID: \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"
Sep 18 19:59:01.094704 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.094517    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1faf5ed6-eee9-41e1-918b-86deb144c0ba-dshm\") pod \"auth-invalid-token-test-kserve-855d876c96-4g6ks\" (UID: \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"
Sep 18 19:59:01.094704 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.094532    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1faf5ed6-eee9-41e1-918b-86deb144c0ba-tmp-dir\") pod \"auth-invalid-token-test-kserve-855d876c96-4g6ks\" (UID: \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"
Sep 18 19:59:01.094704 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.094590    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1faf5ed6-eee9-41e1-918b-86deb144c0ba-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-855d876c96-4g6ks\" (UID: \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"
Sep 18 19:59:01.195178 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.195144    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1faf5ed6-eee9-41e1-918b-86deb144c0ba-dshm\") pod \"auth-invalid-token-test-kserve-855d876c96-4g6ks\" (UID: \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"
Sep 18 19:59:01.195178 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.195177    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1faf5ed6-eee9-41e1-918b-86deb144c0ba-tmp-dir\") pod \"auth-invalid-token-test-kserve-855d876c96-4g6ks\" (UID: \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"
Sep 18 19:59:01.195379 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.195202    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1faf5ed6-eee9-41e1-918b-86deb144c0ba-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-855d876c96-4g6ks\" (UID: \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"
Sep 18 19:59:01.195379 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.195309    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1faf5ed6-eee9-41e1-918b-86deb144c0ba-home\") pod \"auth-invalid-token-test-kserve-855d876c96-4g6ks\" (UID: \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"
Sep 18 19:59:01.195379 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.195344    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1faf5ed6-eee9-41e1-918b-86deb144c0ba-model-cache\") pod \"auth-invalid-token-test-kserve-855d876c96-4g6ks\" (UID: \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"
Sep 18 19:59:01.195379 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.195376    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1faf5ed6-eee9-41e1-918b-86deb144c0ba-tls-certs\") pod \"auth-invalid-token-test-kserve-855d876c96-4g6ks\" (UID: \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"
Sep 18 19:59:01.195586 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.195418    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1faf5ed6-eee9-41e1-918b-86deb144c0ba-cabundle-cert\") pod \"auth-invalid-token-test-kserve-855d876c96-4g6ks\" (UID: \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"
Sep 18 19:59:01.195586 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.195438    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pglht\" (UniqueName: \"kubernetes.io/projected/1faf5ed6-eee9-41e1-918b-86deb144c0ba-kube-api-access-pglht\") pod \"auth-invalid-token-test-kserve-855d876c96-4g6ks\" (UID: \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"
Sep 18 19:59:01.195692 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.195646    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1faf5ed6-eee9-41e1-918b-86deb144c0ba-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-855d876c96-4g6ks\" (UID: \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"
Sep 18 19:59:01.195778 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.195759    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1faf5ed6-eee9-41e1-918b-86deb144c0ba-home\") pod \"auth-invalid-token-test-kserve-855d876c96-4g6ks\" (UID: \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"
Sep 18 19:59:01.195872 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.195805    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1faf5ed6-eee9-41e1-918b-86deb144c0ba-model-cache\") pod \"auth-invalid-token-test-kserve-855d876c96-4g6ks\" (UID: \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"
Sep 18 19:59:01.196012 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.195990    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1faf5ed6-eee9-41e1-918b-86deb144c0ba-tmp-dir\") pod \"auth-invalid-token-test-kserve-855d876c96-4g6ks\" (UID: \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"
Sep 18 19:59:01.196236 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.196219    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1faf5ed6-eee9-41e1-918b-86deb144c0ba-cabundle-cert\") pod \"auth-invalid-token-test-kserve-855d876c96-4g6ks\" (UID: \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"
Sep 18 19:59:01.197598 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.197578    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1faf5ed6-eee9-41e1-918b-86deb144c0ba-dshm\") pod \"auth-invalid-token-test-kserve-855d876c96-4g6ks\" (UID: \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"
Sep 18 19:59:01.197944 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.197929    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1faf5ed6-eee9-41e1-918b-86deb144c0ba-tls-certs\") pod \"auth-invalid-token-test-kserve-855d876c96-4g6ks\" (UID: \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"
Sep 18 19:59:01.202927 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.202895    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pglht\" (UniqueName: \"kubernetes.io/projected/1faf5ed6-eee9-41e1-918b-86deb144c0ba-kube-api-access-pglht\") pod \"auth-invalid-token-test-kserve-855d876c96-4g6ks\" (UID: \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"
Sep 18 19:59:01.332394 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.332366    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"
Sep 18 19:59:01.606860 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.606817    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"]
Sep 18 19:59:01.608045 ip-10-0-143-134 kubenswrapper[2581]: W0918 19:59:01.608021    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1faf5ed6_eee9_41e1_918b_86deb144c0ba.slice/crio-a88051666b566604ffa581dc2390684c63b0cadb43ddf40e1d9d6922a67467a5 WatchSource:0}: Error finding container a88051666b566604ffa581dc2390684c63b0cadb43ddf40e1d9d6922a67467a5: Status 404 returned error can't find the container with id a88051666b566604ffa581dc2390684c63b0cadb43ddf40e1d9d6922a67467a5
Sep 18 19:59:01.961305 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.961211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks" event={"ID":"1faf5ed6-eee9-41e1-918b-86deb144c0ba","Type":"ContainerStarted","Data":"50b972a15746f542c5e491f26fe08265a0ac792c9d6a24bca8adf2b2795a8e80"}
Sep 18 19:59:01.961305 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:01.961275    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks" event={"ID":"1faf5ed6-eee9-41e1-918b-86deb144c0ba","Type":"ContainerStarted","Data":"a88051666b566604ffa581dc2390684c63b0cadb43ddf40e1d9d6922a67467a5"}
Sep 18 19:59:03.973687 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:03.973653    2581 generic.go:356] "Generic (PLEG): container finished" podID="1faf5ed6-eee9-41e1-918b-86deb144c0ba" containerID="50b972a15746f542c5e491f26fe08265a0ac792c9d6a24bca8adf2b2795a8e80" exitCode=0
Sep 18 19:59:03.974060 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:03.973703    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks" event={"ID":"1faf5ed6-eee9-41e1-918b-86deb144c0ba","Type":"ContainerDied","Data":"50b972a15746f542c5e491f26fe08265a0ac792c9d6a24bca8adf2b2795a8e80"}
Sep 18 19:59:04.981496 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:04.981461    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks" event={"ID":"1faf5ed6-eee9-41e1-918b-86deb144c0ba","Type":"ContainerStarted","Data":"0d47cad610250f76b461c8fe70a04ff44ad436fd622cad29764974139af895e1"}
Sep 18 19:59:05.003287 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:05.003230    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks" podStartSLOduration=2.868271594 podStartE2EDuration="5.003216047s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.134944453s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:59:05.001100005 +0000 UTC m=+1268.103388333" watchObservedRunningTime="2026-09-18 19:59:05.003216047 +0000 UTC m=+1268.105504375"
Sep 18 19:59:05.332607 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:05.332573    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"
Sep 18 19:59:11.333266 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:11.333220    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"
Sep 18 19:59:11.334764 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:11.334737    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks" podUID="1faf5ed6-eee9-41e1-918b-86deb144c0ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 18 19:59:21.332865 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:21.332795    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks" podUID="1faf5ed6-eee9-41e1-918b-86deb144c0ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 18 19:59:31.332779 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:31.332737    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks" podUID="1faf5ed6-eee9-41e1-918b-86deb144c0ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 18 19:59:41.333048 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:41.333011    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks" podUID="1faf5ed6-eee9-41e1-918b-86deb144c0ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 18 19:59:51.333393 ip-10-0-143-134 kubenswrapper[2581]: I0918 19:59:51.333343    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks" podUID="1faf5ed6-eee9-41e1-918b-86deb144c0ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 18 20:00:01.333881 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:00:01.333810    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks" podUID="1faf5ed6-eee9-41e1-918b-86deb144c0ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 18 20:00:11.332856 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:00:11.332799    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks" podUID="1faf5ed6-eee9-41e1-918b-86deb144c0ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 18 20:00:21.333115 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:00:21.333027    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks" podUID="1faf5ed6-eee9-41e1-918b-86deb144c0ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 18 20:00:31.333498 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:00:31.333453    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks" podUID="1faf5ed6-eee9-41e1-918b-86deb144c0ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 18 20:00:41.333340 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:00:41.333291    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks" podUID="1faf5ed6-eee9-41e1-918b-86deb144c0ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 18 20:00:51.332895 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:00:51.332824    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks" podUID="1faf5ed6-eee9-41e1-918b-86deb144c0ba" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 18 20:01:01.342938 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:01.342908    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"
Sep 18 20:01:01.350817 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:01.350791    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"
Sep 18 20:01:03.223328 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:03.223289    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"]
Sep 18 20:01:18.249297 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:18.249255    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks" podUID="1faf5ed6-eee9-41e1-918b-86deb144c0ba" containerName="main" containerID="cri-o://0d47cad610250f76b461c8fe70a04ff44ad436fd622cad29764974139af895e1" gracePeriod=45
Sep 18 20:01:18.889272 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:18.889247    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"
Sep 18 20:01:19.025633 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.025608    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1faf5ed6-eee9-41e1-918b-86deb144c0ba-kserve-provision-location\") pod \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\" (UID: \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\") "
Sep 18 20:01:19.025784 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.025659    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1faf5ed6-eee9-41e1-918b-86deb144c0ba-home\") pod \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\" (UID: \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\") "
Sep 18 20:01:19.025784 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.025698    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1faf5ed6-eee9-41e1-918b-86deb144c0ba-tls-certs\") pod \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\" (UID: \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\") "
Sep 18 20:01:19.025784 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.025723    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1faf5ed6-eee9-41e1-918b-86deb144c0ba-tmp-dir\") pod \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\" (UID: \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\") "
Sep 18 20:01:19.025784 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.025746    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1faf5ed6-eee9-41e1-918b-86deb144c0ba-dshm\") pod \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\" (UID: \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\") "
Sep 18 20:01:19.025784 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.025774    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1faf5ed6-eee9-41e1-918b-86deb144c0ba-model-cache\") pod \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\" (UID: \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\") "
Sep 18 20:01:19.026056 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.025874    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pglht\" (UniqueName: \"kubernetes.io/projected/1faf5ed6-eee9-41e1-918b-86deb144c0ba-kube-api-access-pglht\") pod \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\" (UID: \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\") "
Sep 18 20:01:19.026056 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.025910    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1faf5ed6-eee9-41e1-918b-86deb144c0ba-cabundle-cert\") pod \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\" (UID: \"1faf5ed6-eee9-41e1-918b-86deb144c0ba\") "
Sep 18 20:01:19.026517 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.026411    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1faf5ed6-eee9-41e1-918b-86deb144c0ba-model-cache" (OuterVolumeSpecName: "model-cache") pod "1faf5ed6-eee9-41e1-918b-86deb144c0ba" (UID: "1faf5ed6-eee9-41e1-918b-86deb144c0ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:01:19.026517 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.026461    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1faf5ed6-eee9-41e1-918b-86deb144c0ba-home" (OuterVolumeSpecName: "home") pod "1faf5ed6-eee9-41e1-918b-86deb144c0ba" (UID: "1faf5ed6-eee9-41e1-918b-86deb144c0ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:01:19.026517 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.026481    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1faf5ed6-eee9-41e1-918b-86deb144c0ba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1faf5ed6-eee9-41e1-918b-86deb144c0ba" (UID: "1faf5ed6-eee9-41e1-918b-86deb144c0ba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:01:19.028306 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.028273    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1faf5ed6-eee9-41e1-918b-86deb144c0ba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1faf5ed6-eee9-41e1-918b-86deb144c0ba" (UID: "1faf5ed6-eee9-41e1-918b-86deb144c0ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:01:19.028736 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.028716    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1faf5ed6-eee9-41e1-918b-86deb144c0ba-dshm" (OuterVolumeSpecName: "dshm") pod "1faf5ed6-eee9-41e1-918b-86deb144c0ba" (UID: "1faf5ed6-eee9-41e1-918b-86deb144c0ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:01:19.028800 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.028709    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1faf5ed6-eee9-41e1-918b-86deb144c0ba-kube-api-access-pglht" (OuterVolumeSpecName: "kube-api-access-pglht") pod "1faf5ed6-eee9-41e1-918b-86deb144c0ba" (UID: "1faf5ed6-eee9-41e1-918b-86deb144c0ba"). InnerVolumeSpecName "kube-api-access-pglht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:01:19.044226 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.044194    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1faf5ed6-eee9-41e1-918b-86deb144c0ba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1faf5ed6-eee9-41e1-918b-86deb144c0ba" (UID: "1faf5ed6-eee9-41e1-918b-86deb144c0ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:01:19.056993 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.056970    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1faf5ed6-eee9-41e1-918b-86deb144c0ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1faf5ed6-eee9-41e1-918b-86deb144c0ba" (UID: "1faf5ed6-eee9-41e1-918b-86deb144c0ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:01:19.126745 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.126715    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pglht\" (UniqueName: \"kubernetes.io/projected/1faf5ed6-eee9-41e1-918b-86deb144c0ba-kube-api-access-pglht\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:01:19.126745 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.126742    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1faf5ed6-eee9-41e1-918b-86deb144c0ba-cabundle-cert\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:01:19.126745 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.126751    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1faf5ed6-eee9-41e1-918b-86deb144c0ba-kserve-provision-location\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:01:19.126969 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.126760    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1faf5ed6-eee9-41e1-918b-86deb144c0ba-home\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:01:19.126969 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.126769    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1faf5ed6-eee9-41e1-918b-86deb144c0ba-tls-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:01:19.126969 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.126778    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1faf5ed6-eee9-41e1-918b-86deb144c0ba-tmp-dir\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:01:19.126969 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.126785    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1faf5ed6-eee9-41e1-918b-86deb144c0ba-dshm\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:01:19.126969 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.126791    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1faf5ed6-eee9-41e1-918b-86deb144c0ba-model-cache\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:01:19.763604 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.763513    2581 generic.go:356] "Generic (PLEG): container finished" podID="1faf5ed6-eee9-41e1-918b-86deb144c0ba" containerID="0d47cad610250f76b461c8fe70a04ff44ad436fd622cad29764974139af895e1" exitCode=0
Sep 18 20:01:19.763604 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.763565    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks" event={"ID":"1faf5ed6-eee9-41e1-918b-86deb144c0ba","Type":"ContainerDied","Data":"0d47cad610250f76b461c8fe70a04ff44ad436fd622cad29764974139af895e1"}
Sep 18 20:01:19.763604 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.763572    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"
Sep 18 20:01:19.764157 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.763619    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks" event={"ID":"1faf5ed6-eee9-41e1-918b-86deb144c0ba","Type":"ContainerDied","Data":"a88051666b566604ffa581dc2390684c63b0cadb43ddf40e1d9d6922a67467a5"}
Sep 18 20:01:19.764157 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.763648    2581 scope.go:117] "RemoveContainer" containerID="0d47cad610250f76b461c8fe70a04ff44ad436fd622cad29764974139af895e1"
Sep 18 20:01:19.773084 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.773065    2581 scope.go:117] "RemoveContainer" containerID="50b972a15746f542c5e491f26fe08265a0ac792c9d6a24bca8adf2b2795a8e80"
Sep 18 20:01:19.781240 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.781219    2581 scope.go:117] "RemoveContainer" containerID="0d47cad610250f76b461c8fe70a04ff44ad436fd622cad29764974139af895e1"
Sep 18 20:01:19.781565 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:01:19.781537    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d47cad610250f76b461c8fe70a04ff44ad436fd622cad29764974139af895e1\": container with ID starting with 0d47cad610250f76b461c8fe70a04ff44ad436fd622cad29764974139af895e1 not found: ID does not exist" containerID="0d47cad610250f76b461c8fe70a04ff44ad436fd622cad29764974139af895e1"
Sep 18 20:01:19.781657 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.781571    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d47cad610250f76b461c8fe70a04ff44ad436fd622cad29764974139af895e1"} err="failed to get container status \"0d47cad610250f76b461c8fe70a04ff44ad436fd622cad29764974139af895e1\": rpc error: code = NotFound desc = could not find container \"0d47cad610250f76b461c8fe70a04ff44ad436fd622cad29764974139af895e1\": container with ID starting with 0d47cad610250f76b461c8fe70a04ff44ad436fd622cad29764974139af895e1 not found: ID does not exist"
Sep 18 20:01:19.781657 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.781589    2581 scope.go:117] "RemoveContainer" containerID="50b972a15746f542c5e491f26fe08265a0ac792c9d6a24bca8adf2b2795a8e80"
Sep 18 20:01:19.781922 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:01:19.781896    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50b972a15746f542c5e491f26fe08265a0ac792c9d6a24bca8adf2b2795a8e80\": container with ID starting with 50b972a15746f542c5e491f26fe08265a0ac792c9d6a24bca8adf2b2795a8e80 not found: ID does not exist" containerID="50b972a15746f542c5e491f26fe08265a0ac792c9d6a24bca8adf2b2795a8e80"
Sep 18 20:01:19.781972 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.781932    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50b972a15746f542c5e491f26fe08265a0ac792c9d6a24bca8adf2b2795a8e80"} err="failed to get container status \"50b972a15746f542c5e491f26fe08265a0ac792c9d6a24bca8adf2b2795a8e80\": rpc error: code = NotFound desc = could not find container \"50b972a15746f542c5e491f26fe08265a0ac792c9d6a24bca8adf2b2795a8e80\": container with ID starting with 50b972a15746f542c5e491f26fe08265a0ac792c9d6a24bca8adf2b2795a8e80 not found: ID does not exist"
Sep 18 20:01:19.782739 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.782723    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"]
Sep 18 20:01:19.785779 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:19.785757    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-855d876c96-4g6ks"]
Sep 18 20:01:21.433826 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:21.433791    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1faf5ed6-eee9-41e1-918b-86deb144c0ba" path="/var/lib/kubelet/pods/1faf5ed6-eee9-41e1-918b-86deb144c0ba/volumes"
Sep 18 20:01:52.936090 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:52.936000    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx"]
Sep 18 20:01:52.936743 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:52.936413    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1faf5ed6-eee9-41e1-918b-86deb144c0ba" containerName="storage-initializer"
Sep 18 20:01:52.936743 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:52.936428    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1faf5ed6-eee9-41e1-918b-86deb144c0ba" containerName="storage-initializer"
Sep 18 20:01:52.936743 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:52.936443    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1faf5ed6-eee9-41e1-918b-86deb144c0ba" containerName="main"
Sep 18 20:01:52.936743 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:52.936449    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1faf5ed6-eee9-41e1-918b-86deb144c0ba" containerName="main"
Sep 18 20:01:52.936743 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:52.936667    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1faf5ed6-eee9-41e1-918b-86deb144c0ba" containerName="main"
Sep 18 20:01:52.939760 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:52.939741    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx"
Sep 18 20:01:52.942161 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:52.942137    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 18 20:01:52.943309 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:52.943291    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-jh2w2\""
Sep 18 20:01:52.943309 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:52.943304    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 18 20:01:52.943484 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:52.943349    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-pwlm2\""
Sep 18 20:01:52.943484 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:52.943411    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 18 20:01:52.960575 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:52.960547    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx"]
Sep 18 20:01:53.006470 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:53.006440    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8fbh\" (UniqueName: \"kubernetes.io/projected/22a58839-443c-4001-82a2-90408c69efbe-kube-api-access-k8fbh\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx\" (UID: \"22a58839-443c-4001-82a2-90408c69efbe\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx"
Sep 18 20:01:53.006605 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:53.006586    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22a58839-443c-4001-82a2-90408c69efbe-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx\" (UID: \"22a58839-443c-4001-82a2-90408c69efbe\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx"
Sep 18 20:01:53.107513 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:53.107484    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8fbh\" (UniqueName: \"kubernetes.io/projected/22a58839-443c-4001-82a2-90408c69efbe-kube-api-access-k8fbh\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx\" (UID: \"22a58839-443c-4001-82a2-90408c69efbe\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx"
Sep 18 20:01:53.107627 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:53.107597    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22a58839-443c-4001-82a2-90408c69efbe-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx\" (UID: \"22a58839-443c-4001-82a2-90408c69efbe\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx"
Sep 18 20:01:53.110311 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:53.110286    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22a58839-443c-4001-82a2-90408c69efbe-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx\" (UID: \"22a58839-443c-4001-82a2-90408c69efbe\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx"
Sep 18 20:01:53.115252 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:53.115230    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8fbh\" (UniqueName: \"kubernetes.io/projected/22a58839-443c-4001-82a2-90408c69efbe-kube-api-access-k8fbh\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx\" (UID: \"22a58839-443c-4001-82a2-90408c69efbe\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx"
Sep 18 20:01:53.253750 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:53.253679    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx"
Sep 18 20:01:53.532647 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:53.532623    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx"]
Sep 18 20:01:53.534531 ip-10-0-143-134 kubenswrapper[2581]: W0918 20:01:53.534480    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a58839_443c_4001_82a2_90408c69efbe.slice/crio-a78d1f78eefe5b422a983bccf05a76672de482e3b8ddfe35651783a7312d8a68 WatchSource:0}: Error finding container a78d1f78eefe5b422a983bccf05a76672de482e3b8ddfe35651783a7312d8a68: Status 404 returned error can't find the container with id a78d1f78eefe5b422a983bccf05a76672de482e3b8ddfe35651783a7312d8a68
Sep 18 20:01:53.538189 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:53.537280    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:01:53.942416 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:53.942336    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx" event={"ID":"22a58839-443c-4001-82a2-90408c69efbe","Type":"ContainerStarted","Data":"a78d1f78eefe5b422a983bccf05a76672de482e3b8ddfe35651783a7312d8a68"}
Sep 18 20:01:55.955350 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:55.955320    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx" event={"ID":"22a58839-443c-4001-82a2-90408c69efbe","Type":"ContainerStarted","Data":"cd00ca2edb183cb99acacc8a6be0d98b5766062736b63948409db9d6383b379f"}
Sep 18 20:01:55.955788 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:55.955428    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx"
Sep 18 20:01:55.974044 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:01:55.974001    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx" podStartSLOduration=1.6314283139999999 podStartE2EDuration="3.973986106s" totalImagesPullingTime="2.342557792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:01:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:01:55.970717297 +0000 UTC m=+1439.073005626" watchObservedRunningTime="2026-09-18 20:01:55.973986106 +0000 UTC m=+1439.076274434"
Sep 18 20:02:10.891623 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:10.891594    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"]
Sep 18 20:02:10.896284 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:10.896265    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"
Sep 18 20:02:10.898809 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:10.898788    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 18 20:02:10.898931 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:10.898808    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 18 20:02:10.899060 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:10.899044    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:02:10.899143 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:10.899068    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 18 20:02:10.899143 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:10.899099    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 18 20:02:10.899274 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:10.899194    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-4hsxs\""
Sep 18 20:02:10.907929 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:10.907893    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"]
Sep 18 20:02:11.061907 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:11.061881    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f74a4198-a169-4e91-b049-e10b25dca82d-home\") pod \"auth-disabled-test-kserve-5fddfc79bc-lqhw9\" (UID: \"f74a4198-a169-4e91-b049-e10b25dca82d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"
Sep 18 20:02:11.062114 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:11.061913    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f74a4198-a169-4e91-b049-e10b25dca82d-tls-certs\") pod \"auth-disabled-test-kserve-5fddfc79bc-lqhw9\" (UID: \"f74a4198-a169-4e91-b049-e10b25dca82d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"
Sep 18 20:02:11.062114 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:11.061946    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f74a4198-a169-4e91-b049-e10b25dca82d-kserve-provision-location\") pod \"auth-disabled-test-kserve-5fddfc79bc-lqhw9\" (UID: \"f74a4198-a169-4e91-b049-e10b25dca82d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"
Sep 18 20:02:11.062114 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:11.062023    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqpz6\" (UniqueName: \"kubernetes.io/projected/f74a4198-a169-4e91-b049-e10b25dca82d-kube-api-access-zqpz6\") pod \"auth-disabled-test-kserve-5fddfc79bc-lqhw9\" (UID: \"f74a4198-a169-4e91-b049-e10b25dca82d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"
Sep 18 20:02:11.062114 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:11.062087    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f74a4198-a169-4e91-b049-e10b25dca82d-dshm\") pod \"auth-disabled-test-kserve-5fddfc79bc-lqhw9\" (UID: \"f74a4198-a169-4e91-b049-e10b25dca82d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"
Sep 18 20:02:11.062338 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:11.062118    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f74a4198-a169-4e91-b049-e10b25dca82d-cabundle-cert\") pod \"auth-disabled-test-kserve-5fddfc79bc-lqhw9\" (UID: \"f74a4198-a169-4e91-b049-e10b25dca82d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"
Sep 18 20:02:11.062338 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:11.062156    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f74a4198-a169-4e91-b049-e10b25dca82d-model-cache\") pod \"auth-disabled-test-kserve-5fddfc79bc-lqhw9\" (UID: \"f74a4198-a169-4e91-b049-e10b25dca82d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"
Sep 18 20:02:11.062338 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:11.062238    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f74a4198-a169-4e91-b049-e10b25dca82d-tmp-dir\") pod \"auth-disabled-test-kserve-5fddfc79bc-lqhw9\" (UID: \"f74a4198-a169-4e91-b049-e10b25dca82d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"
Sep 18 20:02:11.162781 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:11.162703    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f74a4198-a169-4e91-b049-e10b25dca82d-tmp-dir\") pod \"auth-disabled-test-kserve-5fddfc79bc-lqhw9\" (UID: \"f74a4198-a169-4e91-b049-e10b25dca82d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"
Sep 18 20:02:11.162781 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:11.162754    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f74a4198-a169-4e91-b049-e10b25dca82d-home\") pod \"auth-disabled-test-kserve-5fddfc79bc-lqhw9\" (UID: \"f74a4198-a169-4e91-b049-e10b25dca82d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"
Sep 18 20:02:11.163074 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:11.162782    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f74a4198-a169-4e91-b049-e10b25dca82d-tls-certs\") pod \"auth-disabled-test-kserve-5fddfc79bc-lqhw9\" (UID: \"f74a4198-a169-4e91-b049-e10b25dca82d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"
Sep 18 20:02:11.163074 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:11.162812    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f74a4198-a169-4e91-b049-e10b25dca82d-kserve-provision-location\") pod \"auth-disabled-test-kserve-5fddfc79bc-lqhw9\" (UID: \"f74a4198-a169-4e91-b049-e10b25dca82d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"
Sep 18 20:02:11.163074 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:11.162862    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqpz6\" (UniqueName: \"kubernetes.io/projected/f74a4198-a169-4e91-b049-e10b25dca82d-kube-api-access-zqpz6\") pod \"auth-disabled-test-kserve-5fddfc79bc-lqhw9\" (UID: \"f74a4198-a169-4e91-b049-e10b25dca82d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"
Sep 18 20:02:11.163216 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:11.163157    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f74a4198-a169-4e91-b049-e10b25dca82d-tmp-dir\") pod \"auth-disabled-test-kserve-5fddfc79bc-lqhw9\" (UID: \"f74a4198-a169-4e91-b049-e10b25dca82d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"
Sep 18 20:02:11.163216 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:11.163196    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f74a4198-a169-4e91-b049-e10b25dca82d-home\") pod \"auth-disabled-test-kserve-5fddfc79bc-lqhw9\" (UID: \"f74a4198-a169-4e91-b049-e10b25dca82d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"
Sep 18 20:02:11.163216 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:11.163208    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f74a4198-a169-4e91-b049-e10b25dca82d-dshm\") pod \"auth-disabled-test-kserve-5fddfc79bc-lqhw9\" (UID: \"f74a4198-a169-4e91-b049-e10b25dca82d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"
Sep 18 20:02:11.163374 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:11.163256    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f74a4198-a169-4e91-b049-e10b25dca82d-cabundle-cert\") pod \"auth-disabled-test-kserve-5fddfc79bc-lqhw9\" (UID: \"f74a4198-a169-4e91-b049-e10b25dca82d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"
Sep 18 20:02:11.163374 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:11.163291    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f74a4198-a169-4e91-b049-e10b25dca82d-model-cache\") pod \"auth-disabled-test-kserve-5fddfc79bc-lqhw9\" (UID: \"f74a4198-a169-4e91-b049-e10b25dca82d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"
Sep 18 20:02:11.163374 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:11.163324    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f74a4198-a169-4e91-b049-e10b25dca82d-kserve-provision-location\") pod \"auth-disabled-test-kserve-5fddfc79bc-lqhw9\" (UID: \"f74a4198-a169-4e91-b049-e10b25dca82d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"
Sep 18 20:02:11.163684 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:11.163657    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f74a4198-a169-4e91-b049-e10b25dca82d-model-cache\") pod \"auth-disabled-test-kserve-5fddfc79bc-lqhw9\" (UID: \"f74a4198-a169-4e91-b049-e10b25dca82d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"
Sep 18 20:02:11.164263 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:11.164239    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f74a4198-a169-4e91-b049-e10b25dca82d-cabundle-cert\") pod \"auth-disabled-test-kserve-5fddfc79bc-lqhw9\" (UID: \"f74a4198-a169-4e91-b049-e10b25dca82d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"
Sep 18 20:02:11.165550 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:11.165528    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f74a4198-a169-4e91-b049-e10b25dca82d-dshm\") pod \"auth-disabled-test-kserve-5fddfc79bc-lqhw9\" (UID: \"f74a4198-a169-4e91-b049-e10b25dca82d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"
Sep 18 20:02:11.165785 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:11.165767    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f74a4198-a169-4e91-b049-e10b25dca82d-tls-certs\") pod \"auth-disabled-test-kserve-5fddfc79bc-lqhw9\" (UID: \"f74a4198-a169-4e91-b049-e10b25dca82d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"
Sep 18 20:02:11.171679 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:11.171651    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqpz6\" (UniqueName: \"kubernetes.io/projected/f74a4198-a169-4e91-b049-e10b25dca82d-kube-api-access-zqpz6\") pod \"auth-disabled-test-kserve-5fddfc79bc-lqhw9\" (UID: \"f74a4198-a169-4e91-b049-e10b25dca82d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"
Sep 18 20:02:11.209487 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:11.209461    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"
Sep 18 20:02:11.485302 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:11.485272    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"]
Sep 18 20:02:11.485415 ip-10-0-143-134 kubenswrapper[2581]: W0918 20:02:11.485304    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf74a4198_a169_4e91_b049_e10b25dca82d.slice/crio-acedc3943b3e5a3b739bc0dd9951e3032eab4411541d268606cbc8f6d7698f69 WatchSource:0}: Error finding container acedc3943b3e5a3b739bc0dd9951e3032eab4411541d268606cbc8f6d7698f69: Status 404 returned error can't find the container with id acedc3943b3e5a3b739bc0dd9951e3032eab4411541d268606cbc8f6d7698f69
Sep 18 20:02:12.044253 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:12.044217    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9" event={"ID":"f74a4198-a169-4e91-b049-e10b25dca82d","Type":"ContainerStarted","Data":"b33486665834c02837d88e21aa0a058e934884186d2e7254fce43c58fb8326d6"}
Sep 18 20:02:12.044734 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:12.044277    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9" event={"ID":"f74a4198-a169-4e91-b049-e10b25dca82d","Type":"ContainerStarted","Data":"acedc3943b3e5a3b739bc0dd9951e3032eab4411541d268606cbc8f6d7698f69"}
Sep 18 20:02:14.058506 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:14.058470    2581 generic.go:356] "Generic (PLEG): container finished" podID="f74a4198-a169-4e91-b049-e10b25dca82d" containerID="b33486665834c02837d88e21aa0a058e934884186d2e7254fce43c58fb8326d6" exitCode=0
Sep 18 20:02:14.058915 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:14.058531    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9" event={"ID":"f74a4198-a169-4e91-b049-e10b25dca82d","Type":"ContainerDied","Data":"b33486665834c02837d88e21aa0a058e934884186d2e7254fce43c58fb8326d6"}
Sep 18 20:02:15.066276 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:15.066242    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9" event={"ID":"f74a4198-a169-4e91-b049-e10b25dca82d","Type":"ContainerStarted","Data":"654b74434fc271f51a7f09b88d2d4014ae43337b30cb9b8b26fdbe9ea17f55bc"}
Sep 18 20:02:15.088635 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:15.088580    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9" podStartSLOduration=2.9394037060000002 podStartE2EDuration="5.088563827s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.149160121s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:02:15.087412199 +0000 UTC m=+1458.189700527" watchObservedRunningTime="2026-09-18 20:02:15.088563827 +0000 UTC m=+1458.190852155"
Sep 18 20:02:15.209869 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:15.209823    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"
Sep 18 20:02:21.210213 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:21.210173    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"
Sep 18 20:02:21.211894 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:21.211866    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9" podUID="f74a4198-a169-4e91-b049-e10b25dca82d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Sep 18 20:02:26.962244 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:26.962205    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx" podUID="22a58839-443c-4001-82a2-90408c69efbe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:02:28.550010 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:28.549925    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx"]
Sep 18 20:02:28.551920 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:28.551886    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx" podUID="22a58839-443c-4001-82a2-90408c69efbe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:02:31.210583 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:31.210545    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9" podUID="f74a4198-a169-4e91-b049-e10b25dca82d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Sep 18 20:02:38.552429 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:38.552382    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx" podUID="22a58839-443c-4001-82a2-90408c69efbe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:02:41.209953 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:41.209910    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9" podUID="f74a4198-a169-4e91-b049-e10b25dca82d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Sep 18 20:02:43.574780 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:43.574739    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx" podUID="22a58839-443c-4001-82a2-90408c69efbe" containerName="main" containerID="cri-o://cd00ca2edb183cb99acacc8a6be0d98b5766062736b63948409db9d6383b379f" gracePeriod=45
Sep 18 20:02:48.552198 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:48.552159    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx" podUID="22a58839-443c-4001-82a2-90408c69efbe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:02:51.210592 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:51.210552    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9" podUID="f74a4198-a169-4e91-b049-e10b25dca82d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Sep 18 20:02:58.552466 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:02:58.552429    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx" podUID="22a58839-443c-4001-82a2-90408c69efbe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:03:01.210628 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:01.210581    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9" podUID="f74a4198-a169-4e91-b049-e10b25dca82d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Sep 18 20:03:08.553076 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:08.553034    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx" podUID="22a58839-443c-4001-82a2-90408c69efbe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:03:11.210599 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:11.210562    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9" podUID="f74a4198-a169-4e91-b049-e10b25dca82d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Sep 18 20:03:13.825212 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:13.825185    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx"
Sep 18 20:03:13.923891 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:13.923797    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22a58839-443c-4001-82a2-90408c69efbe-tls-certs\") pod \"22a58839-443c-4001-82a2-90408c69efbe\" (UID: \"22a58839-443c-4001-82a2-90408c69efbe\") "
Sep 18 20:03:13.923891 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:13.923877    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8fbh\" (UniqueName: \"kubernetes.io/projected/22a58839-443c-4001-82a2-90408c69efbe-kube-api-access-k8fbh\") pod \"22a58839-443c-4001-82a2-90408c69efbe\" (UID: \"22a58839-443c-4001-82a2-90408c69efbe\") "
Sep 18 20:03:13.926210 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:13.926180    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22a58839-443c-4001-82a2-90408c69efbe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "22a58839-443c-4001-82a2-90408c69efbe" (UID: "22a58839-443c-4001-82a2-90408c69efbe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:03:13.926210 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:13.926186    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a58839-443c-4001-82a2-90408c69efbe-kube-api-access-k8fbh" (OuterVolumeSpecName: "kube-api-access-k8fbh") pod "22a58839-443c-4001-82a2-90408c69efbe" (UID: "22a58839-443c-4001-82a2-90408c69efbe"). InnerVolumeSpecName "kube-api-access-k8fbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:03:14.025399 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:14.025367    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22a58839-443c-4001-82a2-90408c69efbe-tls-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:03:14.025399 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:14.025390    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8fbh\" (UniqueName: \"kubernetes.io/projected/22a58839-443c-4001-82a2-90408c69efbe-kube-api-access-k8fbh\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:03:14.405162 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:14.405124    2581 generic.go:356] "Generic (PLEG): container finished" podID="22a58839-443c-4001-82a2-90408c69efbe" containerID="cd00ca2edb183cb99acacc8a6be0d98b5766062736b63948409db9d6383b379f" exitCode=0
Sep 18 20:03:14.405326 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:14.405172    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx"
Sep 18 20:03:14.405326 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:14.405202    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx" event={"ID":"22a58839-443c-4001-82a2-90408c69efbe","Type":"ContainerDied","Data":"cd00ca2edb183cb99acacc8a6be0d98b5766062736b63948409db9d6383b379f"}
Sep 18 20:03:14.405326 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:14.405256    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx" event={"ID":"22a58839-443c-4001-82a2-90408c69efbe","Type":"ContainerDied","Data":"a78d1f78eefe5b422a983bccf05a76672de482e3b8ddfe35651783a7312d8a68"}
Sep 18 20:03:14.405326 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:14.405280    2581 scope.go:117] "RemoveContainer" containerID="cd00ca2edb183cb99acacc8a6be0d98b5766062736b63948409db9d6383b379f"
Sep 18 20:03:14.414844 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:14.414815    2581 scope.go:117] "RemoveContainer" containerID="cd00ca2edb183cb99acacc8a6be0d98b5766062736b63948409db9d6383b379f"
Sep 18 20:03:14.415113 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:03:14.415095    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd00ca2edb183cb99acacc8a6be0d98b5766062736b63948409db9d6383b379f\": container with ID starting with cd00ca2edb183cb99acacc8a6be0d98b5766062736b63948409db9d6383b379f not found: ID does not exist" containerID="cd00ca2edb183cb99acacc8a6be0d98b5766062736b63948409db9d6383b379f"
Sep 18 20:03:14.415220 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:14.415118    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd00ca2edb183cb99acacc8a6be0d98b5766062736b63948409db9d6383b379f"} err="failed to get container status \"cd00ca2edb183cb99acacc8a6be0d98b5766062736b63948409db9d6383b379f\": rpc error: code = NotFound desc = could not find container \"cd00ca2edb183cb99acacc8a6be0d98b5766062736b63948409db9d6383b379f\": container with ID starting with cd00ca2edb183cb99acacc8a6be0d98b5766062736b63948409db9d6383b379f not found: ID does not exist"
Sep 18 20:03:14.442473 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:14.442448    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx"]
Sep 18 20:03:14.463416 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:14.463393    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5d5kfx"]
Sep 18 20:03:15.439963 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:15.439927    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22a58839-443c-4001-82a2-90408c69efbe" path="/var/lib/kubelet/pods/22a58839-443c-4001-82a2-90408c69efbe/volumes"
Sep 18 20:03:21.210598 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:21.210495    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9" podUID="f74a4198-a169-4e91-b049-e10b25dca82d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Sep 18 20:03:24.132398 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:24.132358    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt"]
Sep 18 20:03:24.132924 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:24.132907    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22a58839-443c-4001-82a2-90408c69efbe" containerName="main"
Sep 18 20:03:24.133011 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:24.132927    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="22a58839-443c-4001-82a2-90408c69efbe" containerName="main"
Sep 18 20:03:24.133066 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:24.133037    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="22a58839-443c-4001-82a2-90408c69efbe" containerName="main"
Sep 18 20:03:24.137483 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:24.137463    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt"
Sep 18 20:03:24.139870 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:24.139821    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 18 20:03:24.139990 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:24.139853    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-wgvnw\""
Sep 18 20:03:24.140809 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:24.140788    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 18 20:03:24.140946 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:24.140928    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 18 20:03:24.141013 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:24.140940    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-hn5wn\""
Sep 18 20:03:24.145404 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:24.145385    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt"]
Sep 18 20:03:24.218730 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:24.218694    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cq9l\" (UniqueName: \"kubernetes.io/projected/98f478cc-52d9-4a63-9a98-c6a97592a958-kube-api-access-2cq9l\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt\" (UID: \"98f478cc-52d9-4a63-9a98-c6a97592a958\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt"
Sep 18 20:03:24.218924 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:24.218752    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/98f478cc-52d9-4a63-9a98-c6a97592a958-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt\" (UID: \"98f478cc-52d9-4a63-9a98-c6a97592a958\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt"
Sep 18 20:03:24.218924 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:24.218828    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/98f478cc-52d9-4a63-9a98-c6a97592a958-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt\" (UID: \"98f478cc-52d9-4a63-9a98-c6a97592a958\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt"
Sep 18 20:03:24.218924 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:24.218882    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98f478cc-52d9-4a63-9a98-c6a97592a958-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt\" (UID: \"98f478cc-52d9-4a63-9a98-c6a97592a958\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt"
Sep 18 20:03:24.319700 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:24.319670    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cq9l\" (UniqueName: \"kubernetes.io/projected/98f478cc-52d9-4a63-9a98-c6a97592a958-kube-api-access-2cq9l\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt\" (UID: \"98f478cc-52d9-4a63-9a98-c6a97592a958\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt"
Sep 18 20:03:24.319857 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:24.319730    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/98f478cc-52d9-4a63-9a98-c6a97592a958-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt\" (UID: \"98f478cc-52d9-4a63-9a98-c6a97592a958\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt"
Sep 18 20:03:24.319857 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:24.319769    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/98f478cc-52d9-4a63-9a98-c6a97592a958-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt\" (UID: \"98f478cc-52d9-4a63-9a98-c6a97592a958\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt"
Sep 18 20:03:24.319857 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:24.319792    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98f478cc-52d9-4a63-9a98-c6a97592a958-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt\" (UID: \"98f478cc-52d9-4a63-9a98-c6a97592a958\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt"
Sep 18 20:03:24.320184 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:24.320163    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/98f478cc-52d9-4a63-9a98-c6a97592a958-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt\" (UID: \"98f478cc-52d9-4a63-9a98-c6a97592a958\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt"
Sep 18 20:03:24.320268 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:24.320244    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/98f478cc-52d9-4a63-9a98-c6a97592a958-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt\" (UID: \"98f478cc-52d9-4a63-9a98-c6a97592a958\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt"
Sep 18 20:03:24.322401 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:24.322381    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98f478cc-52d9-4a63-9a98-c6a97592a958-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt\" (UID: \"98f478cc-52d9-4a63-9a98-c6a97592a958\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt"
Sep 18 20:03:24.334104 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:24.334085    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cq9l\" (UniqueName: \"kubernetes.io/projected/98f478cc-52d9-4a63-9a98-c6a97592a958-kube-api-access-2cq9l\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt\" (UID: \"98f478cc-52d9-4a63-9a98-c6a97592a958\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt"
Sep 18 20:03:24.451524 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:24.451443    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt"
Sep 18 20:03:24.737513 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:24.737484    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt"]
Sep 18 20:03:24.739305 ip-10-0-143-134 kubenswrapper[2581]: W0918 20:03:24.739278    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f478cc_52d9_4a63_9a98_c6a97592a958.slice/crio-c7f60e73109e9e5565e5974fe86bca767d02222dc272d151fd19ec49126da774 WatchSource:0}: Error finding container c7f60e73109e9e5565e5974fe86bca767d02222dc272d151fd19ec49126da774: Status 404 returned error can't find the container with id c7f60e73109e9e5565e5974fe86bca767d02222dc272d151fd19ec49126da774
Sep 18 20:03:25.472583 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:25.472544    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt" event={"ID":"98f478cc-52d9-4a63-9a98-c6a97592a958","Type":"ContainerStarted","Data":"26351b37d31132016b4cc4cabf52ea59697803cc2214fd7fb1d79997bed88154"}
Sep 18 20:03:25.473007 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:25.472608    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt" event={"ID":"98f478cc-52d9-4a63-9a98-c6a97592a958","Type":"ContainerStarted","Data":"c7f60e73109e9e5565e5974fe86bca767d02222dc272d151fd19ec49126da774"}
Sep 18 20:03:25.473007 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:25.472688    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt"
Sep 18 20:03:25.491864 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:25.491807    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt" podStartSLOduration=1.491791949 podStartE2EDuration="1.491791949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:03:25.490751378 +0000 UTC m=+1528.593039706" watchObservedRunningTime="2026-09-18 20:03:25.491791949 +0000 UTC m=+1528.594080276"
Sep 18 20:03:31.210437 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:31.210387    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9" podUID="f74a4198-a169-4e91-b049-e10b25dca82d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Sep 18 20:03:41.210788 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:41.210743    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9" podUID="f74a4198-a169-4e91-b049-e10b25dca82d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Sep 18 20:03:51.210687 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:51.210646    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9" podUID="f74a4198-a169-4e91-b049-e10b25dca82d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Sep 18 20:03:56.479980 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:56.479951    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt"
Sep 18 20:03:58.145500 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:03:58.145458    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt"]
Sep 18 20:04:01.210288 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:01.210244    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9" podUID="f74a4198-a169-4e91-b049-e10b25dca82d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Sep 18 20:04:06.480324 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:06.480286    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt" podUID="98f478cc-52d9-4a63-9a98-c6a97592a958" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:04:11.220245 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:11.220216    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"
Sep 18 20:04:11.227852 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:11.227820    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"
Sep 18 20:04:13.169532 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:13.169490    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt" podUID="98f478cc-52d9-4a63-9a98-c6a97592a958" containerName="main" containerID="cri-o://26351b37d31132016b4cc4cabf52ea59697803cc2214fd7fb1d79997bed88154" gracePeriod=45
Sep 18 20:04:16.130510 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:16.130462    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"]
Sep 18 20:04:16.480215 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:16.480135    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt" podUID="98f478cc-52d9-4a63-9a98-c6a97592a958" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:04:26.480358 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:26.480315    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt" podUID="98f478cc-52d9-4a63-9a98-c6a97592a958" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:04:26.480734 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:26.480433    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt"
Sep 18 20:04:31.154239 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:31.154200    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9" podUID="f74a4198-a169-4e91-b049-e10b25dca82d" containerName="main" containerID="cri-o://654b74434fc271f51a7f09b88d2d4014ae43337b30cb9b8b26fdbe9ea17f55bc" gracePeriod=45
Sep 18 20:04:31.805477 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:31.805451    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"
Sep 18 20:04:31.839786 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:31.839754    2581 generic.go:356] "Generic (PLEG): container finished" podID="f74a4198-a169-4e91-b049-e10b25dca82d" containerID="654b74434fc271f51a7f09b88d2d4014ae43337b30cb9b8b26fdbe9ea17f55bc" exitCode=0
Sep 18 20:04:31.839933 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:31.839801    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"
Sep 18 20:04:31.839933 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:31.839855    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9" event={"ID":"f74a4198-a169-4e91-b049-e10b25dca82d","Type":"ContainerDied","Data":"654b74434fc271f51a7f09b88d2d4014ae43337b30cb9b8b26fdbe9ea17f55bc"}
Sep 18 20:04:31.839933 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:31.839921    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9" event={"ID":"f74a4198-a169-4e91-b049-e10b25dca82d","Type":"ContainerDied","Data":"acedc3943b3e5a3b739bc0dd9951e3032eab4411541d268606cbc8f6d7698f69"}
Sep 18 20:04:31.840042 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:31.839955    2581 scope.go:117] "RemoveContainer" containerID="654b74434fc271f51a7f09b88d2d4014ae43337b30cb9b8b26fdbe9ea17f55bc"
Sep 18 20:04:31.851062 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:31.851043    2581 scope.go:117] "RemoveContainer" containerID="b33486665834c02837d88e21aa0a058e934884186d2e7254fce43c58fb8326d6"
Sep 18 20:04:31.859420 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:31.859404    2581 scope.go:117] "RemoveContainer" containerID="654b74434fc271f51a7f09b88d2d4014ae43337b30cb9b8b26fdbe9ea17f55bc"
Sep 18 20:04:31.859670 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:04:31.859651    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"654b74434fc271f51a7f09b88d2d4014ae43337b30cb9b8b26fdbe9ea17f55bc\": container with ID starting with 654b74434fc271f51a7f09b88d2d4014ae43337b30cb9b8b26fdbe9ea17f55bc not found: ID does not exist" containerID="654b74434fc271f51a7f09b88d2d4014ae43337b30cb9b8b26fdbe9ea17f55bc"
Sep 18 20:04:31.859728 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:31.859680    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"654b74434fc271f51a7f09b88d2d4014ae43337b30cb9b8b26fdbe9ea17f55bc"} err="failed to get container status \"654b74434fc271f51a7f09b88d2d4014ae43337b30cb9b8b26fdbe9ea17f55bc\": rpc error: code = NotFound desc = could not find container \"654b74434fc271f51a7f09b88d2d4014ae43337b30cb9b8b26fdbe9ea17f55bc\": container with ID starting with 654b74434fc271f51a7f09b88d2d4014ae43337b30cb9b8b26fdbe9ea17f55bc not found: ID does not exist"
Sep 18 20:04:31.859728 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:31.859699    2581 scope.go:117] "RemoveContainer" containerID="b33486665834c02837d88e21aa0a058e934884186d2e7254fce43c58fb8326d6"
Sep 18 20:04:31.859976 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:04:31.859959    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b33486665834c02837d88e21aa0a058e934884186d2e7254fce43c58fb8326d6\": container with ID starting with b33486665834c02837d88e21aa0a058e934884186d2e7254fce43c58fb8326d6 not found: ID does not exist" containerID="b33486665834c02837d88e21aa0a058e934884186d2e7254fce43c58fb8326d6"
Sep 18 20:04:31.860028 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:31.859981    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b33486665834c02837d88e21aa0a058e934884186d2e7254fce43c58fb8326d6"} err="failed to get container status \"b33486665834c02837d88e21aa0a058e934884186d2e7254fce43c58fb8326d6\": rpc error: code = NotFound desc = could not find container \"b33486665834c02837d88e21aa0a058e934884186d2e7254fce43c58fb8326d6\": container with ID starting with b33486665834c02837d88e21aa0a058e934884186d2e7254fce43c58fb8326d6 not found: ID does not exist"
Sep 18 20:04:31.914124 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:31.914061    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f74a4198-a169-4e91-b049-e10b25dca82d-tls-certs\") pod \"f74a4198-a169-4e91-b049-e10b25dca82d\" (UID: \"f74a4198-a169-4e91-b049-e10b25dca82d\") "
Sep 18 20:04:31.914124 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:31.914108    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f74a4198-a169-4e91-b049-e10b25dca82d-model-cache\") pod \"f74a4198-a169-4e91-b049-e10b25dca82d\" (UID: \"f74a4198-a169-4e91-b049-e10b25dca82d\") "
Sep 18 20:04:31.914285 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:31.914139    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f74a4198-a169-4e91-b049-e10b25dca82d-kserve-provision-location\") pod \"f74a4198-a169-4e91-b049-e10b25dca82d\" (UID: \"f74a4198-a169-4e91-b049-e10b25dca82d\") "
Sep 18 20:04:31.914285 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:31.914187    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f74a4198-a169-4e91-b049-e10b25dca82d-tmp-dir\") pod \"f74a4198-a169-4e91-b049-e10b25dca82d\" (UID: \"f74a4198-a169-4e91-b049-e10b25dca82d\") "
Sep 18 20:04:31.914285 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:31.914251    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f74a4198-a169-4e91-b049-e10b25dca82d-home\") pod \"f74a4198-a169-4e91-b049-e10b25dca82d\" (UID: \"f74a4198-a169-4e91-b049-e10b25dca82d\") "
Sep 18 20:04:31.914427 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:31.914289    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqpz6\" (UniqueName: \"kubernetes.io/projected/f74a4198-a169-4e91-b049-e10b25dca82d-kube-api-access-zqpz6\") pod \"f74a4198-a169-4e91-b049-e10b25dca82d\" (UID: \"f74a4198-a169-4e91-b049-e10b25dca82d\") "
Sep 18 20:04:31.914427 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:31.914301    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f74a4198-a169-4e91-b049-e10b25dca82d-model-cache" (OuterVolumeSpecName: "model-cache") pod "f74a4198-a169-4e91-b049-e10b25dca82d" (UID: "f74a4198-a169-4e91-b049-e10b25dca82d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:04:31.914427 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:31.914325    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f74a4198-a169-4e91-b049-e10b25dca82d-cabundle-cert\") pod \"f74a4198-a169-4e91-b049-e10b25dca82d\" (UID: \"f74a4198-a169-4e91-b049-e10b25dca82d\") "
Sep 18 20:04:31.914579 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:31.914431    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f74a4198-a169-4e91-b049-e10b25dca82d-dshm\") pod \"f74a4198-a169-4e91-b049-e10b25dca82d\" (UID: \"f74a4198-a169-4e91-b049-e10b25dca82d\") "
Sep 18 20:04:31.915293 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:31.915271    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f74a4198-a169-4e91-b049-e10b25dca82d-model-cache\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:04:31.915431 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:31.915303    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f74a4198-a169-4e91-b049-e10b25dca82d-home" (OuterVolumeSpecName: "home") pod "f74a4198-a169-4e91-b049-e10b25dca82d" (UID: "f74a4198-a169-4e91-b049-e10b25dca82d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:04:31.915630 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:31.915606    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f74a4198-a169-4e91-b049-e10b25dca82d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f74a4198-a169-4e91-b049-e10b25dca82d" (UID: "f74a4198-a169-4e91-b049-e10b25dca82d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:04:31.923447 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:31.923419    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f74a4198-a169-4e91-b049-e10b25dca82d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f74a4198-a169-4e91-b049-e10b25dca82d" (UID: "f74a4198-a169-4e91-b049-e10b25dca82d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:04:31.924069 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:31.923479    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f74a4198-a169-4e91-b049-e10b25dca82d-kube-api-access-zqpz6" (OuterVolumeSpecName: "kube-api-access-zqpz6") pod "f74a4198-a169-4e91-b049-e10b25dca82d" (UID: "f74a4198-a169-4e91-b049-e10b25dca82d"). InnerVolumeSpecName "kube-api-access-zqpz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:04:31.925027 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:31.924997    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f74a4198-a169-4e91-b049-e10b25dca82d-dshm" (OuterVolumeSpecName: "dshm") pod "f74a4198-a169-4e91-b049-e10b25dca82d" (UID: "f74a4198-a169-4e91-b049-e10b25dca82d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:04:31.931213 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:31.931177    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f74a4198-a169-4e91-b049-e10b25dca82d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f74a4198-a169-4e91-b049-e10b25dca82d" (UID: "f74a4198-a169-4e91-b049-e10b25dca82d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:04:31.944634 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:31.944607    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f74a4198-a169-4e91-b049-e10b25dca82d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f74a4198-a169-4e91-b049-e10b25dca82d" (UID: "f74a4198-a169-4e91-b049-e10b25dca82d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:04:32.016637 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:32.016609    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f74a4198-a169-4e91-b049-e10b25dca82d-dshm\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:04:32.016637 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:32.016633    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f74a4198-a169-4e91-b049-e10b25dca82d-tls-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:04:32.016791 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:32.016675    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f74a4198-a169-4e91-b049-e10b25dca82d-kserve-provision-location\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:04:32.016791 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:32.016688    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f74a4198-a169-4e91-b049-e10b25dca82d-tmp-dir\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:04:32.016791 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:32.016698    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f74a4198-a169-4e91-b049-e10b25dca82d-home\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:04:32.016791 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:32.016705    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqpz6\" (UniqueName: \"kubernetes.io/projected/f74a4198-a169-4e91-b049-e10b25dca82d-kube-api-access-zqpz6\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:04:32.016791 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:32.016713    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f74a4198-a169-4e91-b049-e10b25dca82d-cabundle-cert\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:04:32.161616 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:32.161587    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"]
Sep 18 20:04:32.165054 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:32.165000    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5fddfc79bc-lqhw9"]
Sep 18 20:04:33.436379 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:33.436345    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f74a4198-a169-4e91-b049-e10b25dca82d" path="/var/lib/kubelet/pods/f74a4198-a169-4e91-b049-e10b25dca82d/volumes"
Sep 18 20:04:36.480503 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:36.480463    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt" podUID="98f478cc-52d9-4a63-9a98-c6a97592a958" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 20:04:43.420191 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:43.420167    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt"
Sep 18 20:04:43.516181 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:43.516087    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cq9l\" (UniqueName: \"kubernetes.io/projected/98f478cc-52d9-4a63-9a98-c6a97592a958-kube-api-access-2cq9l\") pod \"98f478cc-52d9-4a63-9a98-c6a97592a958\" (UID: \"98f478cc-52d9-4a63-9a98-c6a97592a958\") "
Sep 18 20:04:43.516181 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:43.516142    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/98f478cc-52d9-4a63-9a98-c6a97592a958-cachi2-cache\") pod \"98f478cc-52d9-4a63-9a98-c6a97592a958\" (UID: \"98f478cc-52d9-4a63-9a98-c6a97592a958\") "
Sep 18 20:04:43.516181 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:43.516176    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/98f478cc-52d9-4a63-9a98-c6a97592a958-tokenizer-cache\") pod \"98f478cc-52d9-4a63-9a98-c6a97592a958\" (UID: \"98f478cc-52d9-4a63-9a98-c6a97592a958\") "
Sep 18 20:04:43.516465 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:43.516243    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98f478cc-52d9-4a63-9a98-c6a97592a958-tls-certs\") pod \"98f478cc-52d9-4a63-9a98-c6a97592a958\" (UID: \"98f478cc-52d9-4a63-9a98-c6a97592a958\") "
Sep 18 20:04:43.516465 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:43.516444    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f478cc-52d9-4a63-9a98-c6a97592a958-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "98f478cc-52d9-4a63-9a98-c6a97592a958" (UID: "98f478cc-52d9-4a63-9a98-c6a97592a958"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:04:43.516557 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:43.516459    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f478cc-52d9-4a63-9a98-c6a97592a958-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "98f478cc-52d9-4a63-9a98-c6a97592a958" (UID: "98f478cc-52d9-4a63-9a98-c6a97592a958"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:04:43.518544 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:43.518517    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98f478cc-52d9-4a63-9a98-c6a97592a958-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "98f478cc-52d9-4a63-9a98-c6a97592a958" (UID: "98f478cc-52d9-4a63-9a98-c6a97592a958"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:04:43.518544 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:43.518538    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98f478cc-52d9-4a63-9a98-c6a97592a958-kube-api-access-2cq9l" (OuterVolumeSpecName: "kube-api-access-2cq9l") pod "98f478cc-52d9-4a63-9a98-c6a97592a958" (UID: "98f478cc-52d9-4a63-9a98-c6a97592a958"). InnerVolumeSpecName "kube-api-access-2cq9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:04:43.617444 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:43.617412    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98f478cc-52d9-4a63-9a98-c6a97592a958-tls-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:04:43.617444 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:43.617439    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cq9l\" (UniqueName: \"kubernetes.io/projected/98f478cc-52d9-4a63-9a98-c6a97592a958-kube-api-access-2cq9l\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:04:43.617444 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:43.617450    2581 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/98f478cc-52d9-4a63-9a98-c6a97592a958-cachi2-cache\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:04:43.617671 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:43.617458    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/98f478cc-52d9-4a63-9a98-c6a97592a958-tokenizer-cache\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:04:43.904526 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:43.904491    2581 generic.go:356] "Generic (PLEG): container finished" podID="98f478cc-52d9-4a63-9a98-c6a97592a958" containerID="26351b37d31132016b4cc4cabf52ea59697803cc2214fd7fb1d79997bed88154" exitCode=0
Sep 18 20:04:43.904700 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:43.904536    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt"
Sep 18 20:04:43.904700 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:43.904587    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt" event={"ID":"98f478cc-52d9-4a63-9a98-c6a97592a958","Type":"ContainerDied","Data":"26351b37d31132016b4cc4cabf52ea59697803cc2214fd7fb1d79997bed88154"}
Sep 18 20:04:43.904700 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:43.904636    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt" event={"ID":"98f478cc-52d9-4a63-9a98-c6a97592a958","Type":"ContainerDied","Data":"c7f60e73109e9e5565e5974fe86bca767d02222dc272d151fd19ec49126da774"}
Sep 18 20:04:43.904700 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:43.904657    2581 scope.go:117] "RemoveContainer" containerID="26351b37d31132016b4cc4cabf52ea59697803cc2214fd7fb1d79997bed88154"
Sep 18 20:04:43.914063 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:43.914045    2581 scope.go:117] "RemoveContainer" containerID="26351b37d31132016b4cc4cabf52ea59697803cc2214fd7fb1d79997bed88154"
Sep 18 20:04:43.914331 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:04:43.914310    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26351b37d31132016b4cc4cabf52ea59697803cc2214fd7fb1d79997bed88154\": container with ID starting with 26351b37d31132016b4cc4cabf52ea59697803cc2214fd7fb1d79997bed88154 not found: ID does not exist" containerID="26351b37d31132016b4cc4cabf52ea59697803cc2214fd7fb1d79997bed88154"
Sep 18 20:04:43.914393 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:43.914343    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26351b37d31132016b4cc4cabf52ea59697803cc2214fd7fb1d79997bed88154"} err="failed to get container status \"26351b37d31132016b4cc4cabf52ea59697803cc2214fd7fb1d79997bed88154\": rpc error: code = NotFound desc = could not find container \"26351b37d31132016b4cc4cabf52ea59697803cc2214fd7fb1d79997bed88154\": container with ID starting with 26351b37d31132016b4cc4cabf52ea59697803cc2214fd7fb1d79997bed88154 not found: ID does not exist"
Sep 18 20:04:43.928319 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:43.928296    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt"]
Sep 18 20:04:43.931736 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:43.931701    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcwgxtt"]
Sep 18 20:04:45.432590 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:45.432554    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98f478cc-52d9-4a63-9a98-c6a97592a958" path="/var/lib/kubelet/pods/98f478cc-52d9-4a63-9a98-c6a97592a958/volumes"
Sep 18 20:04:51.322041 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.321964    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"]
Sep 18 20:04:51.322395 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.322303    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98f478cc-52d9-4a63-9a98-c6a97592a958" containerName="main"
Sep 18 20:04:51.322395 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.322313    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="98f478cc-52d9-4a63-9a98-c6a97592a958" containerName="main"
Sep 18 20:04:51.322395 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.322324    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f74a4198-a169-4e91-b049-e10b25dca82d" containerName="main"
Sep 18 20:04:51.322395 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.322329    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f74a4198-a169-4e91-b049-e10b25dca82d" containerName="main"
Sep 18 20:04:51.322395 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.322342    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f74a4198-a169-4e91-b049-e10b25dca82d" containerName="storage-initializer"
Sep 18 20:04:51.322395 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.322348    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f74a4198-a169-4e91-b049-e10b25dca82d" containerName="storage-initializer"
Sep 18 20:04:51.322587 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.322410    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="98f478cc-52d9-4a63-9a98-c6a97592a958" containerName="main"
Sep 18 20:04:51.322587 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.322421    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f74a4198-a169-4e91-b049-e10b25dca82d" containerName="main"
Sep 18 20:04:51.326857 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.326814    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"
Sep 18 20:04:51.331051 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.330880    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-ttxps\""
Sep 18 20:04:51.331051 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.330931    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-zh2cf\""
Sep 18 20:04:51.331051 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.330934    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 18 20:04:51.331051 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.330954    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 18 20:04:51.331051 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.330962    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 18 20:04:51.339149 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.339111    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"]
Sep 18 20:04:51.385328 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.385298    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b6b2405-27d7-4e74-84f1-3537c63d1bae-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2\" (UID: \"5b6b2405-27d7-4e74-84f1-3537c63d1bae\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"
Sep 18 20:04:51.385494 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.385336    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b6b2405-27d7-4e74-84f1-3537c63d1bae-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2\" (UID: \"5b6b2405-27d7-4e74-84f1-3537c63d1bae\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"
Sep 18 20:04:51.385494 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.385375    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b6b2405-27d7-4e74-84f1-3537c63d1bae-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2\" (UID: \"5b6b2405-27d7-4e74-84f1-3537c63d1bae\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"
Sep 18 20:04:51.385494 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.385439    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nww9z\" (UniqueName: \"kubernetes.io/projected/5b6b2405-27d7-4e74-84f1-3537c63d1bae-kube-api-access-nww9z\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2\" (UID: \"5b6b2405-27d7-4e74-84f1-3537c63d1bae\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"
Sep 18 20:04:51.385494 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.385485    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b6b2405-27d7-4e74-84f1-3537c63d1bae-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2\" (UID: \"5b6b2405-27d7-4e74-84f1-3537c63d1bae\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"
Sep 18 20:04:51.385631 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.385516    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b6b2405-27d7-4e74-84f1-3537c63d1bae-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2\" (UID: \"5b6b2405-27d7-4e74-84f1-3537c63d1bae\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"
Sep 18 20:04:51.486240 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.486203    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b6b2405-27d7-4e74-84f1-3537c63d1bae-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2\" (UID: \"5b6b2405-27d7-4e74-84f1-3537c63d1bae\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"
Sep 18 20:04:51.486240 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.486242    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b6b2405-27d7-4e74-84f1-3537c63d1bae-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2\" (UID: \"5b6b2405-27d7-4e74-84f1-3537c63d1bae\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"
Sep 18 20:04:51.486476 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.486265    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b6b2405-27d7-4e74-84f1-3537c63d1bae-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2\" (UID: \"5b6b2405-27d7-4e74-84f1-3537c63d1bae\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"
Sep 18 20:04:51.486476 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.486284    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nww9z\" (UniqueName: \"kubernetes.io/projected/5b6b2405-27d7-4e74-84f1-3537c63d1bae-kube-api-access-nww9z\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2\" (UID: \"5b6b2405-27d7-4e74-84f1-3537c63d1bae\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"
Sep 18 20:04:51.486476 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.486302    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b6b2405-27d7-4e74-84f1-3537c63d1bae-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2\" (UID: \"5b6b2405-27d7-4e74-84f1-3537c63d1bae\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"
Sep 18 20:04:51.486614 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.486474    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b6b2405-27d7-4e74-84f1-3537c63d1bae-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2\" (UID: \"5b6b2405-27d7-4e74-84f1-3537c63d1bae\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"
Sep 18 20:04:51.486670 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.486651    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b6b2405-27d7-4e74-84f1-3537c63d1bae-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2\" (UID: \"5b6b2405-27d7-4e74-84f1-3537c63d1bae\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"
Sep 18 20:04:51.486728 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.486676    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b6b2405-27d7-4e74-84f1-3537c63d1bae-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2\" (UID: \"5b6b2405-27d7-4e74-84f1-3537c63d1bae\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"
Sep 18 20:04:51.486787 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.486730    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b6b2405-27d7-4e74-84f1-3537c63d1bae-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2\" (UID: \"5b6b2405-27d7-4e74-84f1-3537c63d1bae\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"
Sep 18 20:04:51.488933 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.488903    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b6b2405-27d7-4e74-84f1-3537c63d1bae-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2\" (UID: \"5b6b2405-27d7-4e74-84f1-3537c63d1bae\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"
Sep 18 20:04:51.489104 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.489088    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b6b2405-27d7-4e74-84f1-3537c63d1bae-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2\" (UID: \"5b6b2405-27d7-4e74-84f1-3537c63d1bae\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"
Sep 18 20:04:51.494203 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.494180    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nww9z\" (UniqueName: \"kubernetes.io/projected/5b6b2405-27d7-4e74-84f1-3537c63d1bae-kube-api-access-nww9z\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2\" (UID: \"5b6b2405-27d7-4e74-84f1-3537c63d1bae\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"
Sep 18 20:04:51.637290 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.637202    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"
Sep 18 20:04:51.910946 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.910921    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"]
Sep 18 20:04:51.912433 ip-10-0-143-134 kubenswrapper[2581]: W0918 20:04:51.912413    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b6b2405_27d7_4e74_84f1_3537c63d1bae.slice/crio-dd69db7410532fe35095c90a0901706bd5fd3b6f3553789dd392c0ae987d4614 WatchSource:0}: Error finding container dd69db7410532fe35095c90a0901706bd5fd3b6f3553789dd392c0ae987d4614: Status 404 returned error can't find the container with id dd69db7410532fe35095c90a0901706bd5fd3b6f3553789dd392c0ae987d4614
Sep 18 20:04:51.956216 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:51.956182    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2" event={"ID":"5b6b2405-27d7-4e74-84f1-3537c63d1bae","Type":"ContainerStarted","Data":"dd69db7410532fe35095c90a0901706bd5fd3b6f3553789dd392c0ae987d4614"}
Sep 18 20:04:53.968887 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:53.968819    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2" event={"ID":"5b6b2405-27d7-4e74-84f1-3537c63d1bae","Type":"ContainerStarted","Data":"d1dacb4ccb40d73b4d796693a1fd456ac87e5e8dff81dd2a5a2ff4a9e0d1b2c4"}
Sep 18 20:04:53.969283 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:53.969012    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"
Sep 18 20:04:54.976414 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:54.976382    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2" event={"ID":"5b6b2405-27d7-4e74-84f1-3537c63d1bae","Type":"ContainerStarted","Data":"bd37ab48a20db9341ec125a7909793e7e1fc432dd7cf717f6fa22e376cef7378"}
Sep 18 20:04:54.994024 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:54.993967    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2" podStartSLOduration=2.458681999 podStartE2EDuration="3.993947675s" totalImagesPullingTime="1.535265676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:04:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:04:54.993143597 +0000 UTC m=+1618.095431925" watchObservedRunningTime="2026-09-18 20:04:54.993947675 +0000 UTC m=+1618.096236003"
Sep 18 20:04:55.638178 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:04:55.638147    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"
Sep 18 20:05:01.637907 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:05:01.637872    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"
Sep 18 20:05:01.650311 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:05:01.650285    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"
Sep 18 20:05:02.025225 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:05:02.025204    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"
Sep 18 20:05:03.030953 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:05:03.030930    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"
Sep 18 20:06:04.021213 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:04.021178    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"]
Sep 18 20:06:04.030888 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:06:04.030825    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 20:06:04.030888 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:06:04.030888 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 20:06:04.030888 ip-10-0-143-134 kubenswrapper[2581]:  > containerID="bd37ab48a20db9341ec125a7909793e7e1fc432dd7cf717f6fa22e376cef7378" cmd=["/bin/sleep","15"]
Sep 18 20:06:04.031129 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:04.030898    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2" message="" err=<
Sep 18 20:06:04.031129 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:06:04.031129 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 20:06:04.031129 ip-10-0-143-134 kubenswrapper[2581]:  >
Sep 18 20:06:04.031129 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:06:04.030911    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 20:06:04.031129 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:06:04.031129 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 20:06:04.031129 ip-10-0-143-134 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2" podUID="5b6b2405-27d7-4e74-84f1-3537c63d1bae" containerName="main" containerID="cri-o://bd37ab48a20db9341ec125a7909793e7e1fc432dd7cf717f6fa22e376cef7378"
Sep 18 20:06:04.031129 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:04.030954    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2" podUID="5b6b2405-27d7-4e74-84f1-3537c63d1bae" containerName="main" containerID="cri-o://bd37ab48a20db9341ec125a7909793e7e1fc432dd7cf717f6fa22e376cef7378" gracePeriod=60
Sep 18 20:06:04.246579 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:04.246544    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2" podUID="5b6b2405-27d7-4e74-84f1-3537c63d1bae" containerName="llm-d-routing-sidecar" containerID="cri-o://d1dacb4ccb40d73b4d796693a1fd456ac87e5e8dff81dd2a5a2ff4a9e0d1b2c4" gracePeriod=60
Sep 18 20:06:04.351566 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:04.351482    2581 generic.go:356] "Generic (PLEG): container finished" podID="5b6b2405-27d7-4e74-84f1-3537c63d1bae" containerID="bd37ab48a20db9341ec125a7909793e7e1fc432dd7cf717f6fa22e376cef7378" exitCode=0
Sep 18 20:06:04.351566 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:04.351546    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2" event={"ID":"5b6b2405-27d7-4e74-84f1-3537c63d1bae","Type":"ContainerDied","Data":"bd37ab48a20db9341ec125a7909793e7e1fc432dd7cf717f6fa22e376cef7378"}
Sep 18 20:06:04.491547 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:04.491523    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"
Sep 18 20:06:04.635279 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:04.635194    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b6b2405-27d7-4e74-84f1-3537c63d1bae-model-cache\") pod \"5b6b2405-27d7-4e74-84f1-3537c63d1bae\" (UID: \"5b6b2405-27d7-4e74-84f1-3537c63d1bae\") "
Sep 18 20:06:04.635279 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:04.635237    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b6b2405-27d7-4e74-84f1-3537c63d1bae-home\") pod \"5b6b2405-27d7-4e74-84f1-3537c63d1bae\" (UID: \"5b6b2405-27d7-4e74-84f1-3537c63d1bae\") "
Sep 18 20:06:04.635509 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:04.635301    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b6b2405-27d7-4e74-84f1-3537c63d1bae-tls-certs\") pod \"5b6b2405-27d7-4e74-84f1-3537c63d1bae\" (UID: \"5b6b2405-27d7-4e74-84f1-3537c63d1bae\") "
Sep 18 20:06:04.635509 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:04.635331    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b6b2405-27d7-4e74-84f1-3537c63d1bae-tmp-dir\") pod \"5b6b2405-27d7-4e74-84f1-3537c63d1bae\" (UID: \"5b6b2405-27d7-4e74-84f1-3537c63d1bae\") "
Sep 18 20:06:04.635509 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:04.635380    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b6b2405-27d7-4e74-84f1-3537c63d1bae-dshm\") pod \"5b6b2405-27d7-4e74-84f1-3537c63d1bae\" (UID: \"5b6b2405-27d7-4e74-84f1-3537c63d1bae\") "
Sep 18 20:06:04.635509 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:04.635435    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nww9z\" (UniqueName: \"kubernetes.io/projected/5b6b2405-27d7-4e74-84f1-3537c63d1bae-kube-api-access-nww9z\") pod \"5b6b2405-27d7-4e74-84f1-3537c63d1bae\" (UID: \"5b6b2405-27d7-4e74-84f1-3537c63d1bae\") "
Sep 18 20:06:04.635509 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:04.635487    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6b2405-27d7-4e74-84f1-3537c63d1bae-model-cache" (OuterVolumeSpecName: "model-cache") pod "5b6b2405-27d7-4e74-84f1-3537c63d1bae" (UID: "5b6b2405-27d7-4e74-84f1-3537c63d1bae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:04.635778 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:04.635509    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6b2405-27d7-4e74-84f1-3537c63d1bae-home" (OuterVolumeSpecName: "home") pod "5b6b2405-27d7-4e74-84f1-3537c63d1bae" (UID: "5b6b2405-27d7-4e74-84f1-3537c63d1bae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:04.635778 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:04.635604    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6b2405-27d7-4e74-84f1-3537c63d1bae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5b6b2405-27d7-4e74-84f1-3537c63d1bae" (UID: "5b6b2405-27d7-4e74-84f1-3537c63d1bae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:04.635778 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:04.635773    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b6b2405-27d7-4e74-84f1-3537c63d1bae-model-cache\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:06:04.635927 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:04.635787    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b6b2405-27d7-4e74-84f1-3537c63d1bae-home\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:06:04.635927 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:04.635795    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b6b2405-27d7-4e74-84f1-3537c63d1bae-tmp-dir\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:06:04.637806 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:04.637783    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6b2405-27d7-4e74-84f1-3537c63d1bae-dshm" (OuterVolumeSpecName: "dshm") pod "5b6b2405-27d7-4e74-84f1-3537c63d1bae" (UID: "5b6b2405-27d7-4e74-84f1-3537c63d1bae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:06:04.637955 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:04.637891    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b6b2405-27d7-4e74-84f1-3537c63d1bae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b6b2405-27d7-4e74-84f1-3537c63d1bae" (UID: "5b6b2405-27d7-4e74-84f1-3537c63d1bae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:06:04.637955 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:04.637892    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b6b2405-27d7-4e74-84f1-3537c63d1bae-kube-api-access-nww9z" (OuterVolumeSpecName: "kube-api-access-nww9z") pod "5b6b2405-27d7-4e74-84f1-3537c63d1bae" (UID: "5b6b2405-27d7-4e74-84f1-3537c63d1bae"). InnerVolumeSpecName "kube-api-access-nww9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:06:04.736913 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:04.736886    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b6b2405-27d7-4e74-84f1-3537c63d1bae-tls-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:06:04.736913 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:04.736908    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b6b2405-27d7-4e74-84f1-3537c63d1bae-dshm\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:06:04.736913 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:04.736918    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nww9z\" (UniqueName: \"kubernetes.io/projected/5b6b2405-27d7-4e74-84f1-3537c63d1bae-kube-api-access-nww9z\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:06:05.358366 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:05.358334    2581 generic.go:356] "Generic (PLEG): container finished" podID="5b6b2405-27d7-4e74-84f1-3537c63d1bae" containerID="d1dacb4ccb40d73b4d796693a1fd456ac87e5e8dff81dd2a5a2ff4a9e0d1b2c4" exitCode=0
Sep 18 20:06:05.358778 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:05.358374    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2" event={"ID":"5b6b2405-27d7-4e74-84f1-3537c63d1bae","Type":"ContainerDied","Data":"d1dacb4ccb40d73b4d796693a1fd456ac87e5e8dff81dd2a5a2ff4a9e0d1b2c4"}
Sep 18 20:06:05.358778 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:05.358400    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"
Sep 18 20:06:05.358778 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:05.358446    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2" event={"ID":"5b6b2405-27d7-4e74-84f1-3537c63d1bae","Type":"ContainerDied","Data":"dd69db7410532fe35095c90a0901706bd5fd3b6f3553789dd392c0ae987d4614"}
Sep 18 20:06:05.358778 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:05.358469    2581 scope.go:117] "RemoveContainer" containerID="bd37ab48a20db9341ec125a7909793e7e1fc432dd7cf717f6fa22e376cef7378"
Sep 18 20:06:05.368208 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:05.368190    2581 scope.go:117] "RemoveContainer" containerID="d1dacb4ccb40d73b4d796693a1fd456ac87e5e8dff81dd2a5a2ff4a9e0d1b2c4"
Sep 18 20:06:05.375727 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:05.375697    2581 scope.go:117] "RemoveContainer" containerID="bd37ab48a20db9341ec125a7909793e7e1fc432dd7cf717f6fa22e376cef7378"
Sep 18 20:06:05.376004 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:06:05.375986    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd37ab48a20db9341ec125a7909793e7e1fc432dd7cf717f6fa22e376cef7378\": container with ID starting with bd37ab48a20db9341ec125a7909793e7e1fc432dd7cf717f6fa22e376cef7378 not found: ID does not exist" containerID="bd37ab48a20db9341ec125a7909793e7e1fc432dd7cf717f6fa22e376cef7378"
Sep 18 20:06:05.376064 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:05.376013    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd37ab48a20db9341ec125a7909793e7e1fc432dd7cf717f6fa22e376cef7378"} err="failed to get container status \"bd37ab48a20db9341ec125a7909793e7e1fc432dd7cf717f6fa22e376cef7378\": rpc error: code = NotFound desc = could not find container \"bd37ab48a20db9341ec125a7909793e7e1fc432dd7cf717f6fa22e376cef7378\": container with ID starting with bd37ab48a20db9341ec125a7909793e7e1fc432dd7cf717f6fa22e376cef7378 not found: ID does not exist"
Sep 18 20:06:05.376064 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:05.376032    2581 scope.go:117] "RemoveContainer" containerID="d1dacb4ccb40d73b4d796693a1fd456ac87e5e8dff81dd2a5a2ff4a9e0d1b2c4"
Sep 18 20:06:05.376284 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:06:05.376268    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1dacb4ccb40d73b4d796693a1fd456ac87e5e8dff81dd2a5a2ff4a9e0d1b2c4\": container with ID starting with d1dacb4ccb40d73b4d796693a1fd456ac87e5e8dff81dd2a5a2ff4a9e0d1b2c4 not found: ID does not exist" containerID="d1dacb4ccb40d73b4d796693a1fd456ac87e5e8dff81dd2a5a2ff4a9e0d1b2c4"
Sep 18 20:06:05.376328 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:05.376290    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1dacb4ccb40d73b4d796693a1fd456ac87e5e8dff81dd2a5a2ff4a9e0d1b2c4"} err="failed to get container status \"d1dacb4ccb40d73b4d796693a1fd456ac87e5e8dff81dd2a5a2ff4a9e0d1b2c4\": rpc error: code = NotFound desc = could not find container \"d1dacb4ccb40d73b4d796693a1fd456ac87e5e8dff81dd2a5a2ff4a9e0d1b2c4\": container with ID starting with d1dacb4ccb40d73b4d796693a1fd456ac87e5e8dff81dd2a5a2ff4a9e0d1b2c4 not found: ID does not exist"
Sep 18 20:06:05.379937 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:05.379916    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"]
Sep 18 20:06:05.383010 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:05.382989    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-2vdt2"]
Sep 18 20:06:05.432725 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:06:05.432699    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b6b2405-27d7-4e74-84f1-3537c63d1bae" path="/var/lib/kubelet/pods/5b6b2405-27d7-4e74-84f1-3537c63d1bae/volumes"
Sep 18 20:07:05.438798 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.438765    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"]
Sep 18 20:07:05.439431 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.439185    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b6b2405-27d7-4e74-84f1-3537c63d1bae" containerName="llm-d-routing-sidecar"
Sep 18 20:07:05.439431 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.439201    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b6b2405-27d7-4e74-84f1-3537c63d1bae" containerName="llm-d-routing-sidecar"
Sep 18 20:07:05.439431 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.439223    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b6b2405-27d7-4e74-84f1-3537c63d1bae" containerName="main"
Sep 18 20:07:05.439431 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.439232    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b6b2405-27d7-4e74-84f1-3537c63d1bae" containerName="main"
Sep 18 20:07:05.439431 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.439322    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b6b2405-27d7-4e74-84f1-3537c63d1bae" containerName="llm-d-routing-sidecar"
Sep 18 20:07:05.439431 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.439333    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b6b2405-27d7-4e74-84f1-3537c63d1bae" containerName="main"
Sep 18 20:07:05.442094 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.442067    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"
Sep 18 20:07:05.445292 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.445271    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 18 20:07:05.445426 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.445286    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-brqps\""
Sep 18 20:07:05.445426 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.445332    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 18 20:07:05.445622 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.445599    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 18 20:07:05.445731 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.445655    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-cqs5k\""
Sep 18 20:07:05.453486 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.453464    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"]
Sep 18 20:07:05.540409 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.540376    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl6kf\" (UniqueName: \"kubernetes.io/projected/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-kube-api-access-sl6kf\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7\" (UID: \"92e0e41a-68f1-4ea2-8433-e1105e87b1b0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"
Sep 18 20:07:05.540609 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.540561    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7\" (UID: \"92e0e41a-68f1-4ea2-8433-e1105e87b1b0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"
Sep 18 20:07:05.540724 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.540704    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7\" (UID: \"92e0e41a-68f1-4ea2-8433-e1105e87b1b0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"
Sep 18 20:07:05.540772 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.540743    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7\" (UID: \"92e0e41a-68f1-4ea2-8433-e1105e87b1b0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"
Sep 18 20:07:05.540818 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.540774    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7\" (UID: \"92e0e41a-68f1-4ea2-8433-e1105e87b1b0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"
Sep 18 20:07:05.540890 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.540829    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7\" (UID: \"92e0e41a-68f1-4ea2-8433-e1105e87b1b0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"
Sep 18 20:07:05.642267 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.642225    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl6kf\" (UniqueName: \"kubernetes.io/projected/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-kube-api-access-sl6kf\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7\" (UID: \"92e0e41a-68f1-4ea2-8433-e1105e87b1b0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"
Sep 18 20:07:05.642267 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.642279    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7\" (UID: \"92e0e41a-68f1-4ea2-8433-e1105e87b1b0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"
Sep 18 20:07:05.642552 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.642308    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7\" (UID: \"92e0e41a-68f1-4ea2-8433-e1105e87b1b0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"
Sep 18 20:07:05.642552 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.642399    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7\" (UID: \"92e0e41a-68f1-4ea2-8433-e1105e87b1b0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"
Sep 18 20:07:05.642552 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.642444    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7\" (UID: \"92e0e41a-68f1-4ea2-8433-e1105e87b1b0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"
Sep 18 20:07:05.642552 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.642472    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7\" (UID: \"92e0e41a-68f1-4ea2-8433-e1105e87b1b0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"
Sep 18 20:07:05.642768 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.642661    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7\" (UID: \"92e0e41a-68f1-4ea2-8433-e1105e87b1b0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"
Sep 18 20:07:05.642768 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.642700    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7\" (UID: \"92e0e41a-68f1-4ea2-8433-e1105e87b1b0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"
Sep 18 20:07:05.642768 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.642742    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7\" (UID: \"92e0e41a-68f1-4ea2-8433-e1105e87b1b0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"
Sep 18 20:07:05.644948 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.644931    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7\" (UID: \"92e0e41a-68f1-4ea2-8433-e1105e87b1b0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"
Sep 18 20:07:05.645137 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.645119    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7\" (UID: \"92e0e41a-68f1-4ea2-8433-e1105e87b1b0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"
Sep 18 20:07:05.652264 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.652235    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl6kf\" (UniqueName: \"kubernetes.io/projected/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-kube-api-access-sl6kf\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7\" (UID: \"92e0e41a-68f1-4ea2-8433-e1105e87b1b0\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"
Sep 18 20:07:05.753366 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:05.753264    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"
Sep 18 20:07:06.024531 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:06.024505    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"]
Sep 18 20:07:06.026486 ip-10-0-143-134 kubenswrapper[2581]: W0918 20:07:06.026461    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e0e41a_68f1_4ea2_8433_e1105e87b1b0.slice/crio-f31d46db00b7af3543009faf5951c263daa0e34058a5c8538b3e729211886a73 WatchSource:0}: Error finding container f31d46db00b7af3543009faf5951c263daa0e34058a5c8538b3e729211886a73: Status 404 returned error can't find the container with id f31d46db00b7af3543009faf5951c263daa0e34058a5c8538b3e729211886a73
Sep 18 20:07:06.687391 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:06.687359    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7" event={"ID":"92e0e41a-68f1-4ea2-8433-e1105e87b1b0","Type":"ContainerStarted","Data":"414b1495f975f577b6a9d49ff46328faed098938ce4b7c20fd6688eb48c6b696"}
Sep 18 20:07:06.687891 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:06.687411    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7" event={"ID":"92e0e41a-68f1-4ea2-8433-e1105e87b1b0","Type":"ContainerStarted","Data":"f31d46db00b7af3543009faf5951c263daa0e34058a5c8538b3e729211886a73"}
Sep 18 20:07:06.687891 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:06.687526    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"
Sep 18 20:07:07.694147 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:07.694112    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7" event={"ID":"92e0e41a-68f1-4ea2-8433-e1105e87b1b0","Type":"ContainerStarted","Data":"a753d9a66b5c25d55e44d873a3aa1ad77f5277ea4bbaeef1432eb9e513232ca1"}
Sep 18 20:07:07.714779 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:07.714729    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7" podStartSLOduration=2.714716198 podStartE2EDuration="2.714716198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:07:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:07:07.711527123 +0000 UTC m=+1750.813815450" watchObservedRunningTime="2026-09-18 20:07:07.714716198 +0000 UTC m=+1750.817004526"
Sep 18 20:07:07.753963 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:07.753942    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"
Sep 18 20:07:15.754093 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:15.754063    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"
Sep 18 20:07:15.766608 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:15.766580    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"
Sep 18 20:07:16.751671 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:16.751643    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"
Sep 18 20:07:16.752828 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:16.752807    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"
Sep 18 20:07:41.404700 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:41.404659    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf"]
Sep 18 20:07:41.407126 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:41.407105    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf"
Sep 18 20:07:41.410608 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:41.410584    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"default-dockercfg-4np2k\""
Sep 18 20:07:41.410742 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:41.410618    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"kube-root-ca.crt\""
Sep 18 20:07:41.410742 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:41.410632    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v1-kserve-self-signed-certs\""
Sep 18 20:07:41.410742 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:41.410715    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"openshift-service-ca.crt\""
Sep 18 20:07:41.417624 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:41.417603    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf"]
Sep 18 20:07:41.458974 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:41.458945    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/557d0e89-376a-4154-a271-727abdad13d1-dshm\") pod \"test-div-v1-kserve-fb566d667-sfqrf\" (UID: \"557d0e89-376a-4154-a271-727abdad13d1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf"
Sep 18 20:07:41.459143 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:41.458978    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/557d0e89-376a-4154-a271-727abdad13d1-model-cache\") pod \"test-div-v1-kserve-fb566d667-sfqrf\" (UID: \"557d0e89-376a-4154-a271-727abdad13d1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf"
Sep 18 20:07:41.459143 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:41.459002    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/557d0e89-376a-4154-a271-727abdad13d1-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-sfqrf\" (UID: \"557d0e89-376a-4154-a271-727abdad13d1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf"
Sep 18 20:07:41.459143 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:41.459094    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/557d0e89-376a-4154-a271-727abdad13d1-tls-certs\") pod \"test-div-v1-kserve-fb566d667-sfqrf\" (UID: \"557d0e89-376a-4154-a271-727abdad13d1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf"
Sep 18 20:07:41.459259 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:41.459181    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/557d0e89-376a-4154-a271-727abdad13d1-home\") pod \"test-div-v1-kserve-fb566d667-sfqrf\" (UID: \"557d0e89-376a-4154-a271-727abdad13d1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf"
Sep 18 20:07:41.459259 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:41.459199    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brb6s\" (UniqueName: \"kubernetes.io/projected/557d0e89-376a-4154-a271-727abdad13d1-kube-api-access-brb6s\") pod \"test-div-v1-kserve-fb566d667-sfqrf\" (UID: \"557d0e89-376a-4154-a271-727abdad13d1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf"
Sep 18 20:07:41.560225 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:41.560189    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/557d0e89-376a-4154-a271-727abdad13d1-home\") pod \"test-div-v1-kserve-fb566d667-sfqrf\" (UID: \"557d0e89-376a-4154-a271-727abdad13d1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf"
Sep 18 20:07:41.560409 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:41.560232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brb6s\" (UniqueName: \"kubernetes.io/projected/557d0e89-376a-4154-a271-727abdad13d1-kube-api-access-brb6s\") pod \"test-div-v1-kserve-fb566d667-sfqrf\" (UID: \"557d0e89-376a-4154-a271-727abdad13d1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf"
Sep 18 20:07:41.560409 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:41.560293    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/557d0e89-376a-4154-a271-727abdad13d1-dshm\") pod \"test-div-v1-kserve-fb566d667-sfqrf\" (UID: \"557d0e89-376a-4154-a271-727abdad13d1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf"
Sep 18 20:07:41.560409 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:41.560315    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/557d0e89-376a-4154-a271-727abdad13d1-model-cache\") pod \"test-div-v1-kserve-fb566d667-sfqrf\" (UID: \"557d0e89-376a-4154-a271-727abdad13d1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf"
Sep 18 20:07:41.560409 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:41.560338    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/557d0e89-376a-4154-a271-727abdad13d1-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-sfqrf\" (UID: \"557d0e89-376a-4154-a271-727abdad13d1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf"
Sep 18 20:07:41.560409 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:41.560380    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/557d0e89-376a-4154-a271-727abdad13d1-tls-certs\") pod \"test-div-v1-kserve-fb566d667-sfqrf\" (UID: \"557d0e89-376a-4154-a271-727abdad13d1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf"
Sep 18 20:07:41.560672 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:41.560648    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/557d0e89-376a-4154-a271-727abdad13d1-home\") pod \"test-div-v1-kserve-fb566d667-sfqrf\" (UID: \"557d0e89-376a-4154-a271-727abdad13d1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf"
Sep 18 20:07:41.560737 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:41.560679    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/557d0e89-376a-4154-a271-727abdad13d1-model-cache\") pod \"test-div-v1-kserve-fb566d667-sfqrf\" (UID: \"557d0e89-376a-4154-a271-727abdad13d1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf"
Sep 18 20:07:41.560737 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:41.560722    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/557d0e89-376a-4154-a271-727abdad13d1-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-sfqrf\" (UID: \"557d0e89-376a-4154-a271-727abdad13d1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf"
Sep 18 20:07:41.562730 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:41.562714    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/557d0e89-376a-4154-a271-727abdad13d1-dshm\") pod \"test-div-v1-kserve-fb566d667-sfqrf\" (UID: \"557d0e89-376a-4154-a271-727abdad13d1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf"
Sep 18 20:07:41.563029 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:41.563011    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/557d0e89-376a-4154-a271-727abdad13d1-tls-certs\") pod \"test-div-v1-kserve-fb566d667-sfqrf\" (UID: \"557d0e89-376a-4154-a271-727abdad13d1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf"
Sep 18 20:07:41.568573 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:41.568550    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brb6s\" (UniqueName: \"kubernetes.io/projected/557d0e89-376a-4154-a271-727abdad13d1-kube-api-access-brb6s\") pod \"test-div-v1-kserve-fb566d667-sfqrf\" (UID: \"557d0e89-376a-4154-a271-727abdad13d1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf"
Sep 18 20:07:41.718501 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:41.718418    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf"
Sep 18 20:07:42.014344 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:42.014294    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf"]
Sep 18 20:07:42.017165 ip-10-0-143-134 kubenswrapper[2581]: W0918 20:07:42.017130    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod557d0e89_376a_4154_a271_727abdad13d1.slice/crio-bd5206409445dd2d0d351433432a2d7e76e5d0ac47f81712d37e54d32e71371f WatchSource:0}: Error finding container bd5206409445dd2d0d351433432a2d7e76e5d0ac47f81712d37e54d32e71371f: Status 404 returned error can't find the container with id bd5206409445dd2d0d351433432a2d7e76e5d0ac47f81712d37e54d32e71371f
Sep 18 20:07:42.019331 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:42.019307    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:07:42.880551 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:42.880501    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf" event={"ID":"557d0e89-376a-4154-a271-727abdad13d1","Type":"ContainerStarted","Data":"bd5206409445dd2d0d351433432a2d7e76e5d0ac47f81712d37e54d32e71371f"}
Sep 18 20:07:43.887353 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:43.887317    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf" event={"ID":"557d0e89-376a-4154-a271-727abdad13d1","Type":"ContainerStarted","Data":"7175d2fde32398628e4a7e8ec04de9242ba6416577982e6a37d4eff6f4e9aa2c"}
Sep 18 20:07:43.906324 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:43.906276    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf" podStartSLOduration=1.913464112 podStartE2EDuration="2.906262235s" totalImagesPullingTime="992.798123ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:07:43.904089092 +0000 UTC m=+1787.006377424" watchObservedRunningTime="2026-09-18 20:07:43.906262235 +0000 UTC m=+1787.008550563"
Sep 18 20:07:44.719544 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:44.719504    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf"
Sep 18 20:07:51.719640 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:51.719556    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf"
Sep 18 20:07:51.732016 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:51.731995    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf"
Sep 18 20:07:51.930165 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:51.930134    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"]
Sep 18 20:07:51.942567 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:07:51.942524    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 20:07:51.942567 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:07:51.942567 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 20:07:51.942567 ip-10-0-143-134 kubenswrapper[2581]:  > containerID="a753d9a66b5c25d55e44d873a3aa1ad77f5277ea4bbaeef1432eb9e513232ca1" cmd=["/bin/sleep","15"]
Sep 18 20:07:51.942943 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:51.942574    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7" message="" err=<
Sep 18 20:07:51.942943 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:07:51.942943 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 20:07:51.942943 ip-10-0-143-134 kubenswrapper[2581]:  >
Sep 18 20:07:51.942943 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:07:51.942586    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 20:07:51.942943 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:07:51.942943 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 20:07:51.942943 ip-10-0-143-134 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7" podUID="92e0e41a-68f1-4ea2-8433-e1105e87b1b0" containerName="main" containerID="cri-o://a753d9a66b5c25d55e44d873a3aa1ad77f5277ea4bbaeef1432eb9e513232ca1"
Sep 18 20:07:51.942943 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:51.942625    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7" podUID="92e0e41a-68f1-4ea2-8433-e1105e87b1b0" containerName="main" containerID="cri-o://a753d9a66b5c25d55e44d873a3aa1ad77f5277ea4bbaeef1432eb9e513232ca1" gracePeriod=60
Sep 18 20:07:51.947360 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:51.947333    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf"
Sep 18 20:07:52.158763 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.158731    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7" podUID="92e0e41a-68f1-4ea2-8433-e1105e87b1b0" containerName="llm-d-routing-sidecar" containerID="cri-o://414b1495f975f577b6a9d49ff46328faed098938ce4b7c20fd6688eb48c6b696" gracePeriod=60
Sep 18 20:07:52.397687 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.397665    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"
Sep 18 20:07:52.464372 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.464351    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-home\") pod \"92e0e41a-68f1-4ea2-8433-e1105e87b1b0\" (UID: \"92e0e41a-68f1-4ea2-8433-e1105e87b1b0\") "
Sep 18 20:07:52.464538 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.464402    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-dshm\") pod \"92e0e41a-68f1-4ea2-8433-e1105e87b1b0\" (UID: \"92e0e41a-68f1-4ea2-8433-e1105e87b1b0\") "
Sep 18 20:07:52.464538 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.464423    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-tls-certs\") pod \"92e0e41a-68f1-4ea2-8433-e1105e87b1b0\" (UID: \"92e0e41a-68f1-4ea2-8433-e1105e87b1b0\") "
Sep 18 20:07:52.464538 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.464446    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-tmp-dir\") pod \"92e0e41a-68f1-4ea2-8433-e1105e87b1b0\" (UID: \"92e0e41a-68f1-4ea2-8433-e1105e87b1b0\") "
Sep 18 20:07:52.464538 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.464516    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-model-cache\") pod \"92e0e41a-68f1-4ea2-8433-e1105e87b1b0\" (UID: \"92e0e41a-68f1-4ea2-8433-e1105e87b1b0\") "
Sep 18 20:07:52.464750 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.464550    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sl6kf\" (UniqueName: \"kubernetes.io/projected/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-kube-api-access-sl6kf\") pod \"92e0e41a-68f1-4ea2-8433-e1105e87b1b0\" (UID: \"92e0e41a-68f1-4ea2-8433-e1105e87b1b0\") "
Sep 18 20:07:52.464750 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.464644    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-home" (OuterVolumeSpecName: "home") pod "92e0e41a-68f1-4ea2-8433-e1105e87b1b0" (UID: "92e0e41a-68f1-4ea2-8433-e1105e87b1b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:07:52.464750 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.464713    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "92e0e41a-68f1-4ea2-8433-e1105e87b1b0" (UID: "92e0e41a-68f1-4ea2-8433-e1105e87b1b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:07:52.464904 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.464790    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "92e0e41a-68f1-4ea2-8433-e1105e87b1b0" (UID: "92e0e41a-68f1-4ea2-8433-e1105e87b1b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:07:52.464904 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.464808    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-home\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:07:52.464904 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.464824    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-tmp-dir\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:07:52.466803 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.466782    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-dshm" (OuterVolumeSpecName: "dshm") pod "92e0e41a-68f1-4ea2-8433-e1105e87b1b0" (UID: "92e0e41a-68f1-4ea2-8433-e1105e87b1b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:07:52.466803 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.466789    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-kube-api-access-sl6kf" (OuterVolumeSpecName: "kube-api-access-sl6kf") pod "92e0e41a-68f1-4ea2-8433-e1105e87b1b0" (UID: "92e0e41a-68f1-4ea2-8433-e1105e87b1b0"). InnerVolumeSpecName "kube-api-access-sl6kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:07:52.466970 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.466801    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "92e0e41a-68f1-4ea2-8433-e1105e87b1b0" (UID: "92e0e41a-68f1-4ea2-8433-e1105e87b1b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:07:52.566013 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.565801    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-dshm\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:07:52.566013 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.565827    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-tls-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:07:52.566013 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.565865    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-model-cache\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:07:52.566013 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.565878    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl6kf\" (UniqueName: \"kubernetes.io/projected/92e0e41a-68f1-4ea2-8433-e1105e87b1b0-kube-api-access-sl6kf\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:07:52.942237 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.942205    2581 generic.go:356] "Generic (PLEG): container finished" podID="92e0e41a-68f1-4ea2-8433-e1105e87b1b0" containerID="a753d9a66b5c25d55e44d873a3aa1ad77f5277ea4bbaeef1432eb9e513232ca1" exitCode=0
Sep 18 20:07:52.942237 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.942234    2581 generic.go:356] "Generic (PLEG): container finished" podID="92e0e41a-68f1-4ea2-8433-e1105e87b1b0" containerID="414b1495f975f577b6a9d49ff46328faed098938ce4b7c20fd6688eb48c6b696" exitCode=0
Sep 18 20:07:52.942704 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.942262    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"
Sep 18 20:07:52.942704 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.942261    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7" event={"ID":"92e0e41a-68f1-4ea2-8433-e1105e87b1b0","Type":"ContainerDied","Data":"a753d9a66b5c25d55e44d873a3aa1ad77f5277ea4bbaeef1432eb9e513232ca1"}
Sep 18 20:07:52.942704 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.942326    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7" event={"ID":"92e0e41a-68f1-4ea2-8433-e1105e87b1b0","Type":"ContainerDied","Data":"414b1495f975f577b6a9d49ff46328faed098938ce4b7c20fd6688eb48c6b696"}
Sep 18 20:07:52.942704 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.942351    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7" event={"ID":"92e0e41a-68f1-4ea2-8433-e1105e87b1b0","Type":"ContainerDied","Data":"f31d46db00b7af3543009faf5951c263daa0e34058a5c8538b3e729211886a73"}
Sep 18 20:07:52.942704 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.942382    2581 scope.go:117] "RemoveContainer" containerID="a753d9a66b5c25d55e44d873a3aa1ad77f5277ea4bbaeef1432eb9e513232ca1"
Sep 18 20:07:52.953080 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.953062    2581 scope.go:117] "RemoveContainer" containerID="414b1495f975f577b6a9d49ff46328faed098938ce4b7c20fd6688eb48c6b696"
Sep 18 20:07:52.961059 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.961043    2581 scope.go:117] "RemoveContainer" containerID="a753d9a66b5c25d55e44d873a3aa1ad77f5277ea4bbaeef1432eb9e513232ca1"
Sep 18 20:07:52.961312 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:07:52.961294    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a753d9a66b5c25d55e44d873a3aa1ad77f5277ea4bbaeef1432eb9e513232ca1\": container with ID starting with a753d9a66b5c25d55e44d873a3aa1ad77f5277ea4bbaeef1432eb9e513232ca1 not found: ID does not exist" containerID="a753d9a66b5c25d55e44d873a3aa1ad77f5277ea4bbaeef1432eb9e513232ca1"
Sep 18 20:07:52.961349 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.961321    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a753d9a66b5c25d55e44d873a3aa1ad77f5277ea4bbaeef1432eb9e513232ca1"} err="failed to get container status \"a753d9a66b5c25d55e44d873a3aa1ad77f5277ea4bbaeef1432eb9e513232ca1\": rpc error: code = NotFound desc = could not find container \"a753d9a66b5c25d55e44d873a3aa1ad77f5277ea4bbaeef1432eb9e513232ca1\": container with ID starting with a753d9a66b5c25d55e44d873a3aa1ad77f5277ea4bbaeef1432eb9e513232ca1 not found: ID does not exist"
Sep 18 20:07:52.961349 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.961339    2581 scope.go:117] "RemoveContainer" containerID="414b1495f975f577b6a9d49ff46328faed098938ce4b7c20fd6688eb48c6b696"
Sep 18 20:07:52.961578 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:07:52.961559    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"414b1495f975f577b6a9d49ff46328faed098938ce4b7c20fd6688eb48c6b696\": container with ID starting with 414b1495f975f577b6a9d49ff46328faed098938ce4b7c20fd6688eb48c6b696 not found: ID does not exist" containerID="414b1495f975f577b6a9d49ff46328faed098938ce4b7c20fd6688eb48c6b696"
Sep 18 20:07:52.961617 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.961586    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"414b1495f975f577b6a9d49ff46328faed098938ce4b7c20fd6688eb48c6b696"} err="failed to get container status \"414b1495f975f577b6a9d49ff46328faed098938ce4b7c20fd6688eb48c6b696\": rpc error: code = NotFound desc = could not find container \"414b1495f975f577b6a9d49ff46328faed098938ce4b7c20fd6688eb48c6b696\": container with ID starting with 414b1495f975f577b6a9d49ff46328faed098938ce4b7c20fd6688eb48c6b696 not found: ID does not exist"
Sep 18 20:07:52.961617 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.961602    2581 scope.go:117] "RemoveContainer" containerID="a753d9a66b5c25d55e44d873a3aa1ad77f5277ea4bbaeef1432eb9e513232ca1"
Sep 18 20:07:52.961849 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.961816    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a753d9a66b5c25d55e44d873a3aa1ad77f5277ea4bbaeef1432eb9e513232ca1"} err="failed to get container status \"a753d9a66b5c25d55e44d873a3aa1ad77f5277ea4bbaeef1432eb9e513232ca1\": rpc error: code = NotFound desc = could not find container \"a753d9a66b5c25d55e44d873a3aa1ad77f5277ea4bbaeef1432eb9e513232ca1\": container with ID starting with a753d9a66b5c25d55e44d873a3aa1ad77f5277ea4bbaeef1432eb9e513232ca1 not found: ID does not exist"
Sep 18 20:07:52.961918 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.961851    2581 scope.go:117] "RemoveContainer" containerID="414b1495f975f577b6a9d49ff46328faed098938ce4b7c20fd6688eb48c6b696"
Sep 18 20:07:52.962205 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.962177    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"414b1495f975f577b6a9d49ff46328faed098938ce4b7c20fd6688eb48c6b696"} err="failed to get container status \"414b1495f975f577b6a9d49ff46328faed098938ce4b7c20fd6688eb48c6b696\": rpc error: code = NotFound desc = could not find container \"414b1495f975f577b6a9d49ff46328faed098938ce4b7c20fd6688eb48c6b696\": container with ID starting with 414b1495f975f577b6a9d49ff46328faed098938ce4b7c20fd6688eb48c6b696 not found: ID does not exist"
Sep 18 20:07:52.964784 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.964762    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"]
Sep 18 20:07:52.968684 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:52.968661    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-zgfv7"]
Sep 18 20:07:53.439810 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:07:53.439778    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92e0e41a-68f1-4ea2-8433-e1105e87b1b0" path="/var/lib/kubelet/pods/92e0e41a-68f1-4ea2-8433-e1105e87b1b0/volumes"
Sep 18 20:08:38.999956 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:08:38.999923    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf"]
Sep 18 20:08:54.019350 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:08:54.019289    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf" podUID="557d0e89-376a-4154-a271-727abdad13d1" containerName="main" containerID="cri-o://7175d2fde32398628e4a7e8ec04de9242ba6416577982e6a37d4eff6f4e9aa2c" gracePeriod=45
Sep 18 20:08:54.266228 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:08:54.266199    2581 generic.go:356] "Generic (PLEG): container finished" podID="557d0e89-376a-4154-a271-727abdad13d1" containerID="7175d2fde32398628e4a7e8ec04de9242ba6416577982e6a37d4eff6f4e9aa2c" exitCode=0
Sep 18 20:08:54.266361 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:08:54.266268    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf" event={"ID":"557d0e89-376a-4154-a271-727abdad13d1","Type":"ContainerDied","Data":"7175d2fde32398628e4a7e8ec04de9242ba6416577982e6a37d4eff6f4e9aa2c"}
Sep 18 20:08:54.266361 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:08:54.266313    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf" event={"ID":"557d0e89-376a-4154-a271-727abdad13d1","Type":"ContainerDied","Data":"bd5206409445dd2d0d351433432a2d7e76e5d0ac47f81712d37e54d32e71371f"}
Sep 18 20:08:54.266361 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:08:54.266329    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd5206409445dd2d0d351433432a2d7e76e5d0ac47f81712d37e54d32e71371f"
Sep 18 20:08:54.267661 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:08:54.267644    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf"
Sep 18 20:08:54.396194 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:08:54.396110    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/557d0e89-376a-4154-a271-727abdad13d1-model-cache\") pod \"557d0e89-376a-4154-a271-727abdad13d1\" (UID: \"557d0e89-376a-4154-a271-727abdad13d1\") "
Sep 18 20:08:54.396194 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:08:54.396181    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/557d0e89-376a-4154-a271-727abdad13d1-dshm\") pod \"557d0e89-376a-4154-a271-727abdad13d1\" (UID: \"557d0e89-376a-4154-a271-727abdad13d1\") "
Sep 18 20:08:54.396380 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:08:54.396215    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brb6s\" (UniqueName: \"kubernetes.io/projected/557d0e89-376a-4154-a271-727abdad13d1-kube-api-access-brb6s\") pod \"557d0e89-376a-4154-a271-727abdad13d1\" (UID: \"557d0e89-376a-4154-a271-727abdad13d1\") "
Sep 18 20:08:54.396380 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:08:54.396262    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/557d0e89-376a-4154-a271-727abdad13d1-tmp-dir\") pod \"557d0e89-376a-4154-a271-727abdad13d1\" (UID: \"557d0e89-376a-4154-a271-727abdad13d1\") "
Sep 18 20:08:54.396380 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:08:54.396298    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/557d0e89-376a-4154-a271-727abdad13d1-home\") pod \"557d0e89-376a-4154-a271-727abdad13d1\" (UID: \"557d0e89-376a-4154-a271-727abdad13d1\") "
Sep 18 20:08:54.396380 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:08:54.396329    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/557d0e89-376a-4154-a271-727abdad13d1-tls-certs\") pod \"557d0e89-376a-4154-a271-727abdad13d1\" (UID: \"557d0e89-376a-4154-a271-727abdad13d1\") "
Sep 18 20:08:54.396542 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:08:54.396416    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/557d0e89-376a-4154-a271-727abdad13d1-model-cache" (OuterVolumeSpecName: "model-cache") pod "557d0e89-376a-4154-a271-727abdad13d1" (UID: "557d0e89-376a-4154-a271-727abdad13d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:08:54.396582 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:08:54.396556    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/557d0e89-376a-4154-a271-727abdad13d1-home" (OuterVolumeSpecName: "home") pod "557d0e89-376a-4154-a271-727abdad13d1" (UID: "557d0e89-376a-4154-a271-727abdad13d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:08:54.396643 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:08:54.396572    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/557d0e89-376a-4154-a271-727abdad13d1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "557d0e89-376a-4154-a271-727abdad13d1" (UID: "557d0e89-376a-4154-a271-727abdad13d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:08:54.396696 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:08:54.396645    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/557d0e89-376a-4154-a271-727abdad13d1-model-cache\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:08:54.398598 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:08:54.398569    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/557d0e89-376a-4154-a271-727abdad13d1-dshm" (OuterVolumeSpecName: "dshm") pod "557d0e89-376a-4154-a271-727abdad13d1" (UID: "557d0e89-376a-4154-a271-727abdad13d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:08:54.398710 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:08:54.398617    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/557d0e89-376a-4154-a271-727abdad13d1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "557d0e89-376a-4154-a271-727abdad13d1" (UID: "557d0e89-376a-4154-a271-727abdad13d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:08:54.398710 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:08:54.398643    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/557d0e89-376a-4154-a271-727abdad13d1-kube-api-access-brb6s" (OuterVolumeSpecName: "kube-api-access-brb6s") pod "557d0e89-376a-4154-a271-727abdad13d1" (UID: "557d0e89-376a-4154-a271-727abdad13d1"). InnerVolumeSpecName "kube-api-access-brb6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:08:54.497630 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:08:54.497598    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/557d0e89-376a-4154-a271-727abdad13d1-dshm\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:08:54.497630 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:08:54.497626    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brb6s\" (UniqueName: \"kubernetes.io/projected/557d0e89-376a-4154-a271-727abdad13d1-kube-api-access-brb6s\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:08:54.497804 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:08:54.497657    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/557d0e89-376a-4154-a271-727abdad13d1-tmp-dir\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:08:54.497804 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:08:54.497669    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/557d0e89-376a-4154-a271-727abdad13d1-home\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:08:54.497804 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:08:54.497677    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/557d0e89-376a-4154-a271-727abdad13d1-tls-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:08:55.271658 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:08:55.271625    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf"
Sep 18 20:08:55.294091 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:08:55.294064    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf"]
Sep 18 20:08:55.296516 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:08:55.296495    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-sfqrf"]
Sep 18 20:08:55.433686 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:08:55.433653    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="557d0e89-376a-4154-a271-727abdad13d1" path="/var/lib/kubelet/pods/557d0e89-376a-4154-a271-727abdad13d1/volumes"
Sep 18 20:09:21.193707 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.193629    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"]
Sep 18 20:09:21.194161 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.193992    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92e0e41a-68f1-4ea2-8433-e1105e87b1b0" containerName="main"
Sep 18 20:09:21.194161 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.194003    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="92e0e41a-68f1-4ea2-8433-e1105e87b1b0" containerName="main"
Sep 18 20:09:21.194161 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.194013    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="557d0e89-376a-4154-a271-727abdad13d1" containerName="main"
Sep 18 20:09:21.194161 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.194019    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="557d0e89-376a-4154-a271-727abdad13d1" containerName="main"
Sep 18 20:09:21.194161 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.194034    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92e0e41a-68f1-4ea2-8433-e1105e87b1b0" containerName="llm-d-routing-sidecar"
Sep 18 20:09:21.194161 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.194040    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="92e0e41a-68f1-4ea2-8433-e1105e87b1b0" containerName="llm-d-routing-sidecar"
Sep 18 20:09:21.194161 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.194102    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="92e0e41a-68f1-4ea2-8433-e1105e87b1b0" containerName="llm-d-routing-sidecar"
Sep 18 20:09:21.194161 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.194111    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="557d0e89-376a-4154-a271-727abdad13d1" containerName="main"
Sep 18 20:09:21.194161 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.194120    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="92e0e41a-68f1-4ea2-8433-e1105e87b1b0" containerName="main"
Sep 18 20:09:21.197219 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.197203    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"
Sep 18 20:09:21.200594 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.200569    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 18 20:09:21.200774 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.200671    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 18 20:09:21.200774 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.200675    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 18 20:09:21.200926 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.200792    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-ghbfp\""
Sep 18 20:09:21.200926 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.200898    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 18 20:09:21.201044 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.200977    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:09:21.207685 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.207655    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"]
Sep 18 20:09:21.313080 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.313048    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/115831db-005b-4950-9f85-e9ae92b84598-home\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-thqtr\" (UID: \"115831db-005b-4950-9f85-e9ae92b84598\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"
Sep 18 20:09:21.313238 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.313086    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/115831db-005b-4950-9f85-e9ae92b84598-dshm\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-thqtr\" (UID: \"115831db-005b-4950-9f85-e9ae92b84598\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"
Sep 18 20:09:21.313238 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.313117    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/115831db-005b-4950-9f85-e9ae92b84598-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-thqtr\" (UID: \"115831db-005b-4950-9f85-e9ae92b84598\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"
Sep 18 20:09:21.313238 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.313174    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/115831db-005b-4950-9f85-e9ae92b84598-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-thqtr\" (UID: \"115831db-005b-4950-9f85-e9ae92b84598\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"
Sep 18 20:09:21.313238 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.313210    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/115831db-005b-4950-9f85-e9ae92b84598-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-thqtr\" (UID: \"115831db-005b-4950-9f85-e9ae92b84598\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"
Sep 18 20:09:21.313238 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.313232    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btt2f\" (UniqueName: \"kubernetes.io/projected/115831db-005b-4950-9f85-e9ae92b84598-kube-api-access-btt2f\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-thqtr\" (UID: \"115831db-005b-4950-9f85-e9ae92b84598\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"
Sep 18 20:09:21.313467 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.313264    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/115831db-005b-4950-9f85-e9ae92b84598-model-cache\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-thqtr\" (UID: \"115831db-005b-4950-9f85-e9ae92b84598\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"
Sep 18 20:09:21.313467 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.313302    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/115831db-005b-4950-9f85-e9ae92b84598-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-thqtr\" (UID: \"115831db-005b-4950-9f85-e9ae92b84598\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"
Sep 18 20:09:21.413830 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.413802    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/115831db-005b-4950-9f85-e9ae92b84598-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-thqtr\" (UID: \"115831db-005b-4950-9f85-e9ae92b84598\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"
Sep 18 20:09:21.413992 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.413883    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/115831db-005b-4950-9f85-e9ae92b84598-home\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-thqtr\" (UID: \"115831db-005b-4950-9f85-e9ae92b84598\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"
Sep 18 20:09:21.413992 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.413903    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/115831db-005b-4950-9f85-e9ae92b84598-dshm\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-thqtr\" (UID: \"115831db-005b-4950-9f85-e9ae92b84598\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"
Sep 18 20:09:21.413992 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.413932    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/115831db-005b-4950-9f85-e9ae92b84598-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-thqtr\" (UID: \"115831db-005b-4950-9f85-e9ae92b84598\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"
Sep 18 20:09:21.413992 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.413972    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/115831db-005b-4950-9f85-e9ae92b84598-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-thqtr\" (UID: \"115831db-005b-4950-9f85-e9ae92b84598\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"
Sep 18 20:09:21.414222 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.414009    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/115831db-005b-4950-9f85-e9ae92b84598-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-thqtr\" (UID: \"115831db-005b-4950-9f85-e9ae92b84598\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"
Sep 18 20:09:21.414222 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.414034    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btt2f\" (UniqueName: \"kubernetes.io/projected/115831db-005b-4950-9f85-e9ae92b84598-kube-api-access-btt2f\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-thqtr\" (UID: \"115831db-005b-4950-9f85-e9ae92b84598\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"
Sep 18 20:09:21.414222 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.414070    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/115831db-005b-4950-9f85-e9ae92b84598-model-cache\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-thqtr\" (UID: \"115831db-005b-4950-9f85-e9ae92b84598\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"
Sep 18 20:09:21.414372 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.414309    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/115831db-005b-4950-9f85-e9ae92b84598-home\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-thqtr\" (UID: \"115831db-005b-4950-9f85-e9ae92b84598\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"
Sep 18 20:09:21.414493 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.414472    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/115831db-005b-4950-9f85-e9ae92b84598-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-thqtr\" (UID: \"115831db-005b-4950-9f85-e9ae92b84598\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"
Sep 18 20:09:21.414549 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.414508    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/115831db-005b-4950-9f85-e9ae92b84598-model-cache\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-thqtr\" (UID: \"115831db-005b-4950-9f85-e9ae92b84598\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"
Sep 18 20:09:21.414549 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.414506    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/115831db-005b-4950-9f85-e9ae92b84598-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-thqtr\" (UID: \"115831db-005b-4950-9f85-e9ae92b84598\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"
Sep 18 20:09:21.414653 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.414583    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/115831db-005b-4950-9f85-e9ae92b84598-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-thqtr\" (UID: \"115831db-005b-4950-9f85-e9ae92b84598\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"
Sep 18 20:09:21.416381 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.416360    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/115831db-005b-4950-9f85-e9ae92b84598-dshm\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-thqtr\" (UID: \"115831db-005b-4950-9f85-e9ae92b84598\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"
Sep 18 20:09:21.416883 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.416819    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/115831db-005b-4950-9f85-e9ae92b84598-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-thqtr\" (UID: \"115831db-005b-4950-9f85-e9ae92b84598\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"
Sep 18 20:09:21.427105 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.427079    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btt2f\" (UniqueName: \"kubernetes.io/projected/115831db-005b-4950-9f85-e9ae92b84598-kube-api-access-btt2f\") pod \"tokenizer-clean-path-test-kserve-7bd7858c79-thqtr\" (UID: \"115831db-005b-4950-9f85-e9ae92b84598\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"
Sep 18 20:09:21.435601 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.435579    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p"]
Sep 18 20:09:21.440251 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.440228    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p"
Sep 18 20:09:21.450211 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.450185    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p"]
Sep 18 20:09:21.507699 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.507662    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"
Sep 18 20:09:21.515257 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.515231    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e21a5f1b-2ba0-47e7-9604-e383412704c2-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p\" (UID: \"e21a5f1b-2ba0-47e7-9604-e383412704c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p"
Sep 18 20:09:21.515378 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.515294    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e21a5f1b-2ba0-47e7-9604-e383412704c2-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p\" (UID: \"e21a5f1b-2ba0-47e7-9604-e383412704c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p"
Sep 18 20:09:21.515378 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.515330    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e21a5f1b-2ba0-47e7-9604-e383412704c2-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p\" (UID: \"e21a5f1b-2ba0-47e7-9604-e383412704c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p"
Sep 18 20:09:21.515378 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.515347    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e21a5f1b-2ba0-47e7-9604-e383412704c2-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p\" (UID: \"e21a5f1b-2ba0-47e7-9604-e383412704c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p"
Sep 18 20:09:21.616184 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.616153    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e21a5f1b-2ba0-47e7-9604-e383412704c2-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p\" (UID: \"e21a5f1b-2ba0-47e7-9604-e383412704c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p"
Sep 18 20:09:21.616313 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.616231    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e21a5f1b-2ba0-47e7-9604-e383412704c2-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p\" (UID: \"e21a5f1b-2ba0-47e7-9604-e383412704c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p"
Sep 18 20:09:21.616313 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.616263    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e21a5f1b-2ba0-47e7-9604-e383412704c2-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p\" (UID: \"e21a5f1b-2ba0-47e7-9604-e383412704c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p"
Sep 18 20:09:21.616313 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.616287    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e21a5f1b-2ba0-47e7-9604-e383412704c2-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p\" (UID: \"e21a5f1b-2ba0-47e7-9604-e383412704c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p"
Sep 18 20:09:21.616613 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.616594    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e21a5f1b-2ba0-47e7-9604-e383412704c2-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p\" (UID: \"e21a5f1b-2ba0-47e7-9604-e383412704c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p"
Sep 18 20:09:21.616666 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.616648    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e21a5f1b-2ba0-47e7-9604-e383412704c2-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p\" (UID: \"e21a5f1b-2ba0-47e7-9604-e383412704c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p"
Sep 18 20:09:21.616940 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.616923    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e21a5f1b-2ba0-47e7-9604-e383412704c2-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p\" (UID: \"e21a5f1b-2ba0-47e7-9604-e383412704c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p"
Sep 18 20:09:21.618757 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.618741    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e21a5f1b-2ba0-47e7-9604-e383412704c2-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p\" (UID: \"e21a5f1b-2ba0-47e7-9604-e383412704c2\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p"
Sep 18 20:09:21.752134 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.752051    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p"
Sep 18 20:09:21.782760 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:21.782735    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"]
Sep 18 20:09:21.784316 ip-10-0-143-134 kubenswrapper[2581]: W0918 20:09:21.784291    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod115831db_005b_4950_9f85_e9ae92b84598.slice/crio-d14b289450cd1b9d1b66395042a08a2ef49993225ab75723e70471addce7ee17 WatchSource:0}: Error finding container d14b289450cd1b9d1b66395042a08a2ef49993225ab75723e70471addce7ee17: Status 404 returned error can't find the container with id d14b289450cd1b9d1b66395042a08a2ef49993225ab75723e70471addce7ee17
Sep 18 20:09:22.028163 ip-10-0-143-134 kubenswrapper[2581]: W0918 20:09:22.028136    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode21a5f1b_2ba0_47e7_9604_e383412704c2.slice/crio-f3feda22cdace7de5572103d74345e971c30f6dcc5576e8fcf41d3d794607200 WatchSource:0}: Error finding container f3feda22cdace7de5572103d74345e971c30f6dcc5576e8fcf41d3d794607200: Status 404 returned error can't find the container with id f3feda22cdace7de5572103d74345e971c30f6dcc5576e8fcf41d3d794607200
Sep 18 20:09:22.028297 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:22.028179    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p"]
Sep 18 20:09:22.414296 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:22.414192    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p" event={"ID":"e21a5f1b-2ba0-47e7-9604-e383412704c2","Type":"ContainerStarted","Data":"6e862cafaa068b6af9f5461261e1588a2bc9992c1b492d2b9a7e57e19eb89f22"}
Sep 18 20:09:22.414296 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:22.414272    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p" event={"ID":"e21a5f1b-2ba0-47e7-9604-e383412704c2","Type":"ContainerStarted","Data":"f3feda22cdace7de5572103d74345e971c30f6dcc5576e8fcf41d3d794607200"}
Sep 18 20:09:22.416244 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:22.416213    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr" event={"ID":"115831db-005b-4950-9f85-e9ae92b84598","Type":"ContainerStarted","Data":"53d2ebdfd1cf3a5012a09788c0f88c68648a1ac84f47e9a01eeaa42c918bce62"}
Sep 18 20:09:22.416450 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:22.416425    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr" event={"ID":"115831db-005b-4950-9f85-e9ae92b84598","Type":"ContainerStarted","Data":"d14b289450cd1b9d1b66395042a08a2ef49993225ab75723e70471addce7ee17"}
Sep 18 20:09:24.446148 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:24.446112    2581 generic.go:356] "Generic (PLEG): container finished" podID="115831db-005b-4950-9f85-e9ae92b84598" containerID="53d2ebdfd1cf3a5012a09788c0f88c68648a1ac84f47e9a01eeaa42c918bce62" exitCode=0
Sep 18 20:09:24.446642 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:24.446203    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr" event={"ID":"115831db-005b-4950-9f85-e9ae92b84598","Type":"ContainerDied","Data":"53d2ebdfd1cf3a5012a09788c0f88c68648a1ac84f47e9a01eeaa42c918bce62"}
Sep 18 20:09:25.451910 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:25.451878    2581 generic.go:356] "Generic (PLEG): container finished" podID="e21a5f1b-2ba0-47e7-9604-e383412704c2" containerID="6e862cafaa068b6af9f5461261e1588a2bc9992c1b492d2b9a7e57e19eb89f22" exitCode=0
Sep 18 20:09:25.452331 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:25.451915    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p" event={"ID":"e21a5f1b-2ba0-47e7-9604-e383412704c2","Type":"ContainerDied","Data":"6e862cafaa068b6af9f5461261e1588a2bc9992c1b492d2b9a7e57e19eb89f22"}
Sep 18 20:09:25.454001 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:25.453983    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr" event={"ID":"115831db-005b-4950-9f85-e9ae92b84598","Type":"ContainerStarted","Data":"d799e764b0cc405feb43f7923972726d90831d88b47824d96d3a824c96500ee1"}
Sep 18 20:09:25.488766 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:25.488724    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr" podStartSLOduration=1.9720716010000001 podStartE2EDuration="4.48871053s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.516638929s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:09:25.48699389 +0000 UTC m=+1888.589282222" watchObservedRunningTime="2026-09-18 20:09:25.48871053 +0000 UTC m=+1888.590998858"
Sep 18 20:09:25.508105 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:25.508086    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"
Sep 18 20:09:31.508532 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:31.508494    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"
Sep 18 20:09:31.526328 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:31.526293    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"
Sep 18 20:09:32.521261 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:09:32.521229    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"
Sep 18 20:10:13.788326 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:10:13.788292    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p" event={"ID":"e21a5f1b-2ba0-47e7-9604-e383412704c2","Type":"ContainerStarted","Data":"2cac79e7d894b52762186b95a100d284de6660228cf6f691a622d900a7728b41"}
Sep 18 20:10:13.788736 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:10:13.788499    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p"
Sep 18 20:10:13.807252 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:10:13.807207    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p" podStartSLOduration=2.997190385 podStartE2EDuration="52.807192646s" totalImagesPullingTime="47.33159638s" totalInitContainerRuntime="2.478405881s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:09:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:10:13.806097487 +0000 UTC m=+1936.908385816" watchObservedRunningTime="2026-09-18 20:10:13.807192646 +0000 UTC m=+1936.909480974"
Sep 18 20:10:44.805795 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:10:44.805755    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p"
Sep 18 20:10:47.329625 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:10:47.329597    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p_e21a5f1b-2ba0-47e7-9604-e383412704c2/main/0.log"
Sep 18 20:10:59.817143 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:10:59.817061    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p"]
Sep 18 20:10:59.817576 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:10:59.817288    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p" podUID="e21a5f1b-2ba0-47e7-9604-e383412704c2" containerName="main" containerID="cri-o://2cac79e7d894b52762186b95a100d284de6660228cf6f691a622d900a7728b41" gracePeriod=30
Sep 18 20:10:59.826801 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:10:59.826772    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"]
Sep 18 20:10:59.836303 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:10:59.836269    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 20:10:59.836303 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:10:59.836303 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 20:10:59.836303 ip-10-0-143-134 kubenswrapper[2581]:  > containerID="d799e764b0cc405feb43f7923972726d90831d88b47824d96d3a824c96500ee1" cmd=["/bin/sleep","15"]
Sep 18 20:10:59.836551 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:10:59.836322    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr" message="" err=<
Sep 18 20:10:59.836551 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:10:59.836551 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 20:10:59.836551 ip-10-0-143-134 kubenswrapper[2581]:  >
Sep 18 20:10:59.836551 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:10:59.836334    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 20:10:59.836551 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:10:59.836551 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 20:10:59.836551 ip-10-0-143-134 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr" podUID="115831db-005b-4950-9f85-e9ae92b84598" containerName="main" containerID="cri-o://d799e764b0cc405feb43f7923972726d90831d88b47824d96d3a824c96500ee1"
Sep 18 20:10:59.836551 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:10:59.836373    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr" podUID="115831db-005b-4950-9f85-e9ae92b84598" containerName="main" containerID="cri-o://d799e764b0cc405feb43f7923972726d90831d88b47824d96d3a824c96500ee1" gracePeriod=60
Sep 18 20:11:00.054746 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.054715    2581 generic.go:356] "Generic (PLEG): container finished" podID="115831db-005b-4950-9f85-e9ae92b84598" containerID="d799e764b0cc405feb43f7923972726d90831d88b47824d96d3a824c96500ee1" exitCode=0
Sep 18 20:11:00.054955 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.054770    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr" event={"ID":"115831db-005b-4950-9f85-e9ae92b84598","Type":"ContainerDied","Data":"d799e764b0cc405feb43f7923972726d90831d88b47824d96d3a824c96500ee1"}
Sep 18 20:11:00.182811 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.182778    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"
Sep 18 20:11:00.366290 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.366268    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/115831db-005b-4950-9f85-e9ae92b84598-tmp-dir\") pod \"115831db-005b-4950-9f85-e9ae92b84598\" (UID: \"115831db-005b-4950-9f85-e9ae92b84598\") "
Sep 18 20:11:00.366387 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.366313    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/115831db-005b-4950-9f85-e9ae92b84598-model-cache\") pod \"115831db-005b-4950-9f85-e9ae92b84598\" (UID: \"115831db-005b-4950-9f85-e9ae92b84598\") "
Sep 18 20:11:00.366387 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.366352    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/115831db-005b-4950-9f85-e9ae92b84598-kserve-provision-location\") pod \"115831db-005b-4950-9f85-e9ae92b84598\" (UID: \"115831db-005b-4950-9f85-e9ae92b84598\") "
Sep 18 20:11:00.366479 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.366407    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/115831db-005b-4950-9f85-e9ae92b84598-cabundle-cert\") pod \"115831db-005b-4950-9f85-e9ae92b84598\" (UID: \"115831db-005b-4950-9f85-e9ae92b84598\") "
Sep 18 20:11:00.366479 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.366455    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btt2f\" (UniqueName: \"kubernetes.io/projected/115831db-005b-4950-9f85-e9ae92b84598-kube-api-access-btt2f\") pod \"115831db-005b-4950-9f85-e9ae92b84598\" (UID: \"115831db-005b-4950-9f85-e9ae92b84598\") "
Sep 18 20:11:00.366611 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.366497    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/115831db-005b-4950-9f85-e9ae92b84598-dshm\") pod \"115831db-005b-4950-9f85-e9ae92b84598\" (UID: \"115831db-005b-4950-9f85-e9ae92b84598\") "
Sep 18 20:11:00.366611 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.366582    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/115831db-005b-4950-9f85-e9ae92b84598-home\") pod \"115831db-005b-4950-9f85-e9ae92b84598\" (UID: \"115831db-005b-4950-9f85-e9ae92b84598\") "
Sep 18 20:11:00.366611 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.366592    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115831db-005b-4950-9f85-e9ae92b84598-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "115831db-005b-4950-9f85-e9ae92b84598" (UID: "115831db-005b-4950-9f85-e9ae92b84598"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:11:00.366744 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.366605    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115831db-005b-4950-9f85-e9ae92b84598-model-cache" (OuterVolumeSpecName: "model-cache") pod "115831db-005b-4950-9f85-e9ae92b84598" (UID: "115831db-005b-4950-9f85-e9ae92b84598"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:11:00.366744 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.366606    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/115831db-005b-4950-9f85-e9ae92b84598-tls-certs\") pod \"115831db-005b-4950-9f85-e9ae92b84598\" (UID: \"115831db-005b-4950-9f85-e9ae92b84598\") "
Sep 18 20:11:00.367063 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.366977    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115831db-005b-4950-9f85-e9ae92b84598-home" (OuterVolumeSpecName: "home") pod "115831db-005b-4950-9f85-e9ae92b84598" (UID: "115831db-005b-4950-9f85-e9ae92b84598"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:11:00.367063 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.367005    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/115831db-005b-4950-9f85-e9ae92b84598-tmp-dir\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:11:00.367063 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.367025    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/115831db-005b-4950-9f85-e9ae92b84598-model-cache\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:11:00.367257 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.367103    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/115831db-005b-4950-9f85-e9ae92b84598-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "115831db-005b-4950-9f85-e9ae92b84598" (UID: "115831db-005b-4950-9f85-e9ae92b84598"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:11:00.368944 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.368923    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/115831db-005b-4950-9f85-e9ae92b84598-kube-api-access-btt2f" (OuterVolumeSpecName: "kube-api-access-btt2f") pod "115831db-005b-4950-9f85-e9ae92b84598" (UID: "115831db-005b-4950-9f85-e9ae92b84598"). InnerVolumeSpecName "kube-api-access-btt2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:11:00.369372 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.369350    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115831db-005b-4950-9f85-e9ae92b84598-dshm" (OuterVolumeSpecName: "dshm") pod "115831db-005b-4950-9f85-e9ae92b84598" (UID: "115831db-005b-4950-9f85-e9ae92b84598"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:11:00.369444 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.369370    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/115831db-005b-4950-9f85-e9ae92b84598-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "115831db-005b-4950-9f85-e9ae92b84598" (UID: "115831db-005b-4950-9f85-e9ae92b84598"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:11:00.387171 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.387150    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p"
Sep 18 20:11:00.393644 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.393621    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115831db-005b-4950-9f85-e9ae92b84598-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "115831db-005b-4950-9f85-e9ae92b84598" (UID: "115831db-005b-4950-9f85-e9ae92b84598"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:11:00.467990 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.467958    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/115831db-005b-4950-9f85-e9ae92b84598-kserve-provision-location\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:11:00.467990 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.467983    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/115831db-005b-4950-9f85-e9ae92b84598-cabundle-cert\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:11:00.467990 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.467996    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btt2f\" (UniqueName: \"kubernetes.io/projected/115831db-005b-4950-9f85-e9ae92b84598-kube-api-access-btt2f\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:11:00.468216 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.468004    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/115831db-005b-4950-9f85-e9ae92b84598-dshm\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:11:00.468216 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.468014    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/115831db-005b-4950-9f85-e9ae92b84598-home\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:11:00.468216 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.468021    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/115831db-005b-4950-9f85-e9ae92b84598-tls-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:11:00.568444 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.568411    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e21a5f1b-2ba0-47e7-9604-e383412704c2-tmp-dir\") pod \"e21a5f1b-2ba0-47e7-9604-e383412704c2\" (UID: \"e21a5f1b-2ba0-47e7-9604-e383412704c2\") "
Sep 18 20:11:00.568607 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.568465    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e21a5f1b-2ba0-47e7-9604-e383412704c2-tls-certs\") pod \"e21a5f1b-2ba0-47e7-9604-e383412704c2\" (UID: \"e21a5f1b-2ba0-47e7-9604-e383412704c2\") "
Sep 18 20:11:00.568607 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.568567    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e21a5f1b-2ba0-47e7-9604-e383412704c2-cabundle-cert\") pod \"e21a5f1b-2ba0-47e7-9604-e383412704c2\" (UID: \"e21a5f1b-2ba0-47e7-9604-e383412704c2\") "
Sep 18 20:11:00.568795 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.568619    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e21a5f1b-2ba0-47e7-9604-e383412704c2-kserve-provision-location\") pod \"e21a5f1b-2ba0-47e7-9604-e383412704c2\" (UID: \"e21a5f1b-2ba0-47e7-9604-e383412704c2\") "
Sep 18 20:11:00.568947 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.568927    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e21a5f1b-2ba0-47e7-9604-e383412704c2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e21a5f1b-2ba0-47e7-9604-e383412704c2" (UID: "e21a5f1b-2ba0-47e7-9604-e383412704c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:11:00.569020 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.568980    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e21a5f1b-2ba0-47e7-9604-e383412704c2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e21a5f1b-2ba0-47e7-9604-e383412704c2" (UID: "e21a5f1b-2ba0-47e7-9604-e383412704c2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:11:00.570746 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.570726    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e21a5f1b-2ba0-47e7-9604-e383412704c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e21a5f1b-2ba0-47e7-9604-e383412704c2" (UID: "e21a5f1b-2ba0-47e7-9604-e383412704c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:11:00.590749 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.590705    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e21a5f1b-2ba0-47e7-9604-e383412704c2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e21a5f1b-2ba0-47e7-9604-e383412704c2" (UID: "e21a5f1b-2ba0-47e7-9604-e383412704c2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:11:00.669935 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.669851    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e21a5f1b-2ba0-47e7-9604-e383412704c2-cabundle-cert\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:11:00.669935 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.669884    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e21a5f1b-2ba0-47e7-9604-e383412704c2-kserve-provision-location\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:11:00.669935 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.669899    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e21a5f1b-2ba0-47e7-9604-e383412704c2-tmp-dir\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:11:00.669935 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:00.669911    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e21a5f1b-2ba0-47e7-9604-e383412704c2-tls-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:11:01.061721 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:01.061690    2581 generic.go:356] "Generic (PLEG): container finished" podID="e21a5f1b-2ba0-47e7-9604-e383412704c2" containerID="2cac79e7d894b52762186b95a100d284de6660228cf6f691a622d900a7728b41" exitCode=0
Sep 18 20:11:01.062162 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:01.061740    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p"
Sep 18 20:11:01.062162 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:01.061769    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p" event={"ID":"e21a5f1b-2ba0-47e7-9604-e383412704c2","Type":"ContainerDied","Data":"2cac79e7d894b52762186b95a100d284de6660228cf6f691a622d900a7728b41"}
Sep 18 20:11:01.062162 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:01.061828    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p" event={"ID":"e21a5f1b-2ba0-47e7-9604-e383412704c2","Type":"ContainerDied","Data":"f3feda22cdace7de5572103d74345e971c30f6dcc5576e8fcf41d3d794607200"}
Sep 18 20:11:01.062162 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:01.061877    2581 scope.go:117] "RemoveContainer" containerID="2cac79e7d894b52762186b95a100d284de6660228cf6f691a622d900a7728b41"
Sep 18 20:11:01.063553 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:01.063537    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"
Sep 18 20:11:01.063656 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:01.063581    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr" event={"ID":"115831db-005b-4950-9f85-e9ae92b84598","Type":"ContainerDied","Data":"d14b289450cd1b9d1b66395042a08a2ef49993225ab75723e70471addce7ee17"}
Sep 18 20:11:01.072364 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:01.072212    2581 scope.go:117] "RemoveContainer" containerID="6e862cafaa068b6af9f5461261e1588a2bc9992c1b492d2b9a7e57e19eb89f22"
Sep 18 20:11:01.081353 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:01.081337    2581 scope.go:117] "RemoveContainer" containerID="2cac79e7d894b52762186b95a100d284de6660228cf6f691a622d900a7728b41"
Sep 18 20:11:01.081623 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:11:01.081604    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cac79e7d894b52762186b95a100d284de6660228cf6f691a622d900a7728b41\": container with ID starting with 2cac79e7d894b52762186b95a100d284de6660228cf6f691a622d900a7728b41 not found: ID does not exist" containerID="2cac79e7d894b52762186b95a100d284de6660228cf6f691a622d900a7728b41"
Sep 18 20:11:01.081690 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:01.081632    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cac79e7d894b52762186b95a100d284de6660228cf6f691a622d900a7728b41"} err="failed to get container status \"2cac79e7d894b52762186b95a100d284de6660228cf6f691a622d900a7728b41\": rpc error: code = NotFound desc = could not find container \"2cac79e7d894b52762186b95a100d284de6660228cf6f691a622d900a7728b41\": container with ID starting with 2cac79e7d894b52762186b95a100d284de6660228cf6f691a622d900a7728b41 not found: ID does not exist"
Sep 18 20:11:01.081690 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:01.081649    2581 scope.go:117] "RemoveContainer" containerID="6e862cafaa068b6af9f5461261e1588a2bc9992c1b492d2b9a7e57e19eb89f22"
Sep 18 20:11:01.081875 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:11:01.081853    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e862cafaa068b6af9f5461261e1588a2bc9992c1b492d2b9a7e57e19eb89f22\": container with ID starting with 6e862cafaa068b6af9f5461261e1588a2bc9992c1b492d2b9a7e57e19eb89f22 not found: ID does not exist" containerID="6e862cafaa068b6af9f5461261e1588a2bc9992c1b492d2b9a7e57e19eb89f22"
Sep 18 20:11:01.081919 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:01.081883    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e862cafaa068b6af9f5461261e1588a2bc9992c1b492d2b9a7e57e19eb89f22"} err="failed to get container status \"6e862cafaa068b6af9f5461261e1588a2bc9992c1b492d2b9a7e57e19eb89f22\": rpc error: code = NotFound desc = could not find container \"6e862cafaa068b6af9f5461261e1588a2bc9992c1b492d2b9a7e57e19eb89f22\": container with ID starting with 6e862cafaa068b6af9f5461261e1588a2bc9992c1b492d2b9a7e57e19eb89f22 not found: ID does not exist"
Sep 18 20:11:01.081919 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:01.081897    2581 scope.go:117] "RemoveContainer" containerID="d799e764b0cc405feb43f7923972726d90831d88b47824d96d3a824c96500ee1"
Sep 18 20:11:01.086708 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:01.086688    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"]
Sep 18 20:11:01.090512 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:01.090473    2581 scope.go:117] "RemoveContainer" containerID="53d2ebdfd1cf3a5012a09788c0f88c68648a1ac84f47e9a01eeaa42c918bce62"
Sep 18 20:11:01.093536 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:01.093517    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bd7858c79-thqtr"]
Sep 18 20:11:01.103232 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:01.103211    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p"]
Sep 18 20:11:01.106724 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:01.106703    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f599fd57d-m5r2p"]
Sep 18 20:11:01.433980 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:01.433948    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="115831db-005b-4950-9f85-e9ae92b84598" path="/var/lib/kubelet/pods/115831db-005b-4950-9f85-e9ae92b84598/volumes"
Sep 18 20:11:01.434384 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:11:01.434372    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e21a5f1b-2ba0-47e7-9604-e383412704c2" path="/var/lib/kubelet/pods/e21a5f1b-2ba0-47e7-9604-e383412704c2/volumes"
Sep 18 20:12:13.510933 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.510894    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"]
Sep 18 20:12:13.511396 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.511296    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e21a5f1b-2ba0-47e7-9604-e383412704c2" containerName="main"
Sep 18 20:12:13.511396 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.511309    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e21a5f1b-2ba0-47e7-9604-e383412704c2" containerName="main"
Sep 18 20:12:13.511396 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.511318    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="115831db-005b-4950-9f85-e9ae92b84598" containerName="storage-initializer"
Sep 18 20:12:13.511396 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.511324    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="115831db-005b-4950-9f85-e9ae92b84598" containerName="storage-initializer"
Sep 18 20:12:13.511396 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.511334    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="115831db-005b-4950-9f85-e9ae92b84598" containerName="main"
Sep 18 20:12:13.511396 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.511339    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="115831db-005b-4950-9f85-e9ae92b84598" containerName="main"
Sep 18 20:12:13.511396 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.511360    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e21a5f1b-2ba0-47e7-9604-e383412704c2" containerName="storage-initializer"
Sep 18 20:12:13.511396 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.511365    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e21a5f1b-2ba0-47e7-9604-e383412704c2" containerName="storage-initializer"
Sep 18 20:12:13.511641 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.511418    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="115831db-005b-4950-9f85-e9ae92b84598" containerName="main"
Sep 18 20:12:13.511641 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.511429    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e21a5f1b-2ba0-47e7-9604-e383412704c2" containerName="main"
Sep 18 20:12:13.513287 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.513271    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"
Sep 18 20:12:13.515622 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.515599    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 18 20:12:13.516397 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.516374    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 18 20:12:13.516501 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.516399    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 18 20:12:13.517294 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.517275    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 18 20:12:13.517400 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.517293    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-7q8zm\""
Sep 18 20:12:13.517400 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.517282    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:12:13.526930 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.526909    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"]
Sep 18 20:12:13.577062 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.577026    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29b067e9-c882-4ab8-8dc5-4645a6114189-model-cache\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9\" (UID: \"29b067e9-c882-4ab8-8dc5-4645a6114189\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"
Sep 18 20:12:13.577062 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.577067    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29b067e9-c882-4ab8-8dc5-4645a6114189-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9\" (UID: \"29b067e9-c882-4ab8-8dc5-4645a6114189\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"
Sep 18 20:12:13.577275 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.577139    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b067e9-c882-4ab8-8dc5-4645a6114189-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9\" (UID: \"29b067e9-c882-4ab8-8dc5-4645a6114189\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"
Sep 18 20:12:13.577275 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.577184    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29b067e9-c882-4ab8-8dc5-4645a6114189-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9\" (UID: \"29b067e9-c882-4ab8-8dc5-4645a6114189\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"
Sep 18 20:12:13.577275 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.577216    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29b067e9-c882-4ab8-8dc5-4645a6114189-home\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9\" (UID: \"29b067e9-c882-4ab8-8dc5-4645a6114189\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"
Sep 18 20:12:13.577275 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.577239    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29b067e9-c882-4ab8-8dc5-4645a6114189-dshm\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9\" (UID: \"29b067e9-c882-4ab8-8dc5-4645a6114189\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"
Sep 18 20:12:13.577404 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.577289    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6vf5\" (UniqueName: \"kubernetes.io/projected/29b067e9-c882-4ab8-8dc5-4645a6114189-kube-api-access-l6vf5\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9\" (UID: \"29b067e9-c882-4ab8-8dc5-4645a6114189\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"
Sep 18 20:12:13.577404 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.577314    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29b067e9-c882-4ab8-8dc5-4645a6114189-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9\" (UID: \"29b067e9-c882-4ab8-8dc5-4645a6114189\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"
Sep 18 20:12:13.678555 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.678524    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29b067e9-c882-4ab8-8dc5-4645a6114189-model-cache\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9\" (UID: \"29b067e9-c882-4ab8-8dc5-4645a6114189\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"
Sep 18 20:12:13.678777 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.678564    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29b067e9-c882-4ab8-8dc5-4645a6114189-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9\" (UID: \"29b067e9-c882-4ab8-8dc5-4645a6114189\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"
Sep 18 20:12:13.678777 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.678687    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b067e9-c882-4ab8-8dc5-4645a6114189-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9\" (UID: \"29b067e9-c882-4ab8-8dc5-4645a6114189\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"
Sep 18 20:12:13.678777 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.678740    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29b067e9-c882-4ab8-8dc5-4645a6114189-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9\" (UID: \"29b067e9-c882-4ab8-8dc5-4645a6114189\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"
Sep 18 20:12:13.678939 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.678776    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29b067e9-c882-4ab8-8dc5-4645a6114189-home\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9\" (UID: \"29b067e9-c882-4ab8-8dc5-4645a6114189\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"
Sep 18 20:12:13.678939 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.678797    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29b067e9-c882-4ab8-8dc5-4645a6114189-dshm\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9\" (UID: \"29b067e9-c882-4ab8-8dc5-4645a6114189\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"
Sep 18 20:12:13.678939 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.678823    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6vf5\" (UniqueName: \"kubernetes.io/projected/29b067e9-c882-4ab8-8dc5-4645a6114189-kube-api-access-l6vf5\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9\" (UID: \"29b067e9-c882-4ab8-8dc5-4645a6114189\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"
Sep 18 20:12:13.678939 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.678874    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29b067e9-c882-4ab8-8dc5-4645a6114189-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9\" (UID: \"29b067e9-c882-4ab8-8dc5-4645a6114189\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"
Sep 18 20:12:13.679138 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.678932    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29b067e9-c882-4ab8-8dc5-4645a6114189-model-cache\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9\" (UID: \"29b067e9-c882-4ab8-8dc5-4645a6114189\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"
Sep 18 20:12:13.679138 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.679049    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b067e9-c882-4ab8-8dc5-4645a6114189-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9\" (UID: \"29b067e9-c882-4ab8-8dc5-4645a6114189\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"
Sep 18 20:12:13.679248 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.679148    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29b067e9-c882-4ab8-8dc5-4645a6114189-home\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9\" (UID: \"29b067e9-c882-4ab8-8dc5-4645a6114189\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"
Sep 18 20:12:13.679248 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.679225    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29b067e9-c882-4ab8-8dc5-4645a6114189-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9\" (UID: \"29b067e9-c882-4ab8-8dc5-4645a6114189\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"
Sep 18 20:12:13.679448 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.679431    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29b067e9-c882-4ab8-8dc5-4645a6114189-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9\" (UID: \"29b067e9-c882-4ab8-8dc5-4645a6114189\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"
Sep 18 20:12:13.681140 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.681119    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29b067e9-c882-4ab8-8dc5-4645a6114189-dshm\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9\" (UID: \"29b067e9-c882-4ab8-8dc5-4645a6114189\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"
Sep 18 20:12:13.681233 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.681218    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29b067e9-c882-4ab8-8dc5-4645a6114189-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9\" (UID: \"29b067e9-c882-4ab8-8dc5-4645a6114189\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"
Sep 18 20:12:13.687920 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.687901    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6vf5\" (UniqueName: \"kubernetes.io/projected/29b067e9-c882-4ab8-8dc5-4645a6114189-kube-api-access-l6vf5\") pod \"tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9\" (UID: \"29b067e9-c882-4ab8-8dc5-4645a6114189\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"
Sep 18 20:12:13.779310 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.779279    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t"]
Sep 18 20:12:13.782658 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.782636    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t"
Sep 18 20:12:13.798725 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.798698    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t"]
Sep 18 20:12:13.824425 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.824399    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"
Sep 18 20:12:13.882355 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.882324    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1962c2c5-b469-4a83-aecf-8943ac95665f-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t\" (UID: \"1962c2c5-b469-4a83-aecf-8943ac95665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t"
Sep 18 20:12:13.882487 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.882368    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1962c2c5-b469-4a83-aecf-8943ac95665f-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t\" (UID: \"1962c2c5-b469-4a83-aecf-8943ac95665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t"
Sep 18 20:12:13.882545 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.882490    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1962c2c5-b469-4a83-aecf-8943ac95665f-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t\" (UID: \"1962c2c5-b469-4a83-aecf-8943ac95665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t"
Sep 18 20:12:13.882545 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.882535    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1962c2c5-b469-4a83-aecf-8943ac95665f-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t\" (UID: \"1962c2c5-b469-4a83-aecf-8943ac95665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t"
Sep 18 20:12:13.983650 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.983607    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1962c2c5-b469-4a83-aecf-8943ac95665f-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t\" (UID: \"1962c2c5-b469-4a83-aecf-8943ac95665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t"
Sep 18 20:12:13.983650 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.983654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1962c2c5-b469-4a83-aecf-8943ac95665f-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t\" (UID: \"1962c2c5-b469-4a83-aecf-8943ac95665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t"
Sep 18 20:12:13.983921 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.983701    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1962c2c5-b469-4a83-aecf-8943ac95665f-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t\" (UID: \"1962c2c5-b469-4a83-aecf-8943ac95665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t"
Sep 18 20:12:13.983921 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.983717    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1962c2c5-b469-4a83-aecf-8943ac95665f-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t\" (UID: \"1962c2c5-b469-4a83-aecf-8943ac95665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t"
Sep 18 20:12:13.984066 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.984050    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1962c2c5-b469-4a83-aecf-8943ac95665f-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t\" (UID: \"1962c2c5-b469-4a83-aecf-8943ac95665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t"
Sep 18 20:12:13.984105 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.984072    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1962c2c5-b469-4a83-aecf-8943ac95665f-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t\" (UID: \"1962c2c5-b469-4a83-aecf-8943ac95665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t"
Sep 18 20:12:13.984403 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.984384    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1962c2c5-b469-4a83-aecf-8943ac95665f-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t\" (UID: \"1962c2c5-b469-4a83-aecf-8943ac95665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t"
Sep 18 20:12:13.986294 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:13.986274    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1962c2c5-b469-4a83-aecf-8943ac95665f-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t\" (UID: \"1962c2c5-b469-4a83-aecf-8943ac95665f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t"
Sep 18 20:12:14.095619 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:14.095527    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t"
Sep 18 20:12:14.099109 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:14.099084    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"]
Sep 18 20:12:14.100443 ip-10-0-143-134 kubenswrapper[2581]: W0918 20:12:14.100409    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b067e9_c882_4ab8_8dc5_4645a6114189.slice/crio-dfd13ce5698842511737d0e7f44e8a38f3297ef7f4361f46d2ee70bc7632161e WatchSource:0}: Error finding container dfd13ce5698842511737d0e7f44e8a38f3297ef7f4361f46d2ee70bc7632161e: Status 404 returned error can't find the container with id dfd13ce5698842511737d0e7f44e8a38f3297ef7f4361f46d2ee70bc7632161e
Sep 18 20:12:14.377549 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:14.377516    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t"]
Sep 18 20:12:14.379281 ip-10-0-143-134 kubenswrapper[2581]: W0918 20:12:14.379252    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1962c2c5_b469_4a83_aecf_8943ac95665f.slice/crio-de660dd35f2fb454c7e9a89455c3d8bb6f0e78f2c7044bc45f551ad00740fa75 WatchSource:0}: Error finding container de660dd35f2fb454c7e9a89455c3d8bb6f0e78f2c7044bc45f551ad00740fa75: Status 404 returned error can't find the container with id de660dd35f2fb454c7e9a89455c3d8bb6f0e78f2c7044bc45f551ad00740fa75
Sep 18 20:12:14.462026 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:14.461977    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9" event={"ID":"29b067e9-c882-4ab8-8dc5-4645a6114189","Type":"ContainerStarted","Data":"306b73a86889a18e9aa72d0d00c2f802ed81b30870cbb4b0787d5d8c85c35a2d"}
Sep 18 20:12:14.462203 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:14.462084    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9" event={"ID":"29b067e9-c882-4ab8-8dc5-4645a6114189","Type":"ContainerStarted","Data":"dfd13ce5698842511737d0e7f44e8a38f3297ef7f4361f46d2ee70bc7632161e"}
Sep 18 20:12:14.463879 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:14.463822    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t" event={"ID":"1962c2c5-b469-4a83-aecf-8943ac95665f","Type":"ContainerStarted","Data":"94aef1ad393280c6e4a33c35b0d8a9ff9c5b70b29ac4f1766844f08141f0ebcd"}
Sep 18 20:12:14.464001 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:14.463910    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t" event={"ID":"1962c2c5-b469-4a83-aecf-8943ac95665f","Type":"ContainerStarted","Data":"de660dd35f2fb454c7e9a89455c3d8bb6f0e78f2c7044bc45f551ad00740fa75"}
Sep 18 20:12:17.484242 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:17.484208    2581 generic.go:356] "Generic (PLEG): container finished" podID="29b067e9-c882-4ab8-8dc5-4645a6114189" containerID="306b73a86889a18e9aa72d0d00c2f802ed81b30870cbb4b0787d5d8c85c35a2d" exitCode=0
Sep 18 20:12:17.484703 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:17.484280    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9" event={"ID":"29b067e9-c882-4ab8-8dc5-4645a6114189","Type":"ContainerDied","Data":"306b73a86889a18e9aa72d0d00c2f802ed81b30870cbb4b0787d5d8c85c35a2d"}
Sep 18 20:12:17.485973 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:17.485951    2581 generic.go:356] "Generic (PLEG): container finished" podID="1962c2c5-b469-4a83-aecf-8943ac95665f" containerID="94aef1ad393280c6e4a33c35b0d8a9ff9c5b70b29ac4f1766844f08141f0ebcd" exitCode=0
Sep 18 20:12:17.486101 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:17.486010    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t" event={"ID":"1962c2c5-b469-4a83-aecf-8943ac95665f","Type":"ContainerDied","Data":"94aef1ad393280c6e4a33c35b0d8a9ff9c5b70b29ac4f1766844f08141f0ebcd"}
Sep 18 20:12:18.493374 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:18.493343    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9" event={"ID":"29b067e9-c882-4ab8-8dc5-4645a6114189","Type":"ContainerStarted","Data":"1ade058b1622ccd376612d62794b774248d0837ef8faca157dc243922edb083f"}
Sep 18 20:12:18.495514 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:18.495492    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t" event={"ID":"1962c2c5-b469-4a83-aecf-8943ac95665f","Type":"ContainerStarted","Data":"cc950469e7ea817948176ffbfe3e989882cada7049d8c2342145d537f2aa088d"}
Sep 18 20:12:18.495956 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:18.495935    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t"
Sep 18 20:12:18.515870 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:18.515817    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9" podStartSLOduration=3.06143631 podStartE2EDuration="5.515805206s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.454368896s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:12:18.513671303 +0000 UTC m=+2061.615959645" watchObservedRunningTime="2026-09-18 20:12:18.515805206 +0000 UTC m=+2061.618093534"
Sep 18 20:12:18.534171 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:18.534110    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t" podStartSLOduration=3.036106915 podStartE2EDuration="5.534089692s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.497982777s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:12:18.530200036 +0000 UTC m=+2061.632488362" watchObservedRunningTime="2026-09-18 20:12:18.534089692 +0000 UTC m=+2061.636378026"
Sep 18 20:12:18.824571 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:18.824531    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"
Sep 18 20:12:23.825406 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:23.825366    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"
Sep 18 20:12:23.837812 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:23.837783    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"
Sep 18 20:12:24.548280 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:24.548250    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"
Sep 18 20:12:49.511920 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:49.511885    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t"
Sep 18 20:12:54.038369 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:54.038335    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t_1962c2c5-b469-4a83-aecf-8943ac95665f/main/0.log"
Sep 18 20:12:56.434805 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.434769    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t"]
Sep 18 20:12:56.435279 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.435060    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t" podUID="1962c2c5-b469-4a83-aecf-8943ac95665f" containerName="main" containerID="cri-o://cc950469e7ea817948176ffbfe3e989882cada7049d8c2342145d537f2aa088d" gracePeriod=30
Sep 18 20:12:56.443977 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.443945    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"]
Sep 18 20:12:56.454871 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:12:56.454800    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 20:12:56.454871 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:12:56.454871 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 20:12:56.454871 ip-10-0-143-134 kubenswrapper[2581]:  > containerID="1ade058b1622ccd376612d62794b774248d0837ef8faca157dc243922edb083f" cmd=["/bin/sleep","15"]
Sep 18 20:12:56.455191 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.454882    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9" message="" err=<
Sep 18 20:12:56.455191 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:12:56.455191 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 20:12:56.455191 ip-10-0-143-134 kubenswrapper[2581]:  >
Sep 18 20:12:56.455191 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:12:56.454898    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 20:12:56.455191 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:12:56.455191 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 20:12:56.455191 ip-10-0-143-134 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9" podUID="29b067e9-c882-4ab8-8dc5-4645a6114189" containerName="main" containerID="cri-o://1ade058b1622ccd376612d62794b774248d0837ef8faca157dc243922edb083f"
Sep 18 20:12:56.455191 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.454946    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9" podUID="29b067e9-c882-4ab8-8dc5-4645a6114189" containerName="main" containerID="cri-o://1ade058b1622ccd376612d62794b774248d0837ef8faca157dc243922edb083f" gracePeriod=60
Sep 18 20:12:56.533886 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.533815    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9" podUID="29b067e9-c882-4ab8-8dc5-4645a6114189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Sep 18 20:12:56.726226 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.726195    2581 generic.go:356] "Generic (PLEG): container finished" podID="29b067e9-c882-4ab8-8dc5-4645a6114189" containerID="1ade058b1622ccd376612d62794b774248d0837ef8faca157dc243922edb083f" exitCode=0
Sep 18 20:12:56.726395 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.726226    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9" event={"ID":"29b067e9-c882-4ab8-8dc5-4645a6114189","Type":"ContainerDied","Data":"1ade058b1622ccd376612d62794b774248d0837ef8faca157dc243922edb083f"}
Sep 18 20:12:56.815213 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.815186    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"
Sep 18 20:12:56.866801 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.866778    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29b067e9-c882-4ab8-8dc5-4645a6114189-dshm\") pod \"29b067e9-c882-4ab8-8dc5-4645a6114189\" (UID: \"29b067e9-c882-4ab8-8dc5-4645a6114189\") "
Sep 18 20:12:56.866953 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.866812    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29b067e9-c882-4ab8-8dc5-4645a6114189-model-cache\") pod \"29b067e9-c882-4ab8-8dc5-4645a6114189\" (UID: \"29b067e9-c882-4ab8-8dc5-4645a6114189\") "
Sep 18 20:12:56.866953 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.866846    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29b067e9-c882-4ab8-8dc5-4645a6114189-home\") pod \"29b067e9-c882-4ab8-8dc5-4645a6114189\" (UID: \"29b067e9-c882-4ab8-8dc5-4645a6114189\") "
Sep 18 20:12:56.866953 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.866871    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6vf5\" (UniqueName: \"kubernetes.io/projected/29b067e9-c882-4ab8-8dc5-4645a6114189-kube-api-access-l6vf5\") pod \"29b067e9-c882-4ab8-8dc5-4645a6114189\" (UID: \"29b067e9-c882-4ab8-8dc5-4645a6114189\") "
Sep 18 20:12:56.866953 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.866899    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29b067e9-c882-4ab8-8dc5-4645a6114189-tls-certs\") pod \"29b067e9-c882-4ab8-8dc5-4645a6114189\" (UID: \"29b067e9-c882-4ab8-8dc5-4645a6114189\") "
Sep 18 20:12:56.866953 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.866923    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29b067e9-c882-4ab8-8dc5-4645a6114189-kserve-provision-location\") pod \"29b067e9-c882-4ab8-8dc5-4645a6114189\" (UID: \"29b067e9-c882-4ab8-8dc5-4645a6114189\") "
Sep 18 20:12:56.867226 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.866971    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b067e9-c882-4ab8-8dc5-4645a6114189-tmp-dir\") pod \"29b067e9-c882-4ab8-8dc5-4645a6114189\" (UID: \"29b067e9-c882-4ab8-8dc5-4645a6114189\") "
Sep 18 20:12:56.867226 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.866995    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29b067e9-c882-4ab8-8dc5-4645a6114189-cabundle-cert\") pod \"29b067e9-c882-4ab8-8dc5-4645a6114189\" (UID: \"29b067e9-c882-4ab8-8dc5-4645a6114189\") "
Sep 18 20:12:56.867226 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.867079    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b067e9-c882-4ab8-8dc5-4645a6114189-model-cache" (OuterVolumeSpecName: "model-cache") pod "29b067e9-c882-4ab8-8dc5-4645a6114189" (UID: "29b067e9-c882-4ab8-8dc5-4645a6114189"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:12:56.867226 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.867129    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b067e9-c882-4ab8-8dc5-4645a6114189-home" (OuterVolumeSpecName: "home") pod "29b067e9-c882-4ab8-8dc5-4645a6114189" (UID: "29b067e9-c882-4ab8-8dc5-4645a6114189"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:12:56.867444 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.867324    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29b067e9-c882-4ab8-8dc5-4645a6114189-model-cache\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:12:56.867444 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.867343    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29b067e9-c882-4ab8-8dc5-4645a6114189-home\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:12:56.867444 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.867324    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b067e9-c882-4ab8-8dc5-4645a6114189-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "29b067e9-c882-4ab8-8dc5-4645a6114189" (UID: "29b067e9-c882-4ab8-8dc5-4645a6114189"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:12:56.867646 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.867609    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29b067e9-c882-4ab8-8dc5-4645a6114189-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "29b067e9-c882-4ab8-8dc5-4645a6114189" (UID: "29b067e9-c882-4ab8-8dc5-4645a6114189"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:12:56.870621 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.870596    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b067e9-c882-4ab8-8dc5-4645a6114189-dshm" (OuterVolumeSpecName: "dshm") pod "29b067e9-c882-4ab8-8dc5-4645a6114189" (UID: "29b067e9-c882-4ab8-8dc5-4645a6114189"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:12:56.872918 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.872890    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29b067e9-c882-4ab8-8dc5-4645a6114189-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "29b067e9-c882-4ab8-8dc5-4645a6114189" (UID: "29b067e9-c882-4ab8-8dc5-4645a6114189"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:12:56.873016 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.872987    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b067e9-c882-4ab8-8dc5-4645a6114189-kube-api-access-l6vf5" (OuterVolumeSpecName: "kube-api-access-l6vf5") pod "29b067e9-c882-4ab8-8dc5-4645a6114189" (UID: "29b067e9-c882-4ab8-8dc5-4645a6114189"). InnerVolumeSpecName "kube-api-access-l6vf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:12:56.890118 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.890091    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b067e9-c882-4ab8-8dc5-4645a6114189-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "29b067e9-c882-4ab8-8dc5-4645a6114189" (UID: "29b067e9-c882-4ab8-8dc5-4645a6114189"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:12:56.968626 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.968542    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29b067e9-c882-4ab8-8dc5-4645a6114189-dshm\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:12:56.968626 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.968568    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6vf5\" (UniqueName: \"kubernetes.io/projected/29b067e9-c882-4ab8-8dc5-4645a6114189-kube-api-access-l6vf5\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:12:56.968626 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.968577    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29b067e9-c882-4ab8-8dc5-4645a6114189-tls-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:12:56.968626 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.968587    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29b067e9-c882-4ab8-8dc5-4645a6114189-kserve-provision-location\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:12:56.968626 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.968596    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29b067e9-c882-4ab8-8dc5-4645a6114189-tmp-dir\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:12:56.968626 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:56.968604    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29b067e9-c882-4ab8-8dc5-4645a6114189-cabundle-cert\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:12:57.075456 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:57.075431    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t"
Sep 18 20:12:57.171080 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:57.171050    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1962c2c5-b469-4a83-aecf-8943ac95665f-tls-certs\") pod \"1962c2c5-b469-4a83-aecf-8943ac95665f\" (UID: \"1962c2c5-b469-4a83-aecf-8943ac95665f\") "
Sep 18 20:12:57.171264 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:57.171131    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1962c2c5-b469-4a83-aecf-8943ac95665f-tmp-dir\") pod \"1962c2c5-b469-4a83-aecf-8943ac95665f\" (UID: \"1962c2c5-b469-4a83-aecf-8943ac95665f\") "
Sep 18 20:12:57.171264 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:57.171159    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1962c2c5-b469-4a83-aecf-8943ac95665f-kserve-provision-location\") pod \"1962c2c5-b469-4a83-aecf-8943ac95665f\" (UID: \"1962c2c5-b469-4a83-aecf-8943ac95665f\") "
Sep 18 20:12:57.171264 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:57.171202    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1962c2c5-b469-4a83-aecf-8943ac95665f-cabundle-cert\") pod \"1962c2c5-b469-4a83-aecf-8943ac95665f\" (UID: \"1962c2c5-b469-4a83-aecf-8943ac95665f\") "
Sep 18 20:12:57.171607 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:57.171581    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1962c2c5-b469-4a83-aecf-8943ac95665f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1962c2c5-b469-4a83-aecf-8943ac95665f" (UID: "1962c2c5-b469-4a83-aecf-8943ac95665f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:12:57.171668 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:57.171583    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1962c2c5-b469-4a83-aecf-8943ac95665f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1962c2c5-b469-4a83-aecf-8943ac95665f" (UID: "1962c2c5-b469-4a83-aecf-8943ac95665f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:12:57.173425 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:57.173404    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1962c2c5-b469-4a83-aecf-8943ac95665f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1962c2c5-b469-4a83-aecf-8943ac95665f" (UID: "1962c2c5-b469-4a83-aecf-8943ac95665f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:12:57.194416 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:57.194390    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1962c2c5-b469-4a83-aecf-8943ac95665f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1962c2c5-b469-4a83-aecf-8943ac95665f" (UID: "1962c2c5-b469-4a83-aecf-8943ac95665f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:12:57.272783 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:57.272754    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1962c2c5-b469-4a83-aecf-8943ac95665f-tmp-dir\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:12:57.272783 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:57.272780    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1962c2c5-b469-4a83-aecf-8943ac95665f-kserve-provision-location\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:12:57.272959 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:57.272795    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1962c2c5-b469-4a83-aecf-8943ac95665f-cabundle-cert\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:12:57.272959 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:57.272807    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1962c2c5-b469-4a83-aecf-8943ac95665f-tls-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:12:57.733319 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:57.733287    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"
Sep 18 20:12:57.733763 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:57.733293    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9" event={"ID":"29b067e9-c882-4ab8-8dc5-4645a6114189","Type":"ContainerDied","Data":"dfd13ce5698842511737d0e7f44e8a38f3297ef7f4361f46d2ee70bc7632161e"}
Sep 18 20:12:57.733763 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:57.733453    2581 scope.go:117] "RemoveContainer" containerID="1ade058b1622ccd376612d62794b774248d0837ef8faca157dc243922edb083f"
Sep 18 20:12:57.735328 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:57.735297    2581 generic.go:356] "Generic (PLEG): container finished" podID="1962c2c5-b469-4a83-aecf-8943ac95665f" containerID="cc950469e7ea817948176ffbfe3e989882cada7049d8c2342145d537f2aa088d" exitCode=0
Sep 18 20:12:57.735448 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:57.735390    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t" event={"ID":"1962c2c5-b469-4a83-aecf-8943ac95665f","Type":"ContainerDied","Data":"cc950469e7ea817948176ffbfe3e989882cada7049d8c2342145d537f2aa088d"}
Sep 18 20:12:57.735448 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:57.735405    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t"
Sep 18 20:12:57.735448 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:57.735436    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t" event={"ID":"1962c2c5-b469-4a83-aecf-8943ac95665f","Type":"ContainerDied","Data":"de660dd35f2fb454c7e9a89455c3d8bb6f0e78f2c7044bc45f551ad00740fa75"}
Sep 18 20:12:57.742983 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:57.742964    2581 scope.go:117] "RemoveContainer" containerID="306b73a86889a18e9aa72d0d00c2f802ed81b30870cbb4b0787d5d8c85c35a2d"
Sep 18 20:12:57.751282 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:57.751259    2581 scope.go:117] "RemoveContainer" containerID="cc950469e7ea817948176ffbfe3e989882cada7049d8c2342145d537f2aa088d"
Sep 18 20:12:57.753554 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:57.753523    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t"]
Sep 18 20:12:57.755777 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:57.755756    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-666856fcb4-86d5t"]
Sep 18 20:12:57.759891 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:57.759875    2581 scope.go:117] "RemoveContainer" containerID="94aef1ad393280c6e4a33c35b0d8a9ff9c5b70b29ac4f1766844f08141f0ebcd"
Sep 18 20:12:57.766160 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:57.766139    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"]
Sep 18 20:12:57.768131 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:57.768085    2581 scope.go:117] "RemoveContainer" containerID="cc950469e7ea817948176ffbfe3e989882cada7049d8c2342145d537f2aa088d"
Sep 18 20:12:57.768413 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:12:57.768392    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc950469e7ea817948176ffbfe3e989882cada7049d8c2342145d537f2aa088d\": container with ID starting with cc950469e7ea817948176ffbfe3e989882cada7049d8c2342145d537f2aa088d not found: ID does not exist" containerID="cc950469e7ea817948176ffbfe3e989882cada7049d8c2342145d537f2aa088d"
Sep 18 20:12:57.768507 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:57.768420    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc950469e7ea817948176ffbfe3e989882cada7049d8c2342145d537f2aa088d"} err="failed to get container status \"cc950469e7ea817948176ffbfe3e989882cada7049d8c2342145d537f2aa088d\": rpc error: code = NotFound desc = could not find container \"cc950469e7ea817948176ffbfe3e989882cada7049d8c2342145d537f2aa088d\": container with ID starting with cc950469e7ea817948176ffbfe3e989882cada7049d8c2342145d537f2aa088d not found: ID does not exist"
Sep 18 20:12:57.768507 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:57.768437    2581 scope.go:117] "RemoveContainer" containerID="94aef1ad393280c6e4a33c35b0d8a9ff9c5b70b29ac4f1766844f08141f0ebcd"
Sep 18 20:12:57.768727 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:12:57.768712    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94aef1ad393280c6e4a33c35b0d8a9ff9c5b70b29ac4f1766844f08141f0ebcd\": container with ID starting with 94aef1ad393280c6e4a33c35b0d8a9ff9c5b70b29ac4f1766844f08141f0ebcd not found: ID does not exist" containerID="94aef1ad393280c6e4a33c35b0d8a9ff9c5b70b29ac4f1766844f08141f0ebcd"
Sep 18 20:12:57.768776 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:57.768730    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94aef1ad393280c6e4a33c35b0d8a9ff9c5b70b29ac4f1766844f08141f0ebcd"} err="failed to get container status \"94aef1ad393280c6e4a33c35b0d8a9ff9c5b70b29ac4f1766844f08141f0ebcd\": rpc error: code = NotFound desc = could not find container \"94aef1ad393280c6e4a33c35b0d8a9ff9c5b70b29ac4f1766844f08141f0ebcd\": container with ID starting with 94aef1ad393280c6e4a33c35b0d8a9ff9c5b70b29ac4f1766844f08141f0ebcd not found: ID does not exist"
Sep 18 20:12:57.769974 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:57.769953    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6b48b9d9c8-8dxn9"]
Sep 18 20:12:59.432964 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:59.432930    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1962c2c5-b469-4a83-aecf-8943ac95665f" path="/var/lib/kubelet/pods/1962c2c5-b469-4a83-aecf-8943ac95665f/volumes"
Sep 18 20:12:59.433398 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:12:59.433386    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29b067e9-c882-4ab8-8dc5-4645a6114189" path="/var/lib/kubelet/pods/29b067e9-c882-4ab8-8dc5-4645a6114189/volumes"
Sep 18 20:13:57.658648 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:13:57.658581    2581 scope.go:117] "RemoveContainer" containerID="7175d2fde32398628e4a7e8ec04de9242ba6416577982e6a37d4eff6f4e9aa2c"
Sep 18 20:14:19.471781 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.471750    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"]
Sep 18 20:14:19.472189 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.472136    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29b067e9-c882-4ab8-8dc5-4645a6114189" containerName="storage-initializer"
Sep 18 20:14:19.472189 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.472153    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="29b067e9-c882-4ab8-8dc5-4645a6114189" containerName="storage-initializer"
Sep 18 20:14:19.472189 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.472167    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1962c2c5-b469-4a83-aecf-8943ac95665f" containerName="main"
Sep 18 20:14:19.472189 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.472173    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1962c2c5-b469-4a83-aecf-8943ac95665f" containerName="main"
Sep 18 20:14:19.472189 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.472189    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1962c2c5-b469-4a83-aecf-8943ac95665f" containerName="storage-initializer"
Sep 18 20:14:19.472351 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.472194    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1962c2c5-b469-4a83-aecf-8943ac95665f" containerName="storage-initializer"
Sep 18 20:14:19.472351 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.472201    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29b067e9-c882-4ab8-8dc5-4645a6114189" containerName="main"
Sep 18 20:14:19.472351 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.472205    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="29b067e9-c882-4ab8-8dc5-4645a6114189" containerName="main"
Sep 18 20:14:19.472351 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.472264    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1962c2c5-b469-4a83-aecf-8943ac95665f" containerName="main"
Sep 18 20:14:19.472351 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.472275    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="29b067e9-c882-4ab8-8dc5-4645a6114189" containerName="main"
Sep 18 20:14:19.475499 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.475483    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"
Sep 18 20:14:19.481143 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.481123    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:14:19.481298 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.481129    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 18 20:14:19.482250 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.482206    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-4jbhf\""
Sep 18 20:14:19.482327 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.482250    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 18 20:14:19.482458 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.482442    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 18 20:14:19.482513 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.482446    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 18 20:14:19.500403 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.500381    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"]
Sep 18 20:14:19.547494 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.547467    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fc08935-8e59-4724-813b-094750116547-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-649b57846b-dq8pn\" (UID: \"6fc08935-8e59-4724-813b-094750116547\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"
Sep 18 20:14:19.547612 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.547508    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fc08935-8e59-4724-813b-094750116547-home\") pod \"tokenizer-template-args-test-kserve-649b57846b-dq8pn\" (UID: \"6fc08935-8e59-4724-813b-094750116547\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"
Sep 18 20:14:19.547612 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.547547    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fc08935-8e59-4724-813b-094750116547-model-cache\") pod \"tokenizer-template-args-test-kserve-649b57846b-dq8pn\" (UID: \"6fc08935-8e59-4724-813b-094750116547\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"
Sep 18 20:14:19.547612 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.547566    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fc08935-8e59-4724-813b-094750116547-dshm\") pod \"tokenizer-template-args-test-kserve-649b57846b-dq8pn\" (UID: \"6fc08935-8e59-4724-813b-094750116547\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"
Sep 18 20:14:19.547612 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.547610    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fc08935-8e59-4724-813b-094750116547-tls-certs\") pod \"tokenizer-template-args-test-kserve-649b57846b-dq8pn\" (UID: \"6fc08935-8e59-4724-813b-094750116547\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"
Sep 18 20:14:19.547863 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.547748    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf8ls\" (UniqueName: \"kubernetes.io/projected/6fc08935-8e59-4724-813b-094750116547-kube-api-access-vf8ls\") pod \"tokenizer-template-args-test-kserve-649b57846b-dq8pn\" (UID: \"6fc08935-8e59-4724-813b-094750116547\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"
Sep 18 20:14:19.547863 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.547798    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fc08935-8e59-4724-813b-094750116547-tmp-dir\") pod \"tokenizer-template-args-test-kserve-649b57846b-dq8pn\" (UID: \"6fc08935-8e59-4724-813b-094750116547\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"
Sep 18 20:14:19.548131 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.547862    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fc08935-8e59-4724-813b-094750116547-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-649b57846b-dq8pn\" (UID: \"6fc08935-8e59-4724-813b-094750116547\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"
Sep 18 20:14:19.648713 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.648681    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fc08935-8e59-4724-813b-094750116547-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-649b57846b-dq8pn\" (UID: \"6fc08935-8e59-4724-813b-094750116547\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"
Sep 18 20:14:19.648886 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.648729    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fc08935-8e59-4724-813b-094750116547-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-649b57846b-dq8pn\" (UID: \"6fc08935-8e59-4724-813b-094750116547\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"
Sep 18 20:14:19.648886 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.648764    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fc08935-8e59-4724-813b-094750116547-home\") pod \"tokenizer-template-args-test-kserve-649b57846b-dq8pn\" (UID: \"6fc08935-8e59-4724-813b-094750116547\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"
Sep 18 20:14:19.648886 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.648805    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fc08935-8e59-4724-813b-094750116547-model-cache\") pod \"tokenizer-template-args-test-kserve-649b57846b-dq8pn\" (UID: \"6fc08935-8e59-4724-813b-094750116547\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"
Sep 18 20:14:19.649054 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.648903    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fc08935-8e59-4724-813b-094750116547-dshm\") pod \"tokenizer-template-args-test-kserve-649b57846b-dq8pn\" (UID: \"6fc08935-8e59-4724-813b-094750116547\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"
Sep 18 20:14:19.649054 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.648987    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fc08935-8e59-4724-813b-094750116547-tls-certs\") pod \"tokenizer-template-args-test-kserve-649b57846b-dq8pn\" (UID: \"6fc08935-8e59-4724-813b-094750116547\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"
Sep 18 20:14:19.649054 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.649045    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf8ls\" (UniqueName: \"kubernetes.io/projected/6fc08935-8e59-4724-813b-094750116547-kube-api-access-vf8ls\") pod \"tokenizer-template-args-test-kserve-649b57846b-dq8pn\" (UID: \"6fc08935-8e59-4724-813b-094750116547\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"
Sep 18 20:14:19.649257 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.649086    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fc08935-8e59-4724-813b-094750116547-tmp-dir\") pod \"tokenizer-template-args-test-kserve-649b57846b-dq8pn\" (UID: \"6fc08935-8e59-4724-813b-094750116547\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"
Sep 18 20:14:19.649257 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.649144    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fc08935-8e59-4724-813b-094750116547-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-649b57846b-dq8pn\" (UID: \"6fc08935-8e59-4724-813b-094750116547\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"
Sep 18 20:14:19.649379 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.649266    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fc08935-8e59-4724-813b-094750116547-home\") pod \"tokenizer-template-args-test-kserve-649b57846b-dq8pn\" (UID: \"6fc08935-8e59-4724-813b-094750116547\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"
Sep 18 20:14:19.649430 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.649379    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fc08935-8e59-4724-813b-094750116547-tmp-dir\") pod \"tokenizer-template-args-test-kserve-649b57846b-dq8pn\" (UID: \"6fc08935-8e59-4724-813b-094750116547\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"
Sep 18 20:14:19.649430 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.649394    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fc08935-8e59-4724-813b-094750116547-model-cache\") pod \"tokenizer-template-args-test-kserve-649b57846b-dq8pn\" (UID: \"6fc08935-8e59-4724-813b-094750116547\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"
Sep 18 20:14:19.649531 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.649445    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fc08935-8e59-4724-813b-094750116547-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-649b57846b-dq8pn\" (UID: \"6fc08935-8e59-4724-813b-094750116547\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"
Sep 18 20:14:19.651359 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.651337    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fc08935-8e59-4724-813b-094750116547-dshm\") pod \"tokenizer-template-args-test-kserve-649b57846b-dq8pn\" (UID: \"6fc08935-8e59-4724-813b-094750116547\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"
Sep 18 20:14:19.651634 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.651617    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fc08935-8e59-4724-813b-094750116547-tls-certs\") pod \"tokenizer-template-args-test-kserve-649b57846b-dq8pn\" (UID: \"6fc08935-8e59-4724-813b-094750116547\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"
Sep 18 20:14:19.661902 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.661878    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf8ls\" (UniqueName: \"kubernetes.io/projected/6fc08935-8e59-4724-813b-094750116547-kube-api-access-vf8ls\") pod \"tokenizer-template-args-test-kserve-649b57846b-dq8pn\" (UID: \"6fc08935-8e59-4724-813b-094750116547\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"
Sep 18 20:14:19.785870 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.785823    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"
Sep 18 20:14:19.908525 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.908489    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm"]
Sep 18 20:14:19.913706 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.913687    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm"
Sep 18 20:14:19.959733 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:19.959706    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm"]
Sep 18 20:14:20.052649 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:20.052571    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bacd4494-7d01-45b1-b8b7-2ca15f51d4b6-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm\" (UID: \"bacd4494-7d01-45b1-b8b7-2ca15f51d4b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm"
Sep 18 20:14:20.052649 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:20.052622    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bacd4494-7d01-45b1-b8b7-2ca15f51d4b6-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm\" (UID: \"bacd4494-7d01-45b1-b8b7-2ca15f51d4b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm"
Sep 18 20:14:20.052649 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:20.052643    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bacd4494-7d01-45b1-b8b7-2ca15f51d4b6-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm\" (UID: \"bacd4494-7d01-45b1-b8b7-2ca15f51d4b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm"
Sep 18 20:14:20.052869 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:20.052687    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bacd4494-7d01-45b1-b8b7-2ca15f51d4b6-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm\" (UID: \"bacd4494-7d01-45b1-b8b7-2ca15f51d4b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm"
Sep 18 20:14:20.117637 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:20.117614    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"]
Sep 18 20:14:20.120104 ip-10-0-143-134 kubenswrapper[2581]: W0918 20:14:20.120081    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc08935_8e59_4724_813b_094750116547.slice/crio-756023c505148e52b7b33f98a4ae2cd8ab78d0bc94bda695db405dbca207f5bf WatchSource:0}: Error finding container 756023c505148e52b7b33f98a4ae2cd8ab78d0bc94bda695db405dbca207f5bf: Status 404 returned error can't find the container with id 756023c505148e52b7b33f98a4ae2cd8ab78d0bc94bda695db405dbca207f5bf
Sep 18 20:14:20.153781 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:20.153759    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bacd4494-7d01-45b1-b8b7-2ca15f51d4b6-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm\" (UID: \"bacd4494-7d01-45b1-b8b7-2ca15f51d4b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm"
Sep 18 20:14:20.153918 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:20.153805    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bacd4494-7d01-45b1-b8b7-2ca15f51d4b6-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm\" (UID: \"bacd4494-7d01-45b1-b8b7-2ca15f51d4b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm"
Sep 18 20:14:20.153918 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:20.153859    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bacd4494-7d01-45b1-b8b7-2ca15f51d4b6-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm\" (UID: \"bacd4494-7d01-45b1-b8b7-2ca15f51d4b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm"
Sep 18 20:14:20.153918 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:20.153911    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bacd4494-7d01-45b1-b8b7-2ca15f51d4b6-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm\" (UID: \"bacd4494-7d01-45b1-b8b7-2ca15f51d4b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm"
Sep 18 20:14:20.154171 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:20.154152    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bacd4494-7d01-45b1-b8b7-2ca15f51d4b6-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm\" (UID: \"bacd4494-7d01-45b1-b8b7-2ca15f51d4b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm"
Sep 18 20:14:20.154235 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:20.154188    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bacd4494-7d01-45b1-b8b7-2ca15f51d4b6-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm\" (UID: \"bacd4494-7d01-45b1-b8b7-2ca15f51d4b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm"
Sep 18 20:14:20.154595 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:20.154574    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bacd4494-7d01-45b1-b8b7-2ca15f51d4b6-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm\" (UID: \"bacd4494-7d01-45b1-b8b7-2ca15f51d4b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm"
Sep 18 20:14:20.156565 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:20.156543    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bacd4494-7d01-45b1-b8b7-2ca15f51d4b6-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm\" (UID: \"bacd4494-7d01-45b1-b8b7-2ca15f51d4b6\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm"
Sep 18 20:14:20.197241 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:20.197202    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn" event={"ID":"6fc08935-8e59-4724-813b-094750116547","Type":"ContainerStarted","Data":"e98822727be3cc7b448846333b1fbb9566fc770b9943b3ebf40b5b71fd0737ae"}
Sep 18 20:14:20.197354 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:20.197255    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn" event={"ID":"6fc08935-8e59-4724-813b-094750116547","Type":"ContainerStarted","Data":"756023c505148e52b7b33f98a4ae2cd8ab78d0bc94bda695db405dbca207f5bf"}
Sep 18 20:14:20.226659 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:20.226632    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm"
Sep 18 20:14:20.519275 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:20.519248    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm"]
Sep 18 20:14:20.521096 ip-10-0-143-134 kubenswrapper[2581]: W0918 20:14:20.521065    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbacd4494_7d01_45b1_b8b7_2ca15f51d4b6.slice/crio-4e508ce31adf73edcac0df44990747fada7ffe286ed43438311abd93bed33a03 WatchSource:0}: Error finding container 4e508ce31adf73edcac0df44990747fada7ffe286ed43438311abd93bed33a03: Status 404 returned error can't find the container with id 4e508ce31adf73edcac0df44990747fada7ffe286ed43438311abd93bed33a03
Sep 18 20:14:21.207015 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:21.206971    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm" event={"ID":"bacd4494-7d01-45b1-b8b7-2ca15f51d4b6","Type":"ContainerStarted","Data":"4f6ef7e9ba08f5f638610b1bd18e79a362adfa27b696bb4d984169d6d35bdfc6"}
Sep 18 20:14:21.207504 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:21.207478    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm" event={"ID":"bacd4494-7d01-45b1-b8b7-2ca15f51d4b6","Type":"ContainerStarted","Data":"4e508ce31adf73edcac0df44990747fada7ffe286ed43438311abd93bed33a03"}
Sep 18 20:14:23.222247 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:23.222215    2581 generic.go:356] "Generic (PLEG): container finished" podID="bacd4494-7d01-45b1-b8b7-2ca15f51d4b6" containerID="4f6ef7e9ba08f5f638610b1bd18e79a362adfa27b696bb4d984169d6d35bdfc6" exitCode=0
Sep 18 20:14:23.222745 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:23.222298    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm" event={"ID":"bacd4494-7d01-45b1-b8b7-2ca15f51d4b6","Type":"ContainerDied","Data":"4f6ef7e9ba08f5f638610b1bd18e79a362adfa27b696bb4d984169d6d35bdfc6"}
Sep 18 20:14:23.224027 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:23.224006    2581 generic.go:356] "Generic (PLEG): container finished" podID="6fc08935-8e59-4724-813b-094750116547" containerID="e98822727be3cc7b448846333b1fbb9566fc770b9943b3ebf40b5b71fd0737ae" exitCode=0
Sep 18 20:14:23.224114 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:23.224064    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn" event={"ID":"6fc08935-8e59-4724-813b-094750116547","Type":"ContainerDied","Data":"e98822727be3cc7b448846333b1fbb9566fc770b9943b3ebf40b5b71fd0737ae"}
Sep 18 20:14:24.232052 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:24.232012    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm" event={"ID":"bacd4494-7d01-45b1-b8b7-2ca15f51d4b6","Type":"ContainerStarted","Data":"3a18ff2682f280c6709b322b516c754c53e7dc240efecdc41b17fd1471190f6f"}
Sep 18 20:14:24.232516 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:24.232268    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm"
Sep 18 20:14:24.233998 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:24.233975    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn" event={"ID":"6fc08935-8e59-4724-813b-094750116547","Type":"ContainerStarted","Data":"a4e43d7aac9dbdedfd849a60c6f5ea22deede973a94a38caf7256e0305b22257"}
Sep 18 20:14:24.270616 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:24.270568    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm" podStartSLOduration=2.813243609 podStartE2EDuration="5.270550661s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.457307052s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:14:24.268575676 +0000 UTC m=+2187.370864004" watchObservedRunningTime="2026-09-18 20:14:24.270550661 +0000 UTC m=+2187.372839022"
Sep 18 20:14:24.301057 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:24.301012    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn" podStartSLOduration=2.909470364 podStartE2EDuration="5.300997962s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.391527598s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:14:24.299193246 +0000 UTC m=+2187.401481574" watchObservedRunningTime="2026-09-18 20:14:24.300997962 +0000 UTC m=+2187.403286289"
Sep 18 20:14:24.786970 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:24.786931    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"
Sep 18 20:14:29.786983 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:29.786945    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"
Sep 18 20:14:29.799444 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:29.799418    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"
Sep 18 20:14:30.287412 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:30.287386    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"
Sep 18 20:14:55.251022 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:14:55.250981    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm"
Sep 18 20:15:46.048281 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:15:46.048245    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm_bacd4494-7d01-45b1-b8b7-2ca15f51d4b6/main/0.log"
Sep 18 20:16:18.834355 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:18.834323    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"]
Sep 18 20:16:18.849294 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:16:18.849241    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 20:16:18.849294 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:16:18.849294 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 20:16:18.849294 ip-10-0-143-134 kubenswrapper[2581]:  > containerID="a4e43d7aac9dbdedfd849a60c6f5ea22deede973a94a38caf7256e0305b22257" cmd=["/bin/sleep","15"]
Sep 18 20:16:18.849635 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:18.849327    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn" message="" err=<
Sep 18 20:16:18.849635 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:16:18.849635 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 20:16:18.849635 ip-10-0-143-134 kubenswrapper[2581]:  >
Sep 18 20:16:18.849635 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:16:18.849343    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 20:16:18.849635 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:16:18.849635 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 20:16:18.849635 ip-10-0-143-134 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn" podUID="6fc08935-8e59-4724-813b-094750116547" containerName="main" containerID="cri-o://a4e43d7aac9dbdedfd849a60c6f5ea22deede973a94a38caf7256e0305b22257"
Sep 18 20:16:18.849635 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:18.849402    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn" podUID="6fc08935-8e59-4724-813b-094750116547" containerName="main" containerID="cri-o://a4e43d7aac9dbdedfd849a60c6f5ea22deede973a94a38caf7256e0305b22257" gracePeriod=60
Sep 18 20:16:18.853264 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:18.853160    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm"]
Sep 18 20:16:18.853497 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:18.853453    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm" podUID="bacd4494-7d01-45b1-b8b7-2ca15f51d4b6" containerName="main" containerID="cri-o://3a18ff2682f280c6709b322b516c754c53e7dc240efecdc41b17fd1471190f6f" gracePeriod=30
Sep 18 20:16:19.210750 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.210723    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"
Sep 18 20:16:19.295523 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.295493    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fc08935-8e59-4724-813b-094750116547-model-cache\") pod \"6fc08935-8e59-4724-813b-094750116547\" (UID: \"6fc08935-8e59-4724-813b-094750116547\") "
Sep 18 20:16:19.295523 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.295526    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fc08935-8e59-4724-813b-094750116547-home\") pod \"6fc08935-8e59-4724-813b-094750116547\" (UID: \"6fc08935-8e59-4724-813b-094750116547\") "
Sep 18 20:16:19.295704 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.295572    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fc08935-8e59-4724-813b-094750116547-tmp-dir\") pod \"6fc08935-8e59-4724-813b-094750116547\" (UID: \"6fc08935-8e59-4724-813b-094750116547\") "
Sep 18 20:16:19.295704 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.295618    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fc08935-8e59-4724-813b-094750116547-cabundle-cert\") pod \"6fc08935-8e59-4724-813b-094750116547\" (UID: \"6fc08935-8e59-4724-813b-094750116547\") "
Sep 18 20:16:19.295704 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.295644    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fc08935-8e59-4724-813b-094750116547-tls-certs\") pod \"6fc08935-8e59-4724-813b-094750116547\" (UID: \"6fc08935-8e59-4724-813b-094750116547\") "
Sep 18 20:16:19.295704 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.295680    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fc08935-8e59-4724-813b-094750116547-kserve-provision-location\") pod \"6fc08935-8e59-4724-813b-094750116547\" (UID: \"6fc08935-8e59-4724-813b-094750116547\") "
Sep 18 20:16:19.295956 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.295707    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vf8ls\" (UniqueName: \"kubernetes.io/projected/6fc08935-8e59-4724-813b-094750116547-kube-api-access-vf8ls\") pod \"6fc08935-8e59-4724-813b-094750116547\" (UID: \"6fc08935-8e59-4724-813b-094750116547\") "
Sep 18 20:16:19.295956 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.295748    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fc08935-8e59-4724-813b-094750116547-dshm\") pod \"6fc08935-8e59-4724-813b-094750116547\" (UID: \"6fc08935-8e59-4724-813b-094750116547\") "
Sep 18 20:16:19.295956 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.295826    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fc08935-8e59-4724-813b-094750116547-model-cache" (OuterVolumeSpecName: "model-cache") pod "6fc08935-8e59-4724-813b-094750116547" (UID: "6fc08935-8e59-4724-813b-094750116547"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:19.295956 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.295859    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fc08935-8e59-4724-813b-094750116547-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6fc08935-8e59-4724-813b-094750116547" (UID: "6fc08935-8e59-4724-813b-094750116547"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:19.296164 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.295956    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fc08935-8e59-4724-813b-094750116547-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6fc08935-8e59-4724-813b-094750116547" (UID: "6fc08935-8e59-4724-813b-094750116547"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:16:19.296164 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.295960    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fc08935-8e59-4724-813b-094750116547-home" (OuterVolumeSpecName: "home") pod "6fc08935-8e59-4724-813b-094750116547" (UID: "6fc08935-8e59-4724-813b-094750116547"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:19.296164 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.296111    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fc08935-8e59-4724-813b-094750116547-model-cache\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:16:19.296164 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.296128    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fc08935-8e59-4724-813b-094750116547-home\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:16:19.296164 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.296163    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fc08935-8e59-4724-813b-094750116547-tmp-dir\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:16:19.296448 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.296176    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fc08935-8e59-4724-813b-094750116547-cabundle-cert\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:16:19.298245 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.298209    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc08935-8e59-4724-813b-094750116547-kube-api-access-vf8ls" (OuterVolumeSpecName: "kube-api-access-vf8ls") pod "6fc08935-8e59-4724-813b-094750116547" (UID: "6fc08935-8e59-4724-813b-094750116547"). InnerVolumeSpecName "kube-api-access-vf8ls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:16:19.298245 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.298227    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fc08935-8e59-4724-813b-094750116547-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6fc08935-8e59-4724-813b-094750116547" (UID: "6fc08935-8e59-4724-813b-094750116547"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:16:19.298571 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.298553    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fc08935-8e59-4724-813b-094750116547-dshm" (OuterVolumeSpecName: "dshm") pod "6fc08935-8e59-4724-813b-094750116547" (UID: "6fc08935-8e59-4724-813b-094750116547"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:19.318622 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.318578    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fc08935-8e59-4724-813b-094750116547-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6fc08935-8e59-4724-813b-094750116547" (UID: "6fc08935-8e59-4724-813b-094750116547"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:19.397282 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.397197    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fc08935-8e59-4724-813b-094750116547-tls-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:16:19.397282 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.397223    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fc08935-8e59-4724-813b-094750116547-kserve-provision-location\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:16:19.397282 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.397233    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf8ls\" (UniqueName: \"kubernetes.io/projected/6fc08935-8e59-4724-813b-094750116547-kube-api-access-vf8ls\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:16:19.397282 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.397241    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fc08935-8e59-4724-813b-094750116547-dshm\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:16:19.421195 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.421169    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm"
Sep 18 20:16:19.497954 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.497920    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bacd4494-7d01-45b1-b8b7-2ca15f51d4b6-cabundle-cert\") pod \"bacd4494-7d01-45b1-b8b7-2ca15f51d4b6\" (UID: \"bacd4494-7d01-45b1-b8b7-2ca15f51d4b6\") "
Sep 18 20:16:19.498132 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.498008    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bacd4494-7d01-45b1-b8b7-2ca15f51d4b6-tls-certs\") pod \"bacd4494-7d01-45b1-b8b7-2ca15f51d4b6\" (UID: \"bacd4494-7d01-45b1-b8b7-2ca15f51d4b6\") "
Sep 18 20:16:19.498132 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.498113    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bacd4494-7d01-45b1-b8b7-2ca15f51d4b6-tmp-dir\") pod \"bacd4494-7d01-45b1-b8b7-2ca15f51d4b6\" (UID: \"bacd4494-7d01-45b1-b8b7-2ca15f51d4b6\") "
Sep 18 20:16:19.498271 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.498160    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bacd4494-7d01-45b1-b8b7-2ca15f51d4b6-kserve-provision-location\") pod \"bacd4494-7d01-45b1-b8b7-2ca15f51d4b6\" (UID: \"bacd4494-7d01-45b1-b8b7-2ca15f51d4b6\") "
Sep 18 20:16:19.498471 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.498355    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bacd4494-7d01-45b1-b8b7-2ca15f51d4b6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bacd4494-7d01-45b1-b8b7-2ca15f51d4b6" (UID: "bacd4494-7d01-45b1-b8b7-2ca15f51d4b6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:16:19.498471 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.498453    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bacd4494-7d01-45b1-b8b7-2ca15f51d4b6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bacd4494-7d01-45b1-b8b7-2ca15f51d4b6" (UID: "bacd4494-7d01-45b1-b8b7-2ca15f51d4b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:19.500266 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.500247    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bacd4494-7d01-45b1-b8b7-2ca15f51d4b6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bacd4494-7d01-45b1-b8b7-2ca15f51d4b6" (UID: "bacd4494-7d01-45b1-b8b7-2ca15f51d4b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:16:19.521563 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.521532    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bacd4494-7d01-45b1-b8b7-2ca15f51d4b6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bacd4494-7d01-45b1-b8b7-2ca15f51d4b6" (UID: "bacd4494-7d01-45b1-b8b7-2ca15f51d4b6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:16:19.599887 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.599860    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bacd4494-7d01-45b1-b8b7-2ca15f51d4b6-cabundle-cert\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:16:19.599887 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.599882    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bacd4494-7d01-45b1-b8b7-2ca15f51d4b6-tls-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:16:19.599887 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.599892    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bacd4494-7d01-45b1-b8b7-2ca15f51d4b6-tmp-dir\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:16:19.600082 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.599901    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bacd4494-7d01-45b1-b8b7-2ca15f51d4b6-kserve-provision-location\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:16:19.878827 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.878797    2581 generic.go:356] "Generic (PLEG): container finished" podID="bacd4494-7d01-45b1-b8b7-2ca15f51d4b6" containerID="3a18ff2682f280c6709b322b516c754c53e7dc240efecdc41b17fd1471190f6f" exitCode=0
Sep 18 20:16:19.879309 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.878862    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm"
Sep 18 20:16:19.879309 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.878886    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm" event={"ID":"bacd4494-7d01-45b1-b8b7-2ca15f51d4b6","Type":"ContainerDied","Data":"3a18ff2682f280c6709b322b516c754c53e7dc240efecdc41b17fd1471190f6f"}
Sep 18 20:16:19.879309 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.878946    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm" event={"ID":"bacd4494-7d01-45b1-b8b7-2ca15f51d4b6","Type":"ContainerDied","Data":"4e508ce31adf73edcac0df44990747fada7ffe286ed43438311abd93bed33a03"}
Sep 18 20:16:19.879309 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.878980    2581 scope.go:117] "RemoveContainer" containerID="3a18ff2682f280c6709b322b516c754c53e7dc240efecdc41b17fd1471190f6f"
Sep 18 20:16:19.880792 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.880771    2581 generic.go:356] "Generic (PLEG): container finished" podID="6fc08935-8e59-4724-813b-094750116547" containerID="a4e43d7aac9dbdedfd849a60c6f5ea22deede973a94a38caf7256e0305b22257" exitCode=0
Sep 18 20:16:19.880929 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.880826    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"
Sep 18 20:16:19.880929 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.880877    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn" event={"ID":"6fc08935-8e59-4724-813b-094750116547","Type":"ContainerDied","Data":"a4e43d7aac9dbdedfd849a60c6f5ea22deede973a94a38caf7256e0305b22257"}
Sep 18 20:16:19.881015 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.880929    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn" event={"ID":"6fc08935-8e59-4724-813b-094750116547","Type":"ContainerDied","Data":"756023c505148e52b7b33f98a4ae2cd8ab78d0bc94bda695db405dbca207f5bf"}
Sep 18 20:16:19.889522 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.889501    2581 scope.go:117] "RemoveContainer" containerID="4f6ef7e9ba08f5f638610b1bd18e79a362adfa27b696bb4d984169d6d35bdfc6"
Sep 18 20:16:19.897972 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.897914    2581 scope.go:117] "RemoveContainer" containerID="3a18ff2682f280c6709b322b516c754c53e7dc240efecdc41b17fd1471190f6f"
Sep 18 20:16:19.898403 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:16:19.898377    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a18ff2682f280c6709b322b516c754c53e7dc240efecdc41b17fd1471190f6f\": container with ID starting with 3a18ff2682f280c6709b322b516c754c53e7dc240efecdc41b17fd1471190f6f not found: ID does not exist" containerID="3a18ff2682f280c6709b322b516c754c53e7dc240efecdc41b17fd1471190f6f"
Sep 18 20:16:19.898498 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.898411    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a18ff2682f280c6709b322b516c754c53e7dc240efecdc41b17fd1471190f6f"} err="failed to get container status \"3a18ff2682f280c6709b322b516c754c53e7dc240efecdc41b17fd1471190f6f\": rpc error: code = NotFound desc = could not find container \"3a18ff2682f280c6709b322b516c754c53e7dc240efecdc41b17fd1471190f6f\": container with ID starting with 3a18ff2682f280c6709b322b516c754c53e7dc240efecdc41b17fd1471190f6f not found: ID does not exist"
Sep 18 20:16:19.898498 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.898427    2581 scope.go:117] "RemoveContainer" containerID="4f6ef7e9ba08f5f638610b1bd18e79a362adfa27b696bb4d984169d6d35bdfc6"
Sep 18 20:16:19.898857 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:16:19.898653    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f6ef7e9ba08f5f638610b1bd18e79a362adfa27b696bb4d984169d6d35bdfc6\": container with ID starting with 4f6ef7e9ba08f5f638610b1bd18e79a362adfa27b696bb4d984169d6d35bdfc6 not found: ID does not exist" containerID="4f6ef7e9ba08f5f638610b1bd18e79a362adfa27b696bb4d984169d6d35bdfc6"
Sep 18 20:16:19.898857 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.898668    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f6ef7e9ba08f5f638610b1bd18e79a362adfa27b696bb4d984169d6d35bdfc6"} err="failed to get container status \"4f6ef7e9ba08f5f638610b1bd18e79a362adfa27b696bb4d984169d6d35bdfc6\": rpc error: code = NotFound desc = could not find container \"4f6ef7e9ba08f5f638610b1bd18e79a362adfa27b696bb4d984169d6d35bdfc6\": container with ID starting with 4f6ef7e9ba08f5f638610b1bd18e79a362adfa27b696bb4d984169d6d35bdfc6 not found: ID does not exist"
Sep 18 20:16:19.898857 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.898680    2581 scope.go:117] "RemoveContainer" containerID="a4e43d7aac9dbdedfd849a60c6f5ea22deede973a94a38caf7256e0305b22257"
Sep 18 20:16:19.900888 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.900870    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"]
Sep 18 20:16:19.907518 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.907492    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-649b57846b-dq8pn"]
Sep 18 20:16:19.908162 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.908143    2581 scope.go:117] "RemoveContainer" containerID="e98822727be3cc7b448846333b1fbb9566fc770b9943b3ebf40b5b71fd0737ae"
Sep 18 20:16:19.916754 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.916737    2581 scope.go:117] "RemoveContainer" containerID="a4e43d7aac9dbdedfd849a60c6f5ea22deede973a94a38caf7256e0305b22257"
Sep 18 20:16:19.917038 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:16:19.917017    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4e43d7aac9dbdedfd849a60c6f5ea22deede973a94a38caf7256e0305b22257\": container with ID starting with a4e43d7aac9dbdedfd849a60c6f5ea22deede973a94a38caf7256e0305b22257 not found: ID does not exist" containerID="a4e43d7aac9dbdedfd849a60c6f5ea22deede973a94a38caf7256e0305b22257"
Sep 18 20:16:19.917147 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.917041    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4e43d7aac9dbdedfd849a60c6f5ea22deede973a94a38caf7256e0305b22257"} err="failed to get container status \"a4e43d7aac9dbdedfd849a60c6f5ea22deede973a94a38caf7256e0305b22257\": rpc error: code = NotFound desc = could not find container \"a4e43d7aac9dbdedfd849a60c6f5ea22deede973a94a38caf7256e0305b22257\": container with ID starting with a4e43d7aac9dbdedfd849a60c6f5ea22deede973a94a38caf7256e0305b22257 not found: ID does not exist"
Sep 18 20:16:19.917147 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.917056    2581 scope.go:117] "RemoveContainer" containerID="e98822727be3cc7b448846333b1fbb9566fc770b9943b3ebf40b5b71fd0737ae"
Sep 18 20:16:19.917307 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:16:19.917290    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e98822727be3cc7b448846333b1fbb9566fc770b9943b3ebf40b5b71fd0737ae\": container with ID starting with e98822727be3cc7b448846333b1fbb9566fc770b9943b3ebf40b5b71fd0737ae not found: ID does not exist" containerID="e98822727be3cc7b448846333b1fbb9566fc770b9943b3ebf40b5b71fd0737ae"
Sep 18 20:16:19.917348 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.917311    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e98822727be3cc7b448846333b1fbb9566fc770b9943b3ebf40b5b71fd0737ae"} err="failed to get container status \"e98822727be3cc7b448846333b1fbb9566fc770b9943b3ebf40b5b71fd0737ae\": rpc error: code = NotFound desc = could not find container \"e98822727be3cc7b448846333b1fbb9566fc770b9943b3ebf40b5b71fd0737ae\": container with ID starting with e98822727be3cc7b448846333b1fbb9566fc770b9943b3ebf40b5b71fd0737ae not found: ID does not exist"
Sep 18 20:16:19.919047 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.919026    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm"]
Sep 18 20:16:19.925140 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:19.925121    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6c7f889cf6-vm2hm"]
Sep 18 20:16:21.433238 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:21.433206    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fc08935-8e59-4724-813b-094750116547" path="/var/lib/kubelet/pods/6fc08935-8e59-4724-813b-094750116547/volumes"
Sep 18 20:16:21.433623 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:16:21.433610    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bacd4494-7d01-45b1-b8b7-2ca15f51d4b6" path="/var/lib/kubelet/pods/bacd4494-7d01-45b1-b8b7-2ca15f51d4b6/volumes"
Sep 18 20:17:11.518618 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.518586    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c"]
Sep 18 20:17:11.519083 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.518990    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fc08935-8e59-4724-813b-094750116547" containerName="main"
Sep 18 20:17:11.519083 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.519002    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fc08935-8e59-4724-813b-094750116547" containerName="main"
Sep 18 20:17:11.519083 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.519012    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bacd4494-7d01-45b1-b8b7-2ca15f51d4b6" containerName="storage-initializer"
Sep 18 20:17:11.519083 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.519017    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bacd4494-7d01-45b1-b8b7-2ca15f51d4b6" containerName="storage-initializer"
Sep 18 20:17:11.519083 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.519031    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bacd4494-7d01-45b1-b8b7-2ca15f51d4b6" containerName="main"
Sep 18 20:17:11.519083 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.519037    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bacd4494-7d01-45b1-b8b7-2ca15f51d4b6" containerName="main"
Sep 18 20:17:11.519083 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.519046    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fc08935-8e59-4724-813b-094750116547" containerName="storage-initializer"
Sep 18 20:17:11.519083 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.519052    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fc08935-8e59-4724-813b-094750116547" containerName="storage-initializer"
Sep 18 20:17:11.519332 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.519107    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fc08935-8e59-4724-813b-094750116547" containerName="main"
Sep 18 20:17:11.519332 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.519116    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bacd4494-7d01-45b1-b8b7-2ca15f51d4b6" containerName="main"
Sep 18 20:17:11.522022 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.522006    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c"
Sep 18 20:17:11.525583 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.525561    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-lv8nw\""
Sep 18 20:17:11.526015 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.525995    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 18 20:17:11.526643 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.526625    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 18 20:17:11.526752 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.526690    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 18 20:17:11.533452 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.533431    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c"]
Sep 18 20:17:11.649243 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.649201    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfh9d\" (UniqueName: \"kubernetes.io/projected/2c930104-e93b-407b-9f08-f3b479dc0b6d-kube-api-access-zfh9d\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c\" (UID: \"2c930104-e93b-407b-9f08-f3b479dc0b6d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c"
Sep 18 20:17:11.649447 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.649285    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c930104-e93b-407b-9f08-f3b479dc0b6d-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c\" (UID: \"2c930104-e93b-407b-9f08-f3b479dc0b6d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c"
Sep 18 20:17:11.649520 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.649470    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c930104-e93b-407b-9f08-f3b479dc0b6d-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c\" (UID: \"2c930104-e93b-407b-9f08-f3b479dc0b6d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c"
Sep 18 20:17:11.649520 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.649496    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c930104-e93b-407b-9f08-f3b479dc0b6d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c\" (UID: \"2c930104-e93b-407b-9f08-f3b479dc0b6d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c"
Sep 18 20:17:11.649628 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.649554    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c930104-e93b-407b-9f08-f3b479dc0b6d-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c\" (UID: \"2c930104-e93b-407b-9f08-f3b479dc0b6d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c"
Sep 18 20:17:11.649628 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.649602    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c930104-e93b-407b-9f08-f3b479dc0b6d-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c\" (UID: \"2c930104-e93b-407b-9f08-f3b479dc0b6d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c"
Sep 18 20:17:11.750624 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.750591    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfh9d\" (UniqueName: \"kubernetes.io/projected/2c930104-e93b-407b-9f08-f3b479dc0b6d-kube-api-access-zfh9d\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c\" (UID: \"2c930104-e93b-407b-9f08-f3b479dc0b6d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c"
Sep 18 20:17:11.750789 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.750654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c930104-e93b-407b-9f08-f3b479dc0b6d-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c\" (UID: \"2c930104-e93b-407b-9f08-f3b479dc0b6d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c"
Sep 18 20:17:11.750789 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.750682    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c930104-e93b-407b-9f08-f3b479dc0b6d-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c\" (UID: \"2c930104-e93b-407b-9f08-f3b479dc0b6d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c"
Sep 18 20:17:11.750789 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.750709    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c930104-e93b-407b-9f08-f3b479dc0b6d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c\" (UID: \"2c930104-e93b-407b-9f08-f3b479dc0b6d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c"
Sep 18 20:17:11.750995 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.750875    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c930104-e93b-407b-9f08-f3b479dc0b6d-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c\" (UID: \"2c930104-e93b-407b-9f08-f3b479dc0b6d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c"
Sep 18 20:17:11.750995 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.750935    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c930104-e93b-407b-9f08-f3b479dc0b6d-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c\" (UID: \"2c930104-e93b-407b-9f08-f3b479dc0b6d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c"
Sep 18 20:17:11.751231 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.751202    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c930104-e93b-407b-9f08-f3b479dc0b6d-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c\" (UID: \"2c930104-e93b-407b-9f08-f3b479dc0b6d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c"
Sep 18 20:17:11.751392 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.751215    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c930104-e93b-407b-9f08-f3b479dc0b6d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c\" (UID: \"2c930104-e93b-407b-9f08-f3b479dc0b6d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c"
Sep 18 20:17:11.751392 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.751252    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c930104-e93b-407b-9f08-f3b479dc0b6d-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c\" (UID: \"2c930104-e93b-407b-9f08-f3b479dc0b6d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c"
Sep 18 20:17:11.753123 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.753104    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c930104-e93b-407b-9f08-f3b479dc0b6d-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c\" (UID: \"2c930104-e93b-407b-9f08-f3b479dc0b6d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c"
Sep 18 20:17:11.753507 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.753490    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c930104-e93b-407b-9f08-f3b479dc0b6d-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c\" (UID: \"2c930104-e93b-407b-9f08-f3b479dc0b6d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c"
Sep 18 20:17:11.761483 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.761457    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfh9d\" (UniqueName: \"kubernetes.io/projected/2c930104-e93b-407b-9f08-f3b479dc0b6d-kube-api-access-zfh9d\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c\" (UID: \"2c930104-e93b-407b-9f08-f3b479dc0b6d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c"
Sep 18 20:17:11.833230 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:11.833144    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c"
Sep 18 20:17:12.106651 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:12.106629    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c"]
Sep 18 20:17:12.108761 ip-10-0-143-134 kubenswrapper[2581]: W0918 20:17:12.108729    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c930104_e93b_407b_9f08_f3b479dc0b6d.slice/crio-ec67254186814cec8173a56645964cd14e48c26da1c71cfac7c31002073eb1ba WatchSource:0}: Error finding container ec67254186814cec8173a56645964cd14e48c26da1c71cfac7c31002073eb1ba: Status 404 returned error can't find the container with id ec67254186814cec8173a56645964cd14e48c26da1c71cfac7c31002073eb1ba
Sep 18 20:17:12.173433 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:12.173405    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c" event={"ID":"2c930104-e93b-407b-9f08-f3b479dc0b6d","Type":"ContainerStarted","Data":"ec67254186814cec8173a56645964cd14e48c26da1c71cfac7c31002073eb1ba"}
Sep 18 20:17:13.180273 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:13.180237    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c" event={"ID":"2c930104-e93b-407b-9f08-f3b479dc0b6d","Type":"ContainerStarted","Data":"6346a66498071e046821d5a96da00853cc4ddbc8767c0bc1065ef431a347efc6"}
Sep 18 20:17:13.198805 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:13.198760    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c" podStartSLOduration=2.198744336 podStartE2EDuration="2.198744336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:17:13.196544472 +0000 UTC m=+2356.298832800" watchObservedRunningTime="2026-09-18 20:17:13.198744336 +0000 UTC m=+2356.301032694"
Sep 18 20:17:13.833903 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:13.833853    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c"
Sep 18 20:17:21.833668 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:21.833630    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c"
Sep 18 20:17:21.846274 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:21.846238    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c"
Sep 18 20:17:22.238629 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:22.238532    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c"
Sep 18 20:17:46.864912 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:46.864876    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c"]
Sep 18 20:17:46.875515 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:17:46.875468    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 20:17:46.875515 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:17:46.875515 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 20:17:46.875515 ip-10-0-143-134 kubenswrapper[2581]:  > containerID="6346a66498071e046821d5a96da00853cc4ddbc8767c0bc1065ef431a347efc6" cmd=["/bin/sleep","15"]
Sep 18 20:17:46.875957 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:46.875922    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c" message="" err=<
Sep 18 20:17:46.875957 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:17:46.875957 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 20:17:46.875957 ip-10-0-143-134 kubenswrapper[2581]:  >
Sep 18 20:17:46.876232 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:17:46.875963    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 20:17:46.876232 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:17:46.876232 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 20:17:46.876232 ip-10-0-143-134 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c" podUID="2c930104-e93b-407b-9f08-f3b479dc0b6d" containerName="main" containerID="cri-o://6346a66498071e046821d5a96da00853cc4ddbc8767c0bc1065ef431a347efc6"
Sep 18 20:17:46.876232 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:46.876022    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c" podUID="2c930104-e93b-407b-9f08-f3b479dc0b6d" containerName="main" containerID="cri-o://6346a66498071e046821d5a96da00853cc4ddbc8767c0bc1065ef431a347efc6" gracePeriod=60
Sep 18 20:17:47.225444 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:47.225420    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c"
Sep 18 20:17:47.359054 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:47.359021    2581 generic.go:356] "Generic (PLEG): container finished" podID="2c930104-e93b-407b-9f08-f3b479dc0b6d" containerID="6346a66498071e046821d5a96da00853cc4ddbc8767c0bc1065ef431a347efc6" exitCode=0
Sep 18 20:17:47.359054 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:47.359059    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c" event={"ID":"2c930104-e93b-407b-9f08-f3b479dc0b6d","Type":"ContainerDied","Data":"6346a66498071e046821d5a96da00853cc4ddbc8767c0bc1065ef431a347efc6"}
Sep 18 20:17:47.359302 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:47.359071    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c"
Sep 18 20:17:47.359302 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:47.359101    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c" event={"ID":"2c930104-e93b-407b-9f08-f3b479dc0b6d","Type":"ContainerDied","Data":"ec67254186814cec8173a56645964cd14e48c26da1c71cfac7c31002073eb1ba"}
Sep 18 20:17:47.359302 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:47.359124    2581 scope.go:117] "RemoveContainer" containerID="6346a66498071e046821d5a96da00853cc4ddbc8767c0bc1065ef431a347efc6"
Sep 18 20:17:47.368246 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:47.368227    2581 scope.go:117] "RemoveContainer" containerID="6346a66498071e046821d5a96da00853cc4ddbc8767c0bc1065ef431a347efc6"
Sep 18 20:17:47.368498 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:17:47.368479    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6346a66498071e046821d5a96da00853cc4ddbc8767c0bc1065ef431a347efc6\": container with ID starting with 6346a66498071e046821d5a96da00853cc4ddbc8767c0bc1065ef431a347efc6 not found: ID does not exist" containerID="6346a66498071e046821d5a96da00853cc4ddbc8767c0bc1065ef431a347efc6"
Sep 18 20:17:47.368560 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:47.368511    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6346a66498071e046821d5a96da00853cc4ddbc8767c0bc1065ef431a347efc6"} err="failed to get container status \"6346a66498071e046821d5a96da00853cc4ddbc8767c0bc1065ef431a347efc6\": rpc error: code = NotFound desc = could not find container \"6346a66498071e046821d5a96da00853cc4ddbc8767c0bc1065ef431a347efc6\": container with ID starting with 6346a66498071e046821d5a96da00853cc4ddbc8767c0bc1065ef431a347efc6 not found: ID does not exist"
Sep 18 20:17:47.370797 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:47.370779    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c930104-e93b-407b-9f08-f3b479dc0b6d-tls-certs\") pod \"2c930104-e93b-407b-9f08-f3b479dc0b6d\" (UID: \"2c930104-e93b-407b-9f08-f3b479dc0b6d\") "
Sep 18 20:17:47.370902 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:47.370813    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c930104-e93b-407b-9f08-f3b479dc0b6d-home\") pod \"2c930104-e93b-407b-9f08-f3b479dc0b6d\" (UID: \"2c930104-e93b-407b-9f08-f3b479dc0b6d\") "
Sep 18 20:17:47.370902 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:47.370880    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c930104-e93b-407b-9f08-f3b479dc0b6d-model-cache\") pod \"2c930104-e93b-407b-9f08-f3b479dc0b6d\" (UID: \"2c930104-e93b-407b-9f08-f3b479dc0b6d\") "
Sep 18 20:17:47.371016 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:47.370934    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c930104-e93b-407b-9f08-f3b479dc0b6d-dshm\") pod \"2c930104-e93b-407b-9f08-f3b479dc0b6d\" (UID: \"2c930104-e93b-407b-9f08-f3b479dc0b6d\") "
Sep 18 20:17:47.371016 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:47.370972    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfh9d\" (UniqueName: \"kubernetes.io/projected/2c930104-e93b-407b-9f08-f3b479dc0b6d-kube-api-access-zfh9d\") pod \"2c930104-e93b-407b-9f08-f3b479dc0b6d\" (UID: \"2c930104-e93b-407b-9f08-f3b479dc0b6d\") "
Sep 18 20:17:47.371119 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:47.371015    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c930104-e93b-407b-9f08-f3b479dc0b6d-tmp-dir\") pod \"2c930104-e93b-407b-9f08-f3b479dc0b6d\" (UID: \"2c930104-e93b-407b-9f08-f3b479dc0b6d\") "
Sep 18 20:17:47.371169 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:47.371146    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c930104-e93b-407b-9f08-f3b479dc0b6d-model-cache" (OuterVolumeSpecName: "model-cache") pod "2c930104-e93b-407b-9f08-f3b479dc0b6d" (UID: "2c930104-e93b-407b-9f08-f3b479dc0b6d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:47.371223 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:47.371158    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c930104-e93b-407b-9f08-f3b479dc0b6d-home" (OuterVolumeSpecName: "home") pod "2c930104-e93b-407b-9f08-f3b479dc0b6d" (UID: "2c930104-e93b-407b-9f08-f3b479dc0b6d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:47.371332 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:47.371313    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c930104-e93b-407b-9f08-f3b479dc0b6d-home\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:17:47.371445 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:47.371334    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c930104-e93b-407b-9f08-f3b479dc0b6d-model-cache\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:17:47.371445 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:47.371349    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c930104-e93b-407b-9f08-f3b479dc0b6d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2c930104-e93b-407b-9f08-f3b479dc0b6d" (UID: "2c930104-e93b-407b-9f08-f3b479dc0b6d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:47.373155 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:47.373136    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c930104-e93b-407b-9f08-f3b479dc0b6d-dshm" (OuterVolumeSpecName: "dshm") pod "2c930104-e93b-407b-9f08-f3b479dc0b6d" (UID: "2c930104-e93b-407b-9f08-f3b479dc0b6d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:17:47.373322 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:47.373299    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c930104-e93b-407b-9f08-f3b479dc0b6d-kube-api-access-zfh9d" (OuterVolumeSpecName: "kube-api-access-zfh9d") pod "2c930104-e93b-407b-9f08-f3b479dc0b6d" (UID: "2c930104-e93b-407b-9f08-f3b479dc0b6d"). InnerVolumeSpecName "kube-api-access-zfh9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:17:47.373368 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:47.373301    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c930104-e93b-407b-9f08-f3b479dc0b6d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2c930104-e93b-407b-9f08-f3b479dc0b6d" (UID: "2c930104-e93b-407b-9f08-f3b479dc0b6d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:17:47.472416 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:47.472384    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c930104-e93b-407b-9f08-f3b479dc0b6d-tls-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:17:47.472416 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:47.472407    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c930104-e93b-407b-9f08-f3b479dc0b6d-dshm\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:17:47.472416 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:47.472416    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfh9d\" (UniqueName: \"kubernetes.io/projected/2c930104-e93b-407b-9f08-f3b479dc0b6d-kube-api-access-zfh9d\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:17:47.472610 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:47.472426    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c930104-e93b-407b-9f08-f3b479dc0b6d-tmp-dir\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:17:47.678814 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:47.678780    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c"]
Sep 18 20:17:47.683616 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:47.683591    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2fq2c"]
Sep 18 20:17:49.433249 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:17:49.433221    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c930104-e93b-407b-9f08-f3b479dc0b6d" path="/var/lib/kubelet/pods/2c930104-e93b-407b-9f08-f3b479dc0b6d/volumes"
Sep 18 20:18:57.201245 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.201207    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr"]
Sep 18 20:18:57.201728 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.201663    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c930104-e93b-407b-9f08-f3b479dc0b6d" containerName="main"
Sep 18 20:18:57.201728 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.201678    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c930104-e93b-407b-9f08-f3b479dc0b6d" containerName="main"
Sep 18 20:18:57.201805 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.201745    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c930104-e93b-407b-9f08-f3b479dc0b6d" containerName="main"
Sep 18 20:18:57.204749 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.204733    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr"
Sep 18 20:18:57.208441 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.208402    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 18 20:18:57.208441 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.208401    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 18 20:18:57.208629 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.208460    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 18 20:18:57.208629 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.208473    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-f7bgh\""
Sep 18 20:18:57.213962 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.213941    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr"]
Sep 18 20:18:57.370264 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.370233    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7729312d-0aef-4acd-8ed9-db5e16a442d8-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr\" (UID: \"7729312d-0aef-4acd-8ed9-db5e16a442d8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr"
Sep 18 20:18:57.370435 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.370278    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7729312d-0aef-4acd-8ed9-db5e16a442d8-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr\" (UID: \"7729312d-0aef-4acd-8ed9-db5e16a442d8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr"
Sep 18 20:18:57.370435 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.370366    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7729312d-0aef-4acd-8ed9-db5e16a442d8-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr\" (UID: \"7729312d-0aef-4acd-8ed9-db5e16a442d8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr"
Sep 18 20:18:57.370435 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.370420    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7729312d-0aef-4acd-8ed9-db5e16a442d8-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr\" (UID: \"7729312d-0aef-4acd-8ed9-db5e16a442d8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr"
Sep 18 20:18:57.370567 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.370465    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7729312d-0aef-4acd-8ed9-db5e16a442d8-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr\" (UID: \"7729312d-0aef-4acd-8ed9-db5e16a442d8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr"
Sep 18 20:18:57.370567 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.370502    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmmdv\" (UniqueName: \"kubernetes.io/projected/7729312d-0aef-4acd-8ed9-db5e16a442d8-kube-api-access-wmmdv\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr\" (UID: \"7729312d-0aef-4acd-8ed9-db5e16a442d8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr"
Sep 18 20:18:57.471795 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.471726    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7729312d-0aef-4acd-8ed9-db5e16a442d8-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr\" (UID: \"7729312d-0aef-4acd-8ed9-db5e16a442d8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr"
Sep 18 20:18:57.471795 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.471753    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7729312d-0aef-4acd-8ed9-db5e16a442d8-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr\" (UID: \"7729312d-0aef-4acd-8ed9-db5e16a442d8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr"
Sep 18 20:18:57.471795 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.471777    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7729312d-0aef-4acd-8ed9-db5e16a442d8-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr\" (UID: \"7729312d-0aef-4acd-8ed9-db5e16a442d8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr"
Sep 18 20:18:57.472082 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.471815    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7729312d-0aef-4acd-8ed9-db5e16a442d8-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr\" (UID: \"7729312d-0aef-4acd-8ed9-db5e16a442d8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr"
Sep 18 20:18:57.472082 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.471875    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7729312d-0aef-4acd-8ed9-db5e16a442d8-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr\" (UID: \"7729312d-0aef-4acd-8ed9-db5e16a442d8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr"
Sep 18 20:18:57.472082 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.471914    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmmdv\" (UniqueName: \"kubernetes.io/projected/7729312d-0aef-4acd-8ed9-db5e16a442d8-kube-api-access-wmmdv\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr\" (UID: \"7729312d-0aef-4acd-8ed9-db5e16a442d8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr"
Sep 18 20:18:57.472249 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.472214    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7729312d-0aef-4acd-8ed9-db5e16a442d8-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr\" (UID: \"7729312d-0aef-4acd-8ed9-db5e16a442d8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr"
Sep 18 20:18:57.472249 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.472226    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7729312d-0aef-4acd-8ed9-db5e16a442d8-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr\" (UID: \"7729312d-0aef-4acd-8ed9-db5e16a442d8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr"
Sep 18 20:18:57.472324 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.472262    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7729312d-0aef-4acd-8ed9-db5e16a442d8-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr\" (UID: \"7729312d-0aef-4acd-8ed9-db5e16a442d8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr"
Sep 18 20:18:57.473994 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.473971    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7729312d-0aef-4acd-8ed9-db5e16a442d8-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr\" (UID: \"7729312d-0aef-4acd-8ed9-db5e16a442d8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr"
Sep 18 20:18:57.474312 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.474296    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 18 20:18:57.480471 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.480455    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 18 20:18:57.484658 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.484636    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7729312d-0aef-4acd-8ed9-db5e16a442d8-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr\" (UID: \"7729312d-0aef-4acd-8ed9-db5e16a442d8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr"
Sep 18 20:18:57.490792 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.490772    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 18 20:18:57.501116 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.501096    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmmdv\" (UniqueName: \"kubernetes.io/projected/7729312d-0aef-4acd-8ed9-db5e16a442d8-kube-api-access-wmmdv\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr\" (UID: \"7729312d-0aef-4acd-8ed9-db5e16a442d8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr"
Sep 18 20:18:57.518199 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.518182    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-f7bgh\""
Sep 18 20:18:57.526276 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.526259    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr"
Sep 18 20:18:57.798308 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:57.798284    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr"]
Sep 18 20:18:57.800348 ip-10-0-143-134 kubenswrapper[2581]: W0918 20:18:57.800325    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7729312d_0aef_4acd_8ed9_db5e16a442d8.slice/crio-79c55dc0f026fdb5f33bc9aacf6b770806cd4c426669f4eab1c0ca021f9d6553 WatchSource:0}: Error finding container 79c55dc0f026fdb5f33bc9aacf6b770806cd4c426669f4eab1c0ca021f9d6553: Status 404 returned error can't find the container with id 79c55dc0f026fdb5f33bc9aacf6b770806cd4c426669f4eab1c0ca021f9d6553
Sep 18 20:18:58.739198 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:58.739163    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr" event={"ID":"7729312d-0aef-4acd-8ed9-db5e16a442d8","Type":"ContainerStarted","Data":"4ee6e91c0f31ce2b6eaed040586caa42283c98b9a4a470f88f85e00378699e80"}
Sep 18 20:18:58.739590 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:58.739218    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr" event={"ID":"7729312d-0aef-4acd-8ed9-db5e16a442d8","Type":"ContainerStarted","Data":"79c55dc0f026fdb5f33bc9aacf6b770806cd4c426669f4eab1c0ca021f9d6553"}
Sep 18 20:18:58.760074 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:58.760027    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr" podStartSLOduration=1.760011713 podStartE2EDuration="1.760011713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:18:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:18:58.758823712 +0000 UTC m=+2461.861112052" watchObservedRunningTime="2026-09-18 20:18:58.760011713 +0000 UTC m=+2461.862300044"
Sep 18 20:18:59.527140 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:18:59.527106    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr"
Sep 18 20:19:07.526888 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:19:07.526853    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr"
Sep 18 20:19:07.540074 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:19:07.540046    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr"
Sep 18 20:19:07.798309 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:19:07.798237    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr"
Sep 18 20:21:46.003368 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:21:46.003337    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr"]
Sep 18 20:21:46.013008 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:21:46.012975    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 20:21:46.013008 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:21:46.013008 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 20:21:46.013008 ip-10-0-143-134 kubenswrapper[2581]:  > containerID="4ee6e91c0f31ce2b6eaed040586caa42283c98b9a4a470f88f85e00378699e80" cmd=["/bin/sleep","15"]
Sep 18 20:21:46.013308 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:21:46.013032    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr" message="" err=<
Sep 18 20:21:46.013308 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:21:46.013308 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 20:21:46.013308 ip-10-0-143-134 kubenswrapper[2581]:  >
Sep 18 20:21:46.013308 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:21:46.013046    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 20:21:46.013308 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:21:46.013308 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 20:21:46.013308 ip-10-0-143-134 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr" podUID="7729312d-0aef-4acd-8ed9-db5e16a442d8" containerName="main" containerID="cri-o://4ee6e91c0f31ce2b6eaed040586caa42283c98b9a4a470f88f85e00378699e80"
Sep 18 20:21:46.013308 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:21:46.013089    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr" podUID="7729312d-0aef-4acd-8ed9-db5e16a442d8" containerName="main" containerID="cri-o://4ee6e91c0f31ce2b6eaed040586caa42283c98b9a4a470f88f85e00378699e80" gracePeriod=60
Sep 18 20:21:46.356119 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:21:46.356094    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr"
Sep 18 20:21:46.419126 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:21:46.419096    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7729312d-0aef-4acd-8ed9-db5e16a442d8-home\") pod \"7729312d-0aef-4acd-8ed9-db5e16a442d8\" (UID: \"7729312d-0aef-4acd-8ed9-db5e16a442d8\") "
Sep 18 20:21:46.419314 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:21:46.419153    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7729312d-0aef-4acd-8ed9-db5e16a442d8-tls-certs\") pod \"7729312d-0aef-4acd-8ed9-db5e16a442d8\" (UID: \"7729312d-0aef-4acd-8ed9-db5e16a442d8\") "
Sep 18 20:21:46.419314 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:21:46.419188    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmmdv\" (UniqueName: \"kubernetes.io/projected/7729312d-0aef-4acd-8ed9-db5e16a442d8-kube-api-access-wmmdv\") pod \"7729312d-0aef-4acd-8ed9-db5e16a442d8\" (UID: \"7729312d-0aef-4acd-8ed9-db5e16a442d8\") "
Sep 18 20:21:46.419314 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:21:46.419207    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7729312d-0aef-4acd-8ed9-db5e16a442d8-model-cache\") pod \"7729312d-0aef-4acd-8ed9-db5e16a442d8\" (UID: \"7729312d-0aef-4acd-8ed9-db5e16a442d8\") "
Sep 18 20:21:46.419314 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:21:46.419229    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7729312d-0aef-4acd-8ed9-db5e16a442d8-dshm\") pod \"7729312d-0aef-4acd-8ed9-db5e16a442d8\" (UID: \"7729312d-0aef-4acd-8ed9-db5e16a442d8\") "
Sep 18 20:21:46.419314 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:21:46.419253    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7729312d-0aef-4acd-8ed9-db5e16a442d8-tmp-dir\") pod \"7729312d-0aef-4acd-8ed9-db5e16a442d8\" (UID: \"7729312d-0aef-4acd-8ed9-db5e16a442d8\") "
Sep 18 20:21:46.419561 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:21:46.419422    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7729312d-0aef-4acd-8ed9-db5e16a442d8-home" (OuterVolumeSpecName: "home") pod "7729312d-0aef-4acd-8ed9-db5e16a442d8" (UID: "7729312d-0aef-4acd-8ed9-db5e16a442d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:46.419561 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:21:46.419502    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7729312d-0aef-4acd-8ed9-db5e16a442d8-home\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:21:46.419561 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:21:46.419541    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7729312d-0aef-4acd-8ed9-db5e16a442d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "7729312d-0aef-4acd-8ed9-db5e16a442d8" (UID: "7729312d-0aef-4acd-8ed9-db5e16a442d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:46.419717 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:21:46.419587    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7729312d-0aef-4acd-8ed9-db5e16a442d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7729312d-0aef-4acd-8ed9-db5e16a442d8" (UID: "7729312d-0aef-4acd-8ed9-db5e16a442d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:46.421596 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:21:46.421574    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7729312d-0aef-4acd-8ed9-db5e16a442d8-kube-api-access-wmmdv" (OuterVolumeSpecName: "kube-api-access-wmmdv") pod "7729312d-0aef-4acd-8ed9-db5e16a442d8" (UID: "7729312d-0aef-4acd-8ed9-db5e16a442d8"). InnerVolumeSpecName "kube-api-access-wmmdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:21:46.421683 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:21:46.421609    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7729312d-0aef-4acd-8ed9-db5e16a442d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7729312d-0aef-4acd-8ed9-db5e16a442d8" (UID: "7729312d-0aef-4acd-8ed9-db5e16a442d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:21:46.421683 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:21:46.421631    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7729312d-0aef-4acd-8ed9-db5e16a442d8-dshm" (OuterVolumeSpecName: "dshm") pod "7729312d-0aef-4acd-8ed9-db5e16a442d8" (UID: "7729312d-0aef-4acd-8ed9-db5e16a442d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:21:46.520214 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:21:46.520175    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7729312d-0aef-4acd-8ed9-db5e16a442d8-tls-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:21:46.520214 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:21:46.520206    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmmdv\" (UniqueName: \"kubernetes.io/projected/7729312d-0aef-4acd-8ed9-db5e16a442d8-kube-api-access-wmmdv\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:21:46.520214 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:21:46.520221    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7729312d-0aef-4acd-8ed9-db5e16a442d8-model-cache\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:21:46.520534 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:21:46.520233    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7729312d-0aef-4acd-8ed9-db5e16a442d8-dshm\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:21:46.520534 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:21:46.520244    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7729312d-0aef-4acd-8ed9-db5e16a442d8-tmp-dir\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:21:46.637096 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:21:46.637010    2581 generic.go:356] "Generic (PLEG): container finished" podID="7729312d-0aef-4acd-8ed9-db5e16a442d8" containerID="4ee6e91c0f31ce2b6eaed040586caa42283c98b9a4a470f88f85e00378699e80" exitCode=0
Sep 18 20:21:46.637096 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:21:46.637055    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr"
Sep 18 20:21:46.637283 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:21:46.637103    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr" event={"ID":"7729312d-0aef-4acd-8ed9-db5e16a442d8","Type":"ContainerDied","Data":"4ee6e91c0f31ce2b6eaed040586caa42283c98b9a4a470f88f85e00378699e80"}
Sep 18 20:21:46.637283 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:21:46.637152    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr" event={"ID":"7729312d-0aef-4acd-8ed9-db5e16a442d8","Type":"ContainerDied","Data":"79c55dc0f026fdb5f33bc9aacf6b770806cd4c426669f4eab1c0ca021f9d6553"}
Sep 18 20:21:46.637283 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:21:46.637174    2581 scope.go:117] "RemoveContainer" containerID="4ee6e91c0f31ce2b6eaed040586caa42283c98b9a4a470f88f85e00378699e80"
Sep 18 20:21:46.647042 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:21:46.647023    2581 scope.go:117] "RemoveContainer" containerID="4ee6e91c0f31ce2b6eaed040586caa42283c98b9a4a470f88f85e00378699e80"
Sep 18 20:21:46.647290 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:21:46.647275    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ee6e91c0f31ce2b6eaed040586caa42283c98b9a4a470f88f85e00378699e80\": container with ID starting with 4ee6e91c0f31ce2b6eaed040586caa42283c98b9a4a470f88f85e00378699e80 not found: ID does not exist" containerID="4ee6e91c0f31ce2b6eaed040586caa42283c98b9a4a470f88f85e00378699e80"
Sep 18 20:21:46.647359 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:21:46.647297    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ee6e91c0f31ce2b6eaed040586caa42283c98b9a4a470f88f85e00378699e80"} err="failed to get container status \"4ee6e91c0f31ce2b6eaed040586caa42283c98b9a4a470f88f85e00378699e80\": rpc error: code = NotFound desc = could not find container \"4ee6e91c0f31ce2b6eaed040586caa42283c98b9a4a470f88f85e00378699e80\": container with ID starting with 4ee6e91c0f31ce2b6eaed040586caa42283c98b9a4a470f88f85e00378699e80 not found: ID does not exist"
Sep 18 20:21:46.664575 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:21:46.664545    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr"]
Sep 18 20:21:46.666737 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:21:46.666719    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-pqnnr"]
Sep 18 20:21:47.433205 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:21:47.433173    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7729312d-0aef-4acd-8ed9-db5e16a442d8" path="/var/lib/kubelet/pods/7729312d-0aef-4acd-8ed9-db5e16a442d8/volumes"
Sep 18 20:23:02.210224 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:02.210145    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8"]
Sep 18 20:23:02.210731 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:02.210699    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7729312d-0aef-4acd-8ed9-db5e16a442d8" containerName="main"
Sep 18 20:23:02.210731 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:02.210720    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7729312d-0aef-4acd-8ed9-db5e16a442d8" containerName="main"
Sep 18 20:23:02.210873 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:02.210858    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7729312d-0aef-4acd-8ed9-db5e16a442d8" containerName="main"
Sep 18 20:23:02.213989 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:02.213968    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8"
Sep 18 20:23:02.217715 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:02.217656    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-ckdwr\""
Sep 18 20:23:02.217715 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:02.217668    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 18 20:23:02.217715 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:02.217688    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 18 20:23:02.217715 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:02.217706    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 18 20:23:02.225714 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:02.225693    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8"]
Sep 18 20:23:02.343596 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:02.343556    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13a947ce-18a1-45a4-adea-2c0aa90ae911-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8\" (UID: \"13a947ce-18a1-45a4-adea-2c0aa90ae911\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8"
Sep 18 20:23:02.343596 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:02.343596    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v84h\" (UniqueName: \"kubernetes.io/projected/13a947ce-18a1-45a4-adea-2c0aa90ae911-kube-api-access-7v84h\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8\" (UID: \"13a947ce-18a1-45a4-adea-2c0aa90ae911\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8"
Sep 18 20:23:02.343859 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:02.343715    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13a947ce-18a1-45a4-adea-2c0aa90ae911-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8\" (UID: \"13a947ce-18a1-45a4-adea-2c0aa90ae911\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8"
Sep 18 20:23:02.343859 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:02.343779    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13a947ce-18a1-45a4-adea-2c0aa90ae911-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8\" (UID: \"13a947ce-18a1-45a4-adea-2c0aa90ae911\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8"
Sep 18 20:23:02.343997 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:02.343857    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13a947ce-18a1-45a4-adea-2c0aa90ae911-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8\" (UID: \"13a947ce-18a1-45a4-adea-2c0aa90ae911\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8"
Sep 18 20:23:02.343997 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:02.343896    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13a947ce-18a1-45a4-adea-2c0aa90ae911-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8\" (UID: \"13a947ce-18a1-45a4-adea-2c0aa90ae911\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8"
Sep 18 20:23:02.444692 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:02.444660    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13a947ce-18a1-45a4-adea-2c0aa90ae911-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8\" (UID: \"13a947ce-18a1-45a4-adea-2c0aa90ae911\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8"
Sep 18 20:23:02.444891 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:02.444714    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13a947ce-18a1-45a4-adea-2c0aa90ae911-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8\" (UID: \"13a947ce-18a1-45a4-adea-2c0aa90ae911\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8"
Sep 18 20:23:02.444891 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:02.444735    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13a947ce-18a1-45a4-adea-2c0aa90ae911-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8\" (UID: \"13a947ce-18a1-45a4-adea-2c0aa90ae911\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8"
Sep 18 20:23:02.444891 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:02.444751    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13a947ce-18a1-45a4-adea-2c0aa90ae911-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8\" (UID: \"13a947ce-18a1-45a4-adea-2c0aa90ae911\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8"
Sep 18 20:23:02.444891 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:02.444784    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13a947ce-18a1-45a4-adea-2c0aa90ae911-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8\" (UID: \"13a947ce-18a1-45a4-adea-2c0aa90ae911\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8"
Sep 18 20:23:02.444891 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:02.444805    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v84h\" (UniqueName: \"kubernetes.io/projected/13a947ce-18a1-45a4-adea-2c0aa90ae911-kube-api-access-7v84h\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8\" (UID: \"13a947ce-18a1-45a4-adea-2c0aa90ae911\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8"
Sep 18 20:23:02.445195 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:02.445142    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13a947ce-18a1-45a4-adea-2c0aa90ae911-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8\" (UID: \"13a947ce-18a1-45a4-adea-2c0aa90ae911\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8"
Sep 18 20:23:02.445259 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:02.445189    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13a947ce-18a1-45a4-adea-2c0aa90ae911-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8\" (UID: \"13a947ce-18a1-45a4-adea-2c0aa90ae911\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8"
Sep 18 20:23:02.445334 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:02.445311    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13a947ce-18a1-45a4-adea-2c0aa90ae911-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8\" (UID: \"13a947ce-18a1-45a4-adea-2c0aa90ae911\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8"
Sep 18 20:23:02.447221 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:02.447196    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13a947ce-18a1-45a4-adea-2c0aa90ae911-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8\" (UID: \"13a947ce-18a1-45a4-adea-2c0aa90ae911\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8"
Sep 18 20:23:02.447324 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:02.447285    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13a947ce-18a1-45a4-adea-2c0aa90ae911-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8\" (UID: \"13a947ce-18a1-45a4-adea-2c0aa90ae911\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8"
Sep 18 20:23:02.453131 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:02.453105    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v84h\" (UniqueName: \"kubernetes.io/projected/13a947ce-18a1-45a4-adea-2c0aa90ae911-kube-api-access-7v84h\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8\" (UID: \"13a947ce-18a1-45a4-adea-2c0aa90ae911\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8"
Sep 18 20:23:02.524829 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:02.524800    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8"
Sep 18 20:23:02.796701 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:02.796672    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8"]
Sep 18 20:23:02.798612 ip-10-0-143-134 kubenswrapper[2581]: W0918 20:23:02.798583    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a947ce_18a1_45a4_adea_2c0aa90ae911.slice/crio-0432381dbf2d7625e5850dbd9715442af54e5974921a7a925477e20dc03dfe38 WatchSource:0}: Error finding container 0432381dbf2d7625e5850dbd9715442af54e5974921a7a925477e20dc03dfe38: Status 404 returned error can't find the container with id 0432381dbf2d7625e5850dbd9715442af54e5974921a7a925477e20dc03dfe38
Sep 18 20:23:03.035276 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:03.035239    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8" event={"ID":"13a947ce-18a1-45a4-adea-2c0aa90ae911","Type":"ContainerStarted","Data":"1b01755e3dae8192083b16552ef8e6597ea3314e3bd87b62339f270bc1673a8f"}
Sep 18 20:23:03.035441 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:03.035299    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8" event={"ID":"13a947ce-18a1-45a4-adea-2c0aa90ae911","Type":"ContainerStarted","Data":"0432381dbf2d7625e5850dbd9715442af54e5974921a7a925477e20dc03dfe38"}
Sep 18 20:23:03.053979 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:03.053875    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8" podStartSLOduration=1.053858488 podStartE2EDuration="1.053858488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:23:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:23:03.051961009 +0000 UTC m=+2706.154249337" watchObservedRunningTime="2026-09-18 20:23:03.053858488 +0000 UTC m=+2706.156146816"
Sep 18 20:23:03.525601 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:03.525573    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8"
Sep 18 20:23:12.525682 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:12.525647    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8"
Sep 18 20:23:12.537817 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:12.537795    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8"
Sep 18 20:23:13.099916 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:23:13.099884    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8"
Sep 18 20:24:42.407636 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:42.407598    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c"]
Sep 18 20:24:42.410863 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:42.410844    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c"
Sep 18 20:24:42.414614 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:42.414564    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Sep 18 20:24:42.414614 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:42.414593    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 18 20:24:42.414803 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:42.414611    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 18 20:24:42.414803 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:42.414746    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-68tmb\""
Sep 18 20:24:42.420853 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:42.420815    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c"]
Sep 18 20:24:42.501082 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:42.501055    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade73842-ff21-48f1-bde3-29b0441a31b9-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c\" (UID: \"ade73842-ff21-48f1-bde3-29b0441a31b9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c"
Sep 18 20:24:42.501216 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:42.501098    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fj4n\" (UniqueName: \"kubernetes.io/projected/ade73842-ff21-48f1-bde3-29b0441a31b9-kube-api-access-9fj4n\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c\" (UID: \"ade73842-ff21-48f1-bde3-29b0441a31b9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c"
Sep 18 20:24:42.501216 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:42.501127    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ade73842-ff21-48f1-bde3-29b0441a31b9-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c\" (UID: \"ade73842-ff21-48f1-bde3-29b0441a31b9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c"
Sep 18 20:24:42.501288 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:42.501234    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ade73842-ff21-48f1-bde3-29b0441a31b9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c\" (UID: \"ade73842-ff21-48f1-bde3-29b0441a31b9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c"
Sep 18 20:24:42.501288 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:42.501276    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade73842-ff21-48f1-bde3-29b0441a31b9-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c\" (UID: \"ade73842-ff21-48f1-bde3-29b0441a31b9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c"
Sep 18 20:24:42.501356 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:42.501333    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade73842-ff21-48f1-bde3-29b0441a31b9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c\" (UID: \"ade73842-ff21-48f1-bde3-29b0441a31b9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c"
Sep 18 20:24:42.602582 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:42.602553    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade73842-ff21-48f1-bde3-29b0441a31b9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c\" (UID: \"ade73842-ff21-48f1-bde3-29b0441a31b9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c"
Sep 18 20:24:42.602742 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:42.602589    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade73842-ff21-48f1-bde3-29b0441a31b9-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c\" (UID: \"ade73842-ff21-48f1-bde3-29b0441a31b9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c"
Sep 18 20:24:42.602742 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:42.602633    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fj4n\" (UniqueName: \"kubernetes.io/projected/ade73842-ff21-48f1-bde3-29b0441a31b9-kube-api-access-9fj4n\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c\" (UID: \"ade73842-ff21-48f1-bde3-29b0441a31b9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c"
Sep 18 20:24:42.602742 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:42.602659    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ade73842-ff21-48f1-bde3-29b0441a31b9-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c\" (UID: \"ade73842-ff21-48f1-bde3-29b0441a31b9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c"
Sep 18 20:24:42.602742 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:42.602723    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ade73842-ff21-48f1-bde3-29b0441a31b9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c\" (UID: \"ade73842-ff21-48f1-bde3-29b0441a31b9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c"
Sep 18 20:24:42.602992 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:42.602759    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade73842-ff21-48f1-bde3-29b0441a31b9-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c\" (UID: \"ade73842-ff21-48f1-bde3-29b0441a31b9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c"
Sep 18 20:24:42.603075 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:42.603053    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade73842-ff21-48f1-bde3-29b0441a31b9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c\" (UID: \"ade73842-ff21-48f1-bde3-29b0441a31b9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c"
Sep 18 20:24:42.603135 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:42.603092    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade73842-ff21-48f1-bde3-29b0441a31b9-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c\" (UID: \"ade73842-ff21-48f1-bde3-29b0441a31b9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c"
Sep 18 20:24:42.603177 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:42.603145    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ade73842-ff21-48f1-bde3-29b0441a31b9-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c\" (UID: \"ade73842-ff21-48f1-bde3-29b0441a31b9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c"
Sep 18 20:24:42.605061 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:42.605041    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade73842-ff21-48f1-bde3-29b0441a31b9-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c\" (UID: \"ade73842-ff21-48f1-bde3-29b0441a31b9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c"
Sep 18 20:24:42.605328 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:42.605308    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ade73842-ff21-48f1-bde3-29b0441a31b9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c\" (UID: \"ade73842-ff21-48f1-bde3-29b0441a31b9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c"
Sep 18 20:24:42.610665 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:42.610641    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fj4n\" (UniqueName: \"kubernetes.io/projected/ade73842-ff21-48f1-bde3-29b0441a31b9-kube-api-access-9fj4n\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c\" (UID: \"ade73842-ff21-48f1-bde3-29b0441a31b9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c"
Sep 18 20:24:42.722474 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:42.722381    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c"
Sep 18 20:24:42.995925 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:42.995899    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c"]
Sep 18 20:24:42.998584 ip-10-0-143-134 kubenswrapper[2581]: W0918 20:24:42.998555    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade73842_ff21_48f1_bde3_29b0441a31b9.slice/crio-1286cf0435f8a0de9014254f0630ce0b449f6862cd58502b577e961dfb0a3e87 WatchSource:0}: Error finding container 1286cf0435f8a0de9014254f0630ce0b449f6862cd58502b577e961dfb0a3e87: Status 404 returned error can't find the container with id 1286cf0435f8a0de9014254f0630ce0b449f6862cd58502b577e961dfb0a3e87
Sep 18 20:24:43.569642 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:43.569604    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c" event={"ID":"ade73842-ff21-48f1-bde3-29b0441a31b9","Type":"ContainerStarted","Data":"6ff70926df69e7d60fb0b66e56cc8b2d31f391bb78323b4776b89ca7aedc91bd"}
Sep 18 20:24:43.570059 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:43.569662    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c" event={"ID":"ade73842-ff21-48f1-bde3-29b0441a31b9","Type":"ContainerStarted","Data":"1286cf0435f8a0de9014254f0630ce0b449f6862cd58502b577e961dfb0a3e87"}
Sep 18 20:24:43.599128 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:43.599077    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c" podStartSLOduration=1.599060221 podStartE2EDuration="1.599060221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:24:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:24:43.595776534 +0000 UTC m=+2806.698064863" watchObservedRunningTime="2026-09-18 20:24:43.599060221 +0000 UTC m=+2806.701348548"
Sep 18 20:24:43.722900 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:43.722864    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c"
Sep 18 20:24:52.722709 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:52.722623    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c"
Sep 18 20:24:52.735303 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:52.735279    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c"
Sep 18 20:24:53.632261 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:24:53.632225    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c"
Sep 18 20:25:07.512816 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:07.512783    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c"]
Sep 18 20:25:07.522166 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:25:07.522130    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 20:25:07.522166 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:25:07.522166 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 20:25:07.522166 ip-10-0-143-134 kubenswrapper[2581]:  > containerID="6ff70926df69e7d60fb0b66e56cc8b2d31f391bb78323b4776b89ca7aedc91bd" cmd=["/bin/sleep","15"]
Sep 18 20:25:07.522400 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:07.522182    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c" message="" err=<
Sep 18 20:25:07.522400 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:25:07.522400 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 20:25:07.522400 ip-10-0-143-134 kubenswrapper[2581]:  >
Sep 18 20:25:07.522400 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:25:07.522193    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 20:25:07.522400 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:25:07.522400 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 20:25:07.522400 ip-10-0-143-134 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c" podUID="ade73842-ff21-48f1-bde3-29b0441a31b9" containerName="main" containerID="cri-o://6ff70926df69e7d60fb0b66e56cc8b2d31f391bb78323b4776b89ca7aedc91bd"
Sep 18 20:25:07.522400 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:07.522238    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c" podUID="ade73842-ff21-48f1-bde3-29b0441a31b9" containerName="main" containerID="cri-o://6ff70926df69e7d60fb0b66e56cc8b2d31f391bb78323b4776b89ca7aedc91bd" gracePeriod=60
Sep 18 20:25:07.622469 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:07.622428    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c" podUID="ade73842-ff21-48f1-bde3-29b0441a31b9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 18 20:25:07.875108 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:07.875084    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c"
Sep 18 20:25:08.031460 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:08.031425    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade73842-ff21-48f1-bde3-29b0441a31b9-dshm\") pod \"ade73842-ff21-48f1-bde3-29b0441a31b9\" (UID: \"ade73842-ff21-48f1-bde3-29b0441a31b9\") "
Sep 18 20:25:08.031618 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:08.031512    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade73842-ff21-48f1-bde3-29b0441a31b9-tmp-dir\") pod \"ade73842-ff21-48f1-bde3-29b0441a31b9\" (UID: \"ade73842-ff21-48f1-bde3-29b0441a31b9\") "
Sep 18 20:25:08.031618 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:08.031545    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ade73842-ff21-48f1-bde3-29b0441a31b9-tls-certs\") pod \"ade73842-ff21-48f1-bde3-29b0441a31b9\" (UID: \"ade73842-ff21-48f1-bde3-29b0441a31b9\") "
Sep 18 20:25:08.031618 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:08.031597    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ade73842-ff21-48f1-bde3-29b0441a31b9-home\") pod \"ade73842-ff21-48f1-bde3-29b0441a31b9\" (UID: \"ade73842-ff21-48f1-bde3-29b0441a31b9\") "
Sep 18 20:25:08.031789 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:08.031637    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fj4n\" (UniqueName: \"kubernetes.io/projected/ade73842-ff21-48f1-bde3-29b0441a31b9-kube-api-access-9fj4n\") pod \"ade73842-ff21-48f1-bde3-29b0441a31b9\" (UID: \"ade73842-ff21-48f1-bde3-29b0441a31b9\") "
Sep 18 20:25:08.031789 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:08.031718    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade73842-ff21-48f1-bde3-29b0441a31b9-model-cache\") pod \"ade73842-ff21-48f1-bde3-29b0441a31b9\" (UID: \"ade73842-ff21-48f1-bde3-29b0441a31b9\") "
Sep 18 20:25:08.031789 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:08.031737    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade73842-ff21-48f1-bde3-29b0441a31b9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ade73842-ff21-48f1-bde3-29b0441a31b9" (UID: "ade73842-ff21-48f1-bde3-29b0441a31b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:08.031974 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:08.031893    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade73842-ff21-48f1-bde3-29b0441a31b9-home" (OuterVolumeSpecName: "home") pod "ade73842-ff21-48f1-bde3-29b0441a31b9" (UID: "ade73842-ff21-48f1-bde3-29b0441a31b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:08.032030 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:08.032006    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade73842-ff21-48f1-bde3-29b0441a31b9-model-cache" (OuterVolumeSpecName: "model-cache") pod "ade73842-ff21-48f1-bde3-29b0441a31b9" (UID: "ade73842-ff21-48f1-bde3-29b0441a31b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:08.032087 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:08.032071    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade73842-ff21-48f1-bde3-29b0441a31b9-model-cache\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:25:08.032139 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:08.032088    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade73842-ff21-48f1-bde3-29b0441a31b9-tmp-dir\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:25:08.032139 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:08.032100    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ade73842-ff21-48f1-bde3-29b0441a31b9-home\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:25:08.033999 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:08.033979    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ade73842-ff21-48f1-bde3-29b0441a31b9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ade73842-ff21-48f1-bde3-29b0441a31b9" (UID: "ade73842-ff21-48f1-bde3-29b0441a31b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:25:08.034109 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:08.034093    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade73842-ff21-48f1-bde3-29b0441a31b9-kube-api-access-9fj4n" (OuterVolumeSpecName: "kube-api-access-9fj4n") pod "ade73842-ff21-48f1-bde3-29b0441a31b9" (UID: "ade73842-ff21-48f1-bde3-29b0441a31b9"). InnerVolumeSpecName "kube-api-access-9fj4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:25:08.034206 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:08.034185    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade73842-ff21-48f1-bde3-29b0441a31b9-dshm" (OuterVolumeSpecName: "dshm") pod "ade73842-ff21-48f1-bde3-29b0441a31b9" (UID: "ade73842-ff21-48f1-bde3-29b0441a31b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:08.133487 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:08.133426    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade73842-ff21-48f1-bde3-29b0441a31b9-dshm\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:25:08.133487 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:08.133446    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ade73842-ff21-48f1-bde3-29b0441a31b9-tls-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:25:08.133487 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:08.133454    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fj4n\" (UniqueName: \"kubernetes.io/projected/ade73842-ff21-48f1-bde3-29b0441a31b9-kube-api-access-9fj4n\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:25:08.710129 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:08.710089    2581 generic.go:356] "Generic (PLEG): container finished" podID="ade73842-ff21-48f1-bde3-29b0441a31b9" containerID="6ff70926df69e7d60fb0b66e56cc8b2d31f391bb78323b4776b89ca7aedc91bd" exitCode=0
Sep 18 20:25:08.710523 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:08.710141    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c"
Sep 18 20:25:08.710523 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:08.710150    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c" event={"ID":"ade73842-ff21-48f1-bde3-29b0441a31b9","Type":"ContainerDied","Data":"6ff70926df69e7d60fb0b66e56cc8b2d31f391bb78323b4776b89ca7aedc91bd"}
Sep 18 20:25:08.710523 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:08.710204    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c" event={"ID":"ade73842-ff21-48f1-bde3-29b0441a31b9","Type":"ContainerDied","Data":"1286cf0435f8a0de9014254f0630ce0b449f6862cd58502b577e961dfb0a3e87"}
Sep 18 20:25:08.710523 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:08.710233    2581 scope.go:117] "RemoveContainer" containerID="6ff70926df69e7d60fb0b66e56cc8b2d31f391bb78323b4776b89ca7aedc91bd"
Sep 18 20:25:08.720577 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:08.720560    2581 scope.go:117] "RemoveContainer" containerID="6ff70926df69e7d60fb0b66e56cc8b2d31f391bb78323b4776b89ca7aedc91bd"
Sep 18 20:25:08.720822 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:25:08.720806    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff70926df69e7d60fb0b66e56cc8b2d31f391bb78323b4776b89ca7aedc91bd\": container with ID starting with 6ff70926df69e7d60fb0b66e56cc8b2d31f391bb78323b4776b89ca7aedc91bd not found: ID does not exist" containerID="6ff70926df69e7d60fb0b66e56cc8b2d31f391bb78323b4776b89ca7aedc91bd"
Sep 18 20:25:08.720895 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:08.720828    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff70926df69e7d60fb0b66e56cc8b2d31f391bb78323b4776b89ca7aedc91bd"} err="failed to get container status \"6ff70926df69e7d60fb0b66e56cc8b2d31f391bb78323b4776b89ca7aedc91bd\": rpc error: code = NotFound desc = could not find container \"6ff70926df69e7d60fb0b66e56cc8b2d31f391bb78323b4776b89ca7aedc91bd\": container with ID starting with 6ff70926df69e7d60fb0b66e56cc8b2d31f391bb78323b4776b89ca7aedc91bd not found: ID does not exist"
Sep 18 20:25:08.734900 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:08.734869    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c"]
Sep 18 20:25:08.739631 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:08.739608    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ns78c"]
Sep 18 20:25:09.433391 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:09.433357    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ade73842-ff21-48f1-bde3-29b0441a31b9" path="/var/lib/kubelet/pods/ade73842-ff21-48f1-bde3-29b0441a31b9/volumes"
Sep 18 20:25:40.829105 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:40.829067    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8"]
Sep 18 20:25:40.842614 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:25:40.842578    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 20:25:40.842614 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:25:40.842614 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 20:25:40.842614 ip-10-0-143-134 kubenswrapper[2581]:  > containerID="1b01755e3dae8192083b16552ef8e6597ea3314e3bd87b62339f270bc1673a8f" cmd=["/bin/sleep","15"]
Sep 18 20:25:40.842886 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:40.842627    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8" message="" err=<
Sep 18 20:25:40.842886 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:25:40.842886 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 20:25:40.842886 ip-10-0-143-134 kubenswrapper[2581]:  >
Sep 18 20:25:40.842886 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:25:40.842638    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 20:25:40.842886 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:25:40.842886 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 20:25:40.842886 ip-10-0-143-134 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8" podUID="13a947ce-18a1-45a4-adea-2c0aa90ae911" containerName="main" containerID="cri-o://1b01755e3dae8192083b16552ef8e6597ea3314e3bd87b62339f270bc1673a8f"
Sep 18 20:25:40.842886 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:40.842680    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8" podUID="13a947ce-18a1-45a4-adea-2c0aa90ae911" containerName="main" containerID="cri-o://1b01755e3dae8192083b16552ef8e6597ea3314e3bd87b62339f270bc1673a8f" gracePeriod=60
Sep 18 20:25:41.089037 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:41.088942    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8" podUID="13a947ce-18a1-45a4-adea-2c0aa90ae911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 18 20:25:41.189087 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:41.189062    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8"
Sep 18 20:25:41.227596 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:41.227562    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13a947ce-18a1-45a4-adea-2c0aa90ae911-home\") pod \"13a947ce-18a1-45a4-adea-2c0aa90ae911\" (UID: \"13a947ce-18a1-45a4-adea-2c0aa90ae911\") "
Sep 18 20:25:41.227756 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:41.227605    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13a947ce-18a1-45a4-adea-2c0aa90ae911-tmp-dir\") pod \"13a947ce-18a1-45a4-adea-2c0aa90ae911\" (UID: \"13a947ce-18a1-45a4-adea-2c0aa90ae911\") "
Sep 18 20:25:41.227756 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:41.227634    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13a947ce-18a1-45a4-adea-2c0aa90ae911-dshm\") pod \"13a947ce-18a1-45a4-adea-2c0aa90ae911\" (UID: \"13a947ce-18a1-45a4-adea-2c0aa90ae911\") "
Sep 18 20:25:41.227756 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:41.227719    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13a947ce-18a1-45a4-adea-2c0aa90ae911-tls-certs\") pod \"13a947ce-18a1-45a4-adea-2c0aa90ae911\" (UID: \"13a947ce-18a1-45a4-adea-2c0aa90ae911\") "
Sep 18 20:25:41.227934 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:41.227792    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7v84h\" (UniqueName: \"kubernetes.io/projected/13a947ce-18a1-45a4-adea-2c0aa90ae911-kube-api-access-7v84h\") pod \"13a947ce-18a1-45a4-adea-2c0aa90ae911\" (UID: \"13a947ce-18a1-45a4-adea-2c0aa90ae911\") "
Sep 18 20:25:41.227934 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:41.227824    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13a947ce-18a1-45a4-adea-2c0aa90ae911-model-cache\") pod \"13a947ce-18a1-45a4-adea-2c0aa90ae911\" (UID: \"13a947ce-18a1-45a4-adea-2c0aa90ae911\") "
Sep 18 20:25:41.227934 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:41.227890    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13a947ce-18a1-45a4-adea-2c0aa90ae911-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "13a947ce-18a1-45a4-adea-2c0aa90ae911" (UID: "13a947ce-18a1-45a4-adea-2c0aa90ae911"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:41.228088 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:41.227930    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13a947ce-18a1-45a4-adea-2c0aa90ae911-home" (OuterVolumeSpecName: "home") pod "13a947ce-18a1-45a4-adea-2c0aa90ae911" (UID: "13a947ce-18a1-45a4-adea-2c0aa90ae911"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:41.228144 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:41.228115    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13a947ce-18a1-45a4-adea-2c0aa90ae911-model-cache" (OuterVolumeSpecName: "model-cache") pod "13a947ce-18a1-45a4-adea-2c0aa90ae911" (UID: "13a947ce-18a1-45a4-adea-2c0aa90ae911"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:41.228205 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:41.228144    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13a947ce-18a1-45a4-adea-2c0aa90ae911-home\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:25:41.228205 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:41.228185    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13a947ce-18a1-45a4-adea-2c0aa90ae911-tmp-dir\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:25:41.230169 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:41.230135    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13a947ce-18a1-45a4-adea-2c0aa90ae911-dshm" (OuterVolumeSpecName: "dshm") pod "13a947ce-18a1-45a4-adea-2c0aa90ae911" (UID: "13a947ce-18a1-45a4-adea-2c0aa90ae911"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:25:41.230286 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:41.230178    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13a947ce-18a1-45a4-adea-2c0aa90ae911-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "13a947ce-18a1-45a4-adea-2c0aa90ae911" (UID: "13a947ce-18a1-45a4-adea-2c0aa90ae911"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:25:41.230286 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:41.230209    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13a947ce-18a1-45a4-adea-2c0aa90ae911-kube-api-access-7v84h" (OuterVolumeSpecName: "kube-api-access-7v84h") pod "13a947ce-18a1-45a4-adea-2c0aa90ae911" (UID: "13a947ce-18a1-45a4-adea-2c0aa90ae911"). InnerVolumeSpecName "kube-api-access-7v84h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:25:41.328760 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:41.328720    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13a947ce-18a1-45a4-adea-2c0aa90ae911-dshm\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:25:41.328760 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:41.328754    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13a947ce-18a1-45a4-adea-2c0aa90ae911-tls-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:25:41.328760 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:41.328768    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v84h\" (UniqueName: \"kubernetes.io/projected/13a947ce-18a1-45a4-adea-2c0aa90ae911-kube-api-access-7v84h\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:25:41.329040 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:41.328781    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13a947ce-18a1-45a4-adea-2c0aa90ae911-model-cache\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:25:41.891967 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:41.891929    2581 generic.go:356] "Generic (PLEG): container finished" podID="13a947ce-18a1-45a4-adea-2c0aa90ae911" containerID="1b01755e3dae8192083b16552ef8e6597ea3314e3bd87b62339f270bc1673a8f" exitCode=0
Sep 18 20:25:41.891967 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:41.891970    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8"
Sep 18 20:25:41.892458 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:41.892012    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8" event={"ID":"13a947ce-18a1-45a4-adea-2c0aa90ae911","Type":"ContainerDied","Data":"1b01755e3dae8192083b16552ef8e6597ea3314e3bd87b62339f270bc1673a8f"}
Sep 18 20:25:41.892458 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:41.892057    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8" event={"ID":"13a947ce-18a1-45a4-adea-2c0aa90ae911","Type":"ContainerDied","Data":"0432381dbf2d7625e5850dbd9715442af54e5974921a7a925477e20dc03dfe38"}
Sep 18 20:25:41.892458 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:41.892081    2581 scope.go:117] "RemoveContainer" containerID="1b01755e3dae8192083b16552ef8e6597ea3314e3bd87b62339f270bc1673a8f"
Sep 18 20:25:41.901393 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:41.901373    2581 scope.go:117] "RemoveContainer" containerID="1b01755e3dae8192083b16552ef8e6597ea3314e3bd87b62339f270bc1673a8f"
Sep 18 20:25:41.901678 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:25:41.901654    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b01755e3dae8192083b16552ef8e6597ea3314e3bd87b62339f270bc1673a8f\": container with ID starting with 1b01755e3dae8192083b16552ef8e6597ea3314e3bd87b62339f270bc1673a8f not found: ID does not exist" containerID="1b01755e3dae8192083b16552ef8e6597ea3314e3bd87b62339f270bc1673a8f"
Sep 18 20:25:41.901755 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:41.901684    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b01755e3dae8192083b16552ef8e6597ea3314e3bd87b62339f270bc1673a8f"} err="failed to get container status \"1b01755e3dae8192083b16552ef8e6597ea3314e3bd87b62339f270bc1673a8f\": rpc error: code = NotFound desc = could not find container \"1b01755e3dae8192083b16552ef8e6597ea3314e3bd87b62339f270bc1673a8f\": container with ID starting with 1b01755e3dae8192083b16552ef8e6597ea3314e3bd87b62339f270bc1673a8f not found: ID does not exist"
Sep 18 20:25:41.909781 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:41.909755    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8"]
Sep 18 20:25:41.913749 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:41.913730    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969wfzs8"]
Sep 18 20:25:43.433320 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:25:43.433289    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13a947ce-18a1-45a4-adea-2c0aa90ae911" path="/var/lib/kubelet/pods/13a947ce-18a1-45a4-adea-2c0aa90ae911/volumes"
Sep 18 20:26:36.604382 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.604348    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"]
Sep 18 20:26:36.604925 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.604890    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13a947ce-18a1-45a4-adea-2c0aa90ae911" containerName="main"
Sep 18 20:26:36.604925 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.604909    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="13a947ce-18a1-45a4-adea-2c0aa90ae911" containerName="main"
Sep 18 20:26:36.605050 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.604955    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ade73842-ff21-48f1-bde3-29b0441a31b9" containerName="main"
Sep 18 20:26:36.605050 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.604964    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ade73842-ff21-48f1-bde3-29b0441a31b9" containerName="main"
Sep 18 20:26:36.605164 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.605061    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="13a947ce-18a1-45a4-adea-2c0aa90ae911" containerName="main"
Sep 18 20:26:36.605164 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.605073    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ade73842-ff21-48f1-bde3-29b0441a31b9" containerName="main"
Sep 18 20:26:36.607851 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.607819    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"
Sep 18 20:26:36.610411 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.610386    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 18 20:26:36.610565 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.610440    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 18 20:26:36.610660 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.610592    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-sjpw7\""
Sep 18 20:26:36.610817 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.610801    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 18 20:26:36.610941 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.610920    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Sep 18 20:26:36.611294 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.611278    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:26:36.618931 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.618910    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"]
Sep 18 20:26:36.699831 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.699794    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/895dbc73-3b91-4abd-a377-44381a021115-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7\" (UID: \"895dbc73-3b91-4abd-a377-44381a021115\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"
Sep 18 20:26:36.699997 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.699855    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brdnt\" (UniqueName: \"kubernetes.io/projected/895dbc73-3b91-4abd-a377-44381a021115-kube-api-access-brdnt\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7\" (UID: \"895dbc73-3b91-4abd-a377-44381a021115\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"
Sep 18 20:26:36.699997 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.699932    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/895dbc73-3b91-4abd-a377-44381a021115-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7\" (UID: \"895dbc73-3b91-4abd-a377-44381a021115\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"
Sep 18 20:26:36.699997 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.699988    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/895dbc73-3b91-4abd-a377-44381a021115-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7\" (UID: \"895dbc73-3b91-4abd-a377-44381a021115\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"
Sep 18 20:26:36.700130 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.700020    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/895dbc73-3b91-4abd-a377-44381a021115-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7\" (UID: \"895dbc73-3b91-4abd-a377-44381a021115\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"
Sep 18 20:26:36.700130 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.700038    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/895dbc73-3b91-4abd-a377-44381a021115-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7\" (UID: \"895dbc73-3b91-4abd-a377-44381a021115\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"
Sep 18 20:26:36.700130 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.700052    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/895dbc73-3b91-4abd-a377-44381a021115-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7\" (UID: \"895dbc73-3b91-4abd-a377-44381a021115\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"
Sep 18 20:26:36.700130 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.700111    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/895dbc73-3b91-4abd-a377-44381a021115-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7\" (UID: \"895dbc73-3b91-4abd-a377-44381a021115\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"
Sep 18 20:26:36.800475 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.800447    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/895dbc73-3b91-4abd-a377-44381a021115-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7\" (UID: \"895dbc73-3b91-4abd-a377-44381a021115\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"
Sep 18 20:26:36.800594 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.800494    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/895dbc73-3b91-4abd-a377-44381a021115-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7\" (UID: \"895dbc73-3b91-4abd-a377-44381a021115\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"
Sep 18 20:26:36.800594 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.800527    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/895dbc73-3b91-4abd-a377-44381a021115-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7\" (UID: \"895dbc73-3b91-4abd-a377-44381a021115\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"
Sep 18 20:26:36.800594 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.800551    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/895dbc73-3b91-4abd-a377-44381a021115-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7\" (UID: \"895dbc73-3b91-4abd-a377-44381a021115\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"
Sep 18 20:26:36.800594 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.800574    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/895dbc73-3b91-4abd-a377-44381a021115-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7\" (UID: \"895dbc73-3b91-4abd-a377-44381a021115\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"
Sep 18 20:26:36.800758 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.800606    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/895dbc73-3b91-4abd-a377-44381a021115-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7\" (UID: \"895dbc73-3b91-4abd-a377-44381a021115\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"
Sep 18 20:26:36.800758 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.800656    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/895dbc73-3b91-4abd-a377-44381a021115-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7\" (UID: \"895dbc73-3b91-4abd-a377-44381a021115\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"
Sep 18 20:26:36.800758 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.800682    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brdnt\" (UniqueName: \"kubernetes.io/projected/895dbc73-3b91-4abd-a377-44381a021115-kube-api-access-brdnt\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7\" (UID: \"895dbc73-3b91-4abd-a377-44381a021115\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"
Sep 18 20:26:36.800959 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.800933    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/895dbc73-3b91-4abd-a377-44381a021115-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7\" (UID: \"895dbc73-3b91-4abd-a377-44381a021115\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"
Sep 18 20:26:36.801105 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.800971    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/895dbc73-3b91-4abd-a377-44381a021115-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7\" (UID: \"895dbc73-3b91-4abd-a377-44381a021115\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"
Sep 18 20:26:36.801105 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.800988    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/895dbc73-3b91-4abd-a377-44381a021115-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7\" (UID: \"895dbc73-3b91-4abd-a377-44381a021115\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"
Sep 18 20:26:36.801250 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.801224    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/895dbc73-3b91-4abd-a377-44381a021115-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7\" (UID: \"895dbc73-3b91-4abd-a377-44381a021115\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"
Sep 18 20:26:36.801388 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.801370    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/895dbc73-3b91-4abd-a377-44381a021115-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7\" (UID: \"895dbc73-3b91-4abd-a377-44381a021115\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"
Sep 18 20:26:36.803145 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.803127    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/895dbc73-3b91-4abd-a377-44381a021115-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7\" (UID: \"895dbc73-3b91-4abd-a377-44381a021115\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"
Sep 18 20:26:36.803445 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.803423    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/895dbc73-3b91-4abd-a377-44381a021115-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7\" (UID: \"895dbc73-3b91-4abd-a377-44381a021115\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"
Sep 18 20:26:36.812314 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.812291    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brdnt\" (UniqueName: \"kubernetes.io/projected/895dbc73-3b91-4abd-a377-44381a021115-kube-api-access-brdnt\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7\" (UID: \"895dbc73-3b91-4abd-a377-44381a021115\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"
Sep 18 20:26:36.919870 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:36.919773    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"
Sep 18 20:26:37.197912 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:37.197888    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"]
Sep 18 20:26:37.199148 ip-10-0-143-134 kubenswrapper[2581]: W0918 20:26:37.199120    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod895dbc73_3b91_4abd_a377_44381a021115.slice/crio-da2b5cbd3b76d8418ac2b4d5bbe63a18b15a18df5077c7436ba6d7dadbc03586 WatchSource:0}: Error finding container da2b5cbd3b76d8418ac2b4d5bbe63a18b15a18df5077c7436ba6d7dadbc03586: Status 404 returned error can't find the container with id da2b5cbd3b76d8418ac2b4d5bbe63a18b15a18df5077c7436ba6d7dadbc03586
Sep 18 20:26:38.183359 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:38.183323    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7" event={"ID":"895dbc73-3b91-4abd-a377-44381a021115","Type":"ContainerStarted","Data":"b2108617f71f5c9b010833798991ac5a3de3ece2320c18c02f34de34b25438b6"}
Sep 18 20:26:38.183752 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:38.183375    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7" event={"ID":"895dbc73-3b91-4abd-a377-44381a021115","Type":"ContainerStarted","Data":"da2b5cbd3b76d8418ac2b4d5bbe63a18b15a18df5077c7436ba6d7dadbc03586"}
Sep 18 20:26:40.195039 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:40.194945    2581 generic.go:356] "Generic (PLEG): container finished" podID="895dbc73-3b91-4abd-a377-44381a021115" containerID="b2108617f71f5c9b010833798991ac5a3de3ece2320c18c02f34de34b25438b6" exitCode=0
Sep 18 20:26:40.195039 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:40.195021    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7" event={"ID":"895dbc73-3b91-4abd-a377-44381a021115","Type":"ContainerDied","Data":"b2108617f71f5c9b010833798991ac5a3de3ece2320c18c02f34de34b25438b6"}
Sep 18 20:26:41.202364 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:41.202328    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7" event={"ID":"895dbc73-3b91-4abd-a377-44381a021115","Type":"ContainerStarted","Data":"d97e20e9958d7a67e1b6364e65542c105716aa6104b133708060df2510c8a82e"}
Sep 18 20:26:41.229076 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:41.229026    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7" podStartSLOduration=2.618569088 podStartE2EDuration="5.229010884s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.610441796s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:26:41.226975718 +0000 UTC m=+2924.329264047" watchObservedRunningTime="2026-09-18 20:26:41.229010884 +0000 UTC m=+2924.331299254"
Sep 18 20:26:41.920056 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:41.920026    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"
Sep 18 20:26:46.920017 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:46.919977    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"
Sep 18 20:26:46.922354 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:46.921710    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7" podUID="895dbc73-3b91-4abd-a377-44381a021115" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Sep 18 20:26:56.920429 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:26:56.920382    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7" podUID="895dbc73-3b91-4abd-a377-44381a021115" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Sep 18 20:27:06.920407 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:27:06.920367    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7" podUID="895dbc73-3b91-4abd-a377-44381a021115" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Sep 18 20:27:16.920405 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:27:16.920345    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7" podUID="895dbc73-3b91-4abd-a377-44381a021115" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Sep 18 20:27:26.920450 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:27:26.920352    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7" podUID="895dbc73-3b91-4abd-a377-44381a021115" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Sep 18 20:27:36.920772 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:27:36.920730    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7" podUID="895dbc73-3b91-4abd-a377-44381a021115" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Sep 18 20:27:46.920735 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:27:46.920684    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7" podUID="895dbc73-3b91-4abd-a377-44381a021115" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Sep 18 20:27:56.930456 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:27:56.930424    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"
Sep 18 20:27:56.938332 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:27:56.938303    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"
Sep 18 20:28:51.439187 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:28:51.439091    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"]
Sep 18 20:29:06.461969 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:06.461901    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7" podUID="895dbc73-3b91-4abd-a377-44381a021115" containerName="main" containerID="cri-o://d97e20e9958d7a67e1b6364e65542c105716aa6104b133708060df2510c8a82e" gracePeriod=45
Sep 18 20:29:06.931878 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:06.931821    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7" podUID="895dbc73-3b91-4abd-a377-44381a021115" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Sep 18 20:29:07.006943 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:07.006908    2581 generic.go:356] "Generic (PLEG): container finished" podID="895dbc73-3b91-4abd-a377-44381a021115" containerID="d97e20e9958d7a67e1b6364e65542c105716aa6104b133708060df2510c8a82e" exitCode=0
Sep 18 20:29:07.007091 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:07.006978    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7" event={"ID":"895dbc73-3b91-4abd-a377-44381a021115","Type":"ContainerDied","Data":"d97e20e9958d7a67e1b6364e65542c105716aa6104b133708060df2510c8a82e"}
Sep 18 20:29:07.108010 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:07.107990    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"
Sep 18 20:29:07.213056 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:07.212997    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/895dbc73-3b91-4abd-a377-44381a021115-tls-certs\") pod \"895dbc73-3b91-4abd-a377-44381a021115\" (UID: \"895dbc73-3b91-4abd-a377-44381a021115\") "
Sep 18 20:29:07.213056 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:07.213030    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/895dbc73-3b91-4abd-a377-44381a021115-kserve-provision-location\") pod \"895dbc73-3b91-4abd-a377-44381a021115\" (UID: \"895dbc73-3b91-4abd-a377-44381a021115\") "
Sep 18 20:29:07.213056 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:07.213047    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/895dbc73-3b91-4abd-a377-44381a021115-cabundle-cert\") pod \"895dbc73-3b91-4abd-a377-44381a021115\" (UID: \"895dbc73-3b91-4abd-a377-44381a021115\") "
Sep 18 20:29:07.213302 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:07.213082    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/895dbc73-3b91-4abd-a377-44381a021115-model-cache\") pod \"895dbc73-3b91-4abd-a377-44381a021115\" (UID: \"895dbc73-3b91-4abd-a377-44381a021115\") "
Sep 18 20:29:07.213302 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:07.213106    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brdnt\" (UniqueName: \"kubernetes.io/projected/895dbc73-3b91-4abd-a377-44381a021115-kube-api-access-brdnt\") pod \"895dbc73-3b91-4abd-a377-44381a021115\" (UID: \"895dbc73-3b91-4abd-a377-44381a021115\") "
Sep 18 20:29:07.213302 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:07.213133    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/895dbc73-3b91-4abd-a377-44381a021115-dshm\") pod \"895dbc73-3b91-4abd-a377-44381a021115\" (UID: \"895dbc73-3b91-4abd-a377-44381a021115\") "
Sep 18 20:29:07.213302 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:07.213159    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/895dbc73-3b91-4abd-a377-44381a021115-home\") pod \"895dbc73-3b91-4abd-a377-44381a021115\" (UID: \"895dbc73-3b91-4abd-a377-44381a021115\") "
Sep 18 20:29:07.213302 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:07.213185    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/895dbc73-3b91-4abd-a377-44381a021115-tmp-dir\") pod \"895dbc73-3b91-4abd-a377-44381a021115\" (UID: \"895dbc73-3b91-4abd-a377-44381a021115\") "
Sep 18 20:29:07.213559 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:07.213320    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895dbc73-3b91-4abd-a377-44381a021115-model-cache" (OuterVolumeSpecName: "model-cache") pod "895dbc73-3b91-4abd-a377-44381a021115" (UID: "895dbc73-3b91-4abd-a377-44381a021115"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:29:07.213559 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:07.213428    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/895dbc73-3b91-4abd-a377-44381a021115-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "895dbc73-3b91-4abd-a377-44381a021115" (UID: "895dbc73-3b91-4abd-a377-44381a021115"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:29:07.213670 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:07.213622    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/895dbc73-3b91-4abd-a377-44381a021115-cabundle-cert\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:29:07.213670 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:07.213639    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/895dbc73-3b91-4abd-a377-44381a021115-model-cache\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:29:07.213983 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:07.213954    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895dbc73-3b91-4abd-a377-44381a021115-home" (OuterVolumeSpecName: "home") pod "895dbc73-3b91-4abd-a377-44381a021115" (UID: "895dbc73-3b91-4abd-a377-44381a021115"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:29:07.215529 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:07.215495    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/895dbc73-3b91-4abd-a377-44381a021115-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "895dbc73-3b91-4abd-a377-44381a021115" (UID: "895dbc73-3b91-4abd-a377-44381a021115"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:29:07.215622 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:07.215567    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/895dbc73-3b91-4abd-a377-44381a021115-kube-api-access-brdnt" (OuterVolumeSpecName: "kube-api-access-brdnt") pod "895dbc73-3b91-4abd-a377-44381a021115" (UID: "895dbc73-3b91-4abd-a377-44381a021115"). InnerVolumeSpecName "kube-api-access-brdnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:29:07.215810 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:07.215790    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895dbc73-3b91-4abd-a377-44381a021115-dshm" (OuterVolumeSpecName: "dshm") pod "895dbc73-3b91-4abd-a377-44381a021115" (UID: "895dbc73-3b91-4abd-a377-44381a021115"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:29:07.225909 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:07.225887    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895dbc73-3b91-4abd-a377-44381a021115-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "895dbc73-3b91-4abd-a377-44381a021115" (UID: "895dbc73-3b91-4abd-a377-44381a021115"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:29:07.239089 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:07.239068    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895dbc73-3b91-4abd-a377-44381a021115-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "895dbc73-3b91-4abd-a377-44381a021115" (UID: "895dbc73-3b91-4abd-a377-44381a021115"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:29:07.314543 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:07.314517    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/895dbc73-3b91-4abd-a377-44381a021115-tls-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:29:07.314543 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:07.314541    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/895dbc73-3b91-4abd-a377-44381a021115-kserve-provision-location\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:29:07.314694 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:07.314552    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brdnt\" (UniqueName: \"kubernetes.io/projected/895dbc73-3b91-4abd-a377-44381a021115-kube-api-access-brdnt\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:29:07.314694 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:07.314560    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/895dbc73-3b91-4abd-a377-44381a021115-dshm\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:29:07.314694 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:07.314569    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/895dbc73-3b91-4abd-a377-44381a021115-home\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:29:07.314694 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:07.314576    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/895dbc73-3b91-4abd-a377-44381a021115-tmp-dir\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:29:08.013732 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:08.013695    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7" event={"ID":"895dbc73-3b91-4abd-a377-44381a021115","Type":"ContainerDied","Data":"da2b5cbd3b76d8418ac2b4d5bbe63a18b15a18df5077c7436ba6d7dadbc03586"}
Sep 18 20:29:08.014179 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:08.013759    2581 scope.go:117] "RemoveContainer" containerID="d97e20e9958d7a67e1b6364e65542c105716aa6104b133708060df2510c8a82e"
Sep 18 20:29:08.014179 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:08.013711    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"
Sep 18 20:29:08.023099 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:08.023075    2581 scope.go:117] "RemoveContainer" containerID="b2108617f71f5c9b010833798991ac5a3de3ece2320c18c02f34de34b25438b6"
Sep 18 20:29:08.032942 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:08.032919    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"]
Sep 18 20:29:08.036922 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:08.036901    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c846579b8-2pwh7"]
Sep 18 20:29:09.433476 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:09.433442    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="895dbc73-3b91-4abd-a377-44381a021115" path="/var/lib/kubelet/pods/895dbc73-3b91-4abd-a377-44381a021115/volumes"
Sep 18 20:29:46.524799 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.524761    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"]
Sep 18 20:29:46.525291 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.525149    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="895dbc73-3b91-4abd-a377-44381a021115" containerName="storage-initializer"
Sep 18 20:29:46.525291 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.525162    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="895dbc73-3b91-4abd-a377-44381a021115" containerName="storage-initializer"
Sep 18 20:29:46.525291 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.525178    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="895dbc73-3b91-4abd-a377-44381a021115" containerName="main"
Sep 18 20:29:46.525291 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.525183    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="895dbc73-3b91-4abd-a377-44381a021115" containerName="main"
Sep 18 20:29:46.525291 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.525263    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="895dbc73-3b91-4abd-a377-44381a021115" containerName="main"
Sep 18 20:29:46.528431 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.528416    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"
Sep 18 20:29:46.535611 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.535587    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-9pdbw\""
Sep 18 20:29:46.535782 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.535709    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 18 20:29:46.535782 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.535755    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 18 20:29:46.535782 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.535616    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 18 20:29:46.535782 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.535716    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:29:46.536057 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.535772    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 18 20:29:46.539458 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.539424    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"]
Sep 18 20:29:46.554133 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.554109    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2\" (UID: \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"
Sep 18 20:29:46.554254 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.554153    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2\" (UID: \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"
Sep 18 20:29:46.554254 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.554176    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2ml6\" (UniqueName: \"kubernetes.io/projected/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-kube-api-access-m2ml6\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2\" (UID: \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"
Sep 18 20:29:46.554254 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.554226    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2\" (UID: \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"
Sep 18 20:29:46.554432 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.554255    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2\" (UID: \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"
Sep 18 20:29:46.554432 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.554284    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2\" (UID: \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"
Sep 18 20:29:46.554432 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.554317    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2\" (UID: \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"
Sep 18 20:29:46.554432 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.554386    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2\" (UID: \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"
Sep 18 20:29:46.655203 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.655168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2\" (UID: \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"
Sep 18 20:29:46.655422 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.655215    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2ml6\" (UniqueName: \"kubernetes.io/projected/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-kube-api-access-m2ml6\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2\" (UID: \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"
Sep 18 20:29:46.655422 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.655244    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2\" (UID: \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"
Sep 18 20:29:46.655422 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.655280    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2\" (UID: \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"
Sep 18 20:29:46.655422 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.655310    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2\" (UID: \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"
Sep 18 20:29:46.655422 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.655360    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2\" (UID: \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"
Sep 18 20:29:46.655422 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.655395    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2\" (UID: \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"
Sep 18 20:29:46.655773 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.655433    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2\" (UID: \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"
Sep 18 20:29:46.655773 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.655632    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2\" (UID: \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"
Sep 18 20:29:46.655773 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.655646    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2\" (UID: \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"
Sep 18 20:29:46.655985 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.655911    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2\" (UID: \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"
Sep 18 20:29:46.656050 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.656016    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2\" (UID: \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"
Sep 18 20:29:46.656203 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.656184    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2\" (UID: \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"
Sep 18 20:29:46.658236 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.658217    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2\" (UID: \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"
Sep 18 20:29:46.658627 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.658608    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2\" (UID: \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"
Sep 18 20:29:46.664108 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.664089    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2ml6\" (UniqueName: \"kubernetes.io/projected/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-kube-api-access-m2ml6\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2\" (UID: \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"
Sep 18 20:29:46.846598 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:46.846514    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"
Sep 18 20:29:47.136056 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:47.136027    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"]
Sep 18 20:29:47.137549 ip-10-0-143-134 kubenswrapper[2581]: W0918 20:29:47.137524    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ed12ccf_e2d7_4995_a4ec_58e86d6ea351.slice/crio-ccee3a8fcd96f66420c63b578be2717ea35c3b9e8b813232f28dde30ad854e71 WatchSource:0}: Error finding container ccee3a8fcd96f66420c63b578be2717ea35c3b9e8b813232f28dde30ad854e71: Status 404 returned error can't find the container with id ccee3a8fcd96f66420c63b578be2717ea35c3b9e8b813232f28dde30ad854e71
Sep 18 20:29:47.218820 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:47.218794    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2" event={"ID":"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351","Type":"ContainerStarted","Data":"03528fe8d4e01ce800b2350347024341c2831a15a8c16e286483bc66c55a3c50"}
Sep 18 20:29:47.218945 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:47.218862    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2" event={"ID":"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351","Type":"ContainerStarted","Data":"ccee3a8fcd96f66420c63b578be2717ea35c3b9e8b813232f28dde30ad854e71"}
Sep 18 20:29:50.239871 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:50.239762    2581 generic.go:356] "Generic (PLEG): container finished" podID="8ed12ccf-e2d7-4995-a4ec-58e86d6ea351" containerID="03528fe8d4e01ce800b2350347024341c2831a15a8c16e286483bc66c55a3c50" exitCode=0
Sep 18 20:29:50.240252 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:50.239858    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2" event={"ID":"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351","Type":"ContainerDied","Data":"03528fe8d4e01ce800b2350347024341c2831a15a8c16e286483bc66c55a3c50"}
Sep 18 20:29:51.254002 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:51.253966    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2" event={"ID":"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351","Type":"ContainerStarted","Data":"92b441d0b9707129ce734b10c50f678eb9a03d00725979473b44f1b457ef1130"}
Sep 18 20:29:51.280116 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:51.280066    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2" podStartSLOduration=2.6069446 podStartE2EDuration="5.280051502s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.673106902s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:29:51.27781614 +0000 UTC m=+3114.380104467" watchObservedRunningTime="2026-09-18 20:29:51.280051502 +0000 UTC m=+3114.382339829"
Sep 18 20:29:51.847442 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:51.847409    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"
Sep 18 20:29:56.847719 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:56.847678    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"
Sep 18 20:29:56.849101 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:29:56.849069    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2" podUID="8ed12ccf-e2d7-4995-a4ec-58e86d6ea351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 18 20:30:06.847639 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:30:06.847595    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2" podUID="8ed12ccf-e2d7-4995-a4ec-58e86d6ea351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 18 20:30:16.847280 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:30:16.847235    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2" podUID="8ed12ccf-e2d7-4995-a4ec-58e86d6ea351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 18 20:30:26.847514 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:30:26.847408    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2" podUID="8ed12ccf-e2d7-4995-a4ec-58e86d6ea351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 18 20:30:36.847644 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:30:36.847592    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2" podUID="8ed12ccf-e2d7-4995-a4ec-58e86d6ea351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 18 20:30:46.847369 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:30:46.847326    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2" podUID="8ed12ccf-e2d7-4995-a4ec-58e86d6ea351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 18 20:30:56.847963 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:30:56.847913    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2" podUID="8ed12ccf-e2d7-4995-a4ec-58e86d6ea351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 18 20:31:06.857186 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:06.857151    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"
Sep 18 20:31:06.864908 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:06.864883    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"
Sep 18 20:31:08.651328 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:08.651298    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"]
Sep 18 20:31:23.707170 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:23.707102    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2" podUID="8ed12ccf-e2d7-4995-a4ec-58e86d6ea351" containerName="main" containerID="cri-o://92b441d0b9707129ce734b10c50f678eb9a03d00725979473b44f1b457ef1130" gracePeriod=45
Sep 18 20:31:23.858085 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:23.858041    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2" podUID="8ed12ccf-e2d7-4995-a4ec-58e86d6ea351" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 18 20:31:24.256877 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.256851    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"
Sep 18 20:31:24.306194 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.306168    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-model-cache\") pod \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\" (UID: \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\") "
Sep 18 20:31:24.306322 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.306211    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-dshm\") pod \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\" (UID: \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\") "
Sep 18 20:31:24.306322 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.306245    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-home\") pod \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\" (UID: \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\") "
Sep 18 20:31:24.306322 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.306273    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2ml6\" (UniqueName: \"kubernetes.io/projected/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-kube-api-access-m2ml6\") pod \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\" (UID: \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\") "
Sep 18 20:31:24.306322 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.306300    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-tls-certs\") pod \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\" (UID: \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\") "
Sep 18 20:31:24.306537 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.306378    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-cabundle-cert\") pod \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\" (UID: \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\") "
Sep 18 20:31:24.306537 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.306431    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-kserve-provision-location\") pod \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\" (UID: \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\") "
Sep 18 20:31:24.307359 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.306451    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-model-cache" (OuterVolumeSpecName: "model-cache") pod "8ed12ccf-e2d7-4995-a4ec-58e86d6ea351" (UID: "8ed12ccf-e2d7-4995-a4ec-58e86d6ea351"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:31:24.307359 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.306967    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-tmp-dir\") pod \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\" (UID: \"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351\") "
Sep 18 20:31:24.307359 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.307034    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-home" (OuterVolumeSpecName: "home") pod "8ed12ccf-e2d7-4995-a4ec-58e86d6ea351" (UID: "8ed12ccf-e2d7-4995-a4ec-58e86d6ea351"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:31:24.307359 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.307121    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8ed12ccf-e2d7-4995-a4ec-58e86d6ea351" (UID: "8ed12ccf-e2d7-4995-a4ec-58e86d6ea351"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:31:24.307830 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.307762    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8ed12ccf-e2d7-4995-a4ec-58e86d6ea351" (UID: "8ed12ccf-e2d7-4995-a4ec-58e86d6ea351"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:31:24.308146 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.308018    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-cabundle-cert\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:31:24.308146 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.308060    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-tmp-dir\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:31:24.308146 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.308074    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-model-cache\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:31:24.308146 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.308086    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-home\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:31:24.309814 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.309781    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-kube-api-access-m2ml6" (OuterVolumeSpecName: "kube-api-access-m2ml6") pod "8ed12ccf-e2d7-4995-a4ec-58e86d6ea351" (UID: "8ed12ccf-e2d7-4995-a4ec-58e86d6ea351"). InnerVolumeSpecName "kube-api-access-m2ml6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:31:24.309947 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.309905    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-dshm" (OuterVolumeSpecName: "dshm") pod "8ed12ccf-e2d7-4995-a4ec-58e86d6ea351" (UID: "8ed12ccf-e2d7-4995-a4ec-58e86d6ea351"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:31:24.310961 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.310932    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8ed12ccf-e2d7-4995-a4ec-58e86d6ea351" (UID: "8ed12ccf-e2d7-4995-a4ec-58e86d6ea351"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:31:24.338314 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.338252    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8ed12ccf-e2d7-4995-a4ec-58e86d6ea351" (UID: "8ed12ccf-e2d7-4995-a4ec-58e86d6ea351"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:31:24.409118 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.409099    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-kserve-provision-location\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:31:24.409118 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.409119    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-dshm\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:31:24.409232 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.409129    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2ml6\" (UniqueName: \"kubernetes.io/projected/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-kube-api-access-m2ml6\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:31:24.409232 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.409138    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351-tls-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:31:24.780176 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.780136    2581 generic.go:356] "Generic (PLEG): container finished" podID="8ed12ccf-e2d7-4995-a4ec-58e86d6ea351" containerID="92b441d0b9707129ce734b10c50f678eb9a03d00725979473b44f1b457ef1130" exitCode=0
Sep 18 20:31:24.780576 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.780212    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"
Sep 18 20:31:24.780576 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.780213    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2" event={"ID":"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351","Type":"ContainerDied","Data":"92b441d0b9707129ce734b10c50f678eb9a03d00725979473b44f1b457ef1130"}
Sep 18 20:31:24.780576 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.780270    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2" event={"ID":"8ed12ccf-e2d7-4995-a4ec-58e86d6ea351","Type":"ContainerDied","Data":"ccee3a8fcd96f66420c63b578be2717ea35c3b9e8b813232f28dde30ad854e71"}
Sep 18 20:31:24.780576 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.780291    2581 scope.go:117] "RemoveContainer" containerID="92b441d0b9707129ce734b10c50f678eb9a03d00725979473b44f1b457ef1130"
Sep 18 20:31:24.790217 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.790203    2581 scope.go:117] "RemoveContainer" containerID="03528fe8d4e01ce800b2350347024341c2831a15a8c16e286483bc66c55a3c50"
Sep 18 20:31:24.798771 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.798754    2581 scope.go:117] "RemoveContainer" containerID="92b441d0b9707129ce734b10c50f678eb9a03d00725979473b44f1b457ef1130"
Sep 18 20:31:24.799460 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:31:24.799438    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92b441d0b9707129ce734b10c50f678eb9a03d00725979473b44f1b457ef1130\": container with ID starting with 92b441d0b9707129ce734b10c50f678eb9a03d00725979473b44f1b457ef1130 not found: ID does not exist" containerID="92b441d0b9707129ce734b10c50f678eb9a03d00725979473b44f1b457ef1130"
Sep 18 20:31:24.799514 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.799468    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92b441d0b9707129ce734b10c50f678eb9a03d00725979473b44f1b457ef1130"} err="failed to get container status \"92b441d0b9707129ce734b10c50f678eb9a03d00725979473b44f1b457ef1130\": rpc error: code = NotFound desc = could not find container \"92b441d0b9707129ce734b10c50f678eb9a03d00725979473b44f1b457ef1130\": container with ID starting with 92b441d0b9707129ce734b10c50f678eb9a03d00725979473b44f1b457ef1130 not found: ID does not exist"
Sep 18 20:31:24.799514 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.799485    2581 scope.go:117] "RemoveContainer" containerID="03528fe8d4e01ce800b2350347024341c2831a15a8c16e286483bc66c55a3c50"
Sep 18 20:31:24.799695 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:31:24.799673    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03528fe8d4e01ce800b2350347024341c2831a15a8c16e286483bc66c55a3c50\": container with ID starting with 03528fe8d4e01ce800b2350347024341c2831a15a8c16e286483bc66c55a3c50 not found: ID does not exist" containerID="03528fe8d4e01ce800b2350347024341c2831a15a8c16e286483bc66c55a3c50"
Sep 18 20:31:24.799811 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.799691    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03528fe8d4e01ce800b2350347024341c2831a15a8c16e286483bc66c55a3c50"} err="failed to get container status \"03528fe8d4e01ce800b2350347024341c2831a15a8c16e286483bc66c55a3c50\": rpc error: code = NotFound desc = could not find container \"03528fe8d4e01ce800b2350347024341c2831a15a8c16e286483bc66c55a3c50\": container with ID starting with 03528fe8d4e01ce800b2350347024341c2831a15a8c16e286483bc66c55a3c50 not found: ID does not exist"
Sep 18 20:31:24.802685 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.802665    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"]
Sep 18 20:31:24.808515 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:24.808495    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-85555df775-ccnv2"]
Sep 18 20:31:25.433026 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:25.432991    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ed12ccf-e2d7-4995-a4ec-58e86d6ea351" path="/var/lib/kubelet/pods/8ed12ccf-e2d7-4995-a4ec-58e86d6ea351/volumes"
Sep 18 20:31:27.434923 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.434896    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"]
Sep 18 20:31:27.435298 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.435215    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ed12ccf-e2d7-4995-a4ec-58e86d6ea351" containerName="storage-initializer"
Sep 18 20:31:27.435298 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.435227    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ed12ccf-e2d7-4995-a4ec-58e86d6ea351" containerName="storage-initializer"
Sep 18 20:31:27.435298 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.435240    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ed12ccf-e2d7-4995-a4ec-58e86d6ea351" containerName="main"
Sep 18 20:31:27.435298 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.435245    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ed12ccf-e2d7-4995-a4ec-58e86d6ea351" containerName="main"
Sep 18 20:31:27.435429 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.435301    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ed12ccf-e2d7-4995-a4ec-58e86d6ea351" containerName="main"
Sep 18 20:31:27.439940 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.439918    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"
Sep 18 20:31:27.446294 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.446267    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Sep 18 20:31:27.447056 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.447034    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-rlfvg\""
Sep 18 20:31:27.447164 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.447059    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:31:27.447164 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.447061    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 18 20:31:27.447341 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.447323    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 18 20:31:27.448541 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.448524    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 18 20:31:27.460667 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.460642    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"]
Sep 18 20:31:27.534967 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.534936    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/929e03e5-5147-49af-994e-bd4e797ee2b2-tls-certs\") pod \"custom-route-timeout-test-kserve-85f684d5cc-rgc89\" (UID: \"929e03e5-5147-49af-994e-bd4e797ee2b2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"
Sep 18 20:31:27.535132 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.534974    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/929e03e5-5147-49af-994e-bd4e797ee2b2-cabundle-cert\") pod \"custom-route-timeout-test-kserve-85f684d5cc-rgc89\" (UID: \"929e03e5-5147-49af-994e-bd4e797ee2b2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"
Sep 18 20:31:27.535132 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.535000    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/929e03e5-5147-49af-994e-bd4e797ee2b2-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-85f684d5cc-rgc89\" (UID: \"929e03e5-5147-49af-994e-bd4e797ee2b2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"
Sep 18 20:31:27.535132 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.535045    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/929e03e5-5147-49af-994e-bd4e797ee2b2-home\") pod \"custom-route-timeout-test-kserve-85f684d5cc-rgc89\" (UID: \"929e03e5-5147-49af-994e-bd4e797ee2b2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"
Sep 18 20:31:27.535132 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.535072    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/929e03e5-5147-49af-994e-bd4e797ee2b2-dshm\") pod \"custom-route-timeout-test-kserve-85f684d5cc-rgc89\" (UID: \"929e03e5-5147-49af-994e-bd4e797ee2b2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"
Sep 18 20:31:27.535132 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.535096    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/929e03e5-5147-49af-994e-bd4e797ee2b2-tmp-dir\") pod \"custom-route-timeout-test-kserve-85f684d5cc-rgc89\" (UID: \"929e03e5-5147-49af-994e-bd4e797ee2b2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"
Sep 18 20:31:27.535132 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.535120    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnvfz\" (UniqueName: \"kubernetes.io/projected/929e03e5-5147-49af-994e-bd4e797ee2b2-kube-api-access-bnvfz\") pod \"custom-route-timeout-test-kserve-85f684d5cc-rgc89\" (UID: \"929e03e5-5147-49af-994e-bd4e797ee2b2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"
Sep 18 20:31:27.535333 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.535147    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/929e03e5-5147-49af-994e-bd4e797ee2b2-model-cache\") pod \"custom-route-timeout-test-kserve-85f684d5cc-rgc89\" (UID: \"929e03e5-5147-49af-994e-bd4e797ee2b2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"
Sep 18 20:31:27.636398 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.636366    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/929e03e5-5147-49af-994e-bd4e797ee2b2-home\") pod \"custom-route-timeout-test-kserve-85f684d5cc-rgc89\" (UID: \"929e03e5-5147-49af-994e-bd4e797ee2b2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"
Sep 18 20:31:27.636398 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.636402    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/929e03e5-5147-49af-994e-bd4e797ee2b2-dshm\") pod \"custom-route-timeout-test-kserve-85f684d5cc-rgc89\" (UID: \"929e03e5-5147-49af-994e-bd4e797ee2b2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"
Sep 18 20:31:27.636620 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.636421    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/929e03e5-5147-49af-994e-bd4e797ee2b2-tmp-dir\") pod \"custom-route-timeout-test-kserve-85f684d5cc-rgc89\" (UID: \"929e03e5-5147-49af-994e-bd4e797ee2b2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"
Sep 18 20:31:27.636620 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.636444    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnvfz\" (UniqueName: \"kubernetes.io/projected/929e03e5-5147-49af-994e-bd4e797ee2b2-kube-api-access-bnvfz\") pod \"custom-route-timeout-test-kserve-85f684d5cc-rgc89\" (UID: \"929e03e5-5147-49af-994e-bd4e797ee2b2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"
Sep 18 20:31:27.636620 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.636478    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/929e03e5-5147-49af-994e-bd4e797ee2b2-model-cache\") pod \"custom-route-timeout-test-kserve-85f684d5cc-rgc89\" (UID: \"929e03e5-5147-49af-994e-bd4e797ee2b2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"
Sep 18 20:31:27.636620 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.636532    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/929e03e5-5147-49af-994e-bd4e797ee2b2-tls-certs\") pod \"custom-route-timeout-test-kserve-85f684d5cc-rgc89\" (UID: \"929e03e5-5147-49af-994e-bd4e797ee2b2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"
Sep 18 20:31:27.636620 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.636557    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/929e03e5-5147-49af-994e-bd4e797ee2b2-cabundle-cert\") pod \"custom-route-timeout-test-kserve-85f684d5cc-rgc89\" (UID: \"929e03e5-5147-49af-994e-bd4e797ee2b2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"
Sep 18 20:31:27.636620 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.636581    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/929e03e5-5147-49af-994e-bd4e797ee2b2-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-85f684d5cc-rgc89\" (UID: \"929e03e5-5147-49af-994e-bd4e797ee2b2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"
Sep 18 20:31:27.636975 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.636803    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/929e03e5-5147-49af-994e-bd4e797ee2b2-home\") pod \"custom-route-timeout-test-kserve-85f684d5cc-rgc89\" (UID: \"929e03e5-5147-49af-994e-bd4e797ee2b2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"
Sep 18 20:31:27.636975 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.636927    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/929e03e5-5147-49af-994e-bd4e797ee2b2-tmp-dir\") pod \"custom-route-timeout-test-kserve-85f684d5cc-rgc89\" (UID: \"929e03e5-5147-49af-994e-bd4e797ee2b2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"
Sep 18 20:31:27.636975 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.636965    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/929e03e5-5147-49af-994e-bd4e797ee2b2-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-85f684d5cc-rgc89\" (UID: \"929e03e5-5147-49af-994e-bd4e797ee2b2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"
Sep 18 20:31:27.637121 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.637048    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/929e03e5-5147-49af-994e-bd4e797ee2b2-model-cache\") pod \"custom-route-timeout-test-kserve-85f684d5cc-rgc89\" (UID: \"929e03e5-5147-49af-994e-bd4e797ee2b2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"
Sep 18 20:31:27.637352 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.637251    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/929e03e5-5147-49af-994e-bd4e797ee2b2-cabundle-cert\") pod \"custom-route-timeout-test-kserve-85f684d5cc-rgc89\" (UID: \"929e03e5-5147-49af-994e-bd4e797ee2b2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"
Sep 18 20:31:27.638918 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.638896    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/929e03e5-5147-49af-994e-bd4e797ee2b2-dshm\") pod \"custom-route-timeout-test-kserve-85f684d5cc-rgc89\" (UID: \"929e03e5-5147-49af-994e-bd4e797ee2b2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"
Sep 18 20:31:27.639172 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.639152    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/929e03e5-5147-49af-994e-bd4e797ee2b2-tls-certs\") pod \"custom-route-timeout-test-kserve-85f684d5cc-rgc89\" (UID: \"929e03e5-5147-49af-994e-bd4e797ee2b2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"
Sep 18 20:31:27.645563 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.645546    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnvfz\" (UniqueName: \"kubernetes.io/projected/929e03e5-5147-49af-994e-bd4e797ee2b2-kube-api-access-bnvfz\") pod \"custom-route-timeout-test-kserve-85f684d5cc-rgc89\" (UID: \"929e03e5-5147-49af-994e-bd4e797ee2b2\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"
Sep 18 20:31:27.750155 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:27.750077    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"
Sep 18 20:31:28.029692 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:28.029665    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"]
Sep 18 20:31:28.030636 ip-10-0-143-134 kubenswrapper[2581]: W0918 20:31:28.030609    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod929e03e5_5147_49af_994e_bd4e797ee2b2.slice/crio-4f4747be9fcea47658e09578408072881ff737480ac70157a4b08e598e4eff2a WatchSource:0}: Error finding container 4f4747be9fcea47658e09578408072881ff737480ac70157a4b08e598e4eff2a: Status 404 returned error can't find the container with id 4f4747be9fcea47658e09578408072881ff737480ac70157a4b08e598e4eff2a
Sep 18 20:31:28.803913 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:28.803864    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89" event={"ID":"929e03e5-5147-49af-994e-bd4e797ee2b2","Type":"ContainerStarted","Data":"a6c7c7c37b9d8d95e964fbf4305a7691696e022b205dea9a7ac1a75b153eca52"}
Sep 18 20:31:28.804338 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:28.803950    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89" event={"ID":"929e03e5-5147-49af-994e-bd4e797ee2b2","Type":"ContainerStarted","Data":"4f4747be9fcea47658e09578408072881ff737480ac70157a4b08e598e4eff2a"}
Sep 18 20:31:30.816852 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:30.816805    2581 generic.go:356] "Generic (PLEG): container finished" podID="929e03e5-5147-49af-994e-bd4e797ee2b2" containerID="a6c7c7c37b9d8d95e964fbf4305a7691696e022b205dea9a7ac1a75b153eca52" exitCode=0
Sep 18 20:31:30.817244 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:30.816878    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89" event={"ID":"929e03e5-5147-49af-994e-bd4e797ee2b2","Type":"ContainerDied","Data":"a6c7c7c37b9d8d95e964fbf4305a7691696e022b205dea9a7ac1a75b153eca52"}
Sep 18 20:31:31.824300 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:31.824266    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89" event={"ID":"929e03e5-5147-49af-994e-bd4e797ee2b2","Type":"ContainerStarted","Data":"49f7ab2826011fcf2d35ad85a0d6b77d5cd17706f95ebf438a47c9b5aeed6980"}
Sep 18 20:31:31.846791 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:31.846738    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89" podStartSLOduration=2.744168526 podStartE2EDuration="4.84672162s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.102553094s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:31:31.844972171 +0000 UTC m=+3214.947260501" watchObservedRunningTime="2026-09-18 20:31:31.84672162 +0000 UTC m=+3214.949009948"
Sep 18 20:31:32.750276 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:32.750234    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"
Sep 18 20:31:37.750773 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:37.750725    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"
Sep 18 20:31:37.752467 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:37.752437    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89" podUID="929e03e5-5147-49af-994e-bd4e797ee2b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 18 20:31:47.751463 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:47.751418    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89" podUID="929e03e5-5147-49af-994e-bd4e797ee2b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 18 20:31:57.751367 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:31:57.751251    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89" podUID="929e03e5-5147-49af-994e-bd4e797ee2b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 18 20:32:07.751414 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:32:07.751366    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89" podUID="929e03e5-5147-49af-994e-bd4e797ee2b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 18 20:32:17.751050 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:32:17.751006    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89" podUID="929e03e5-5147-49af-994e-bd4e797ee2b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 18 20:32:27.751241 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:32:27.751200    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89" podUID="929e03e5-5147-49af-994e-bd4e797ee2b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 18 20:32:37.751376 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:32:37.751324    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89" podUID="929e03e5-5147-49af-994e-bd4e797ee2b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 18 20:32:47.751141 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:32:47.751096    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89" podUID="929e03e5-5147-49af-994e-bd4e797ee2b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 18 20:32:57.750487 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:32:57.750445    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89" podUID="929e03e5-5147-49af-994e-bd4e797ee2b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 18 20:33:07.751358 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:07.751305    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89" podUID="929e03e5-5147-49af-994e-bd4e797ee2b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 18 20:33:17.751563 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:17.751505    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89" podUID="929e03e5-5147-49af-994e-bd4e797ee2b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 18 20:33:27.761188 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:27.761109    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"
Sep 18 20:33:27.768920 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:27.768898    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"
Sep 18 20:33:32.243693 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:32.243658    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"]
Sep 18 20:33:47.267157 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:47.267088    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89" podUID="929e03e5-5147-49af-994e-bd4e797ee2b2" containerName="main" containerID="cri-o://49f7ab2826011fcf2d35ad85a0d6b77d5cd17706f95ebf438a47c9b5aeed6980" gracePeriod=45
Sep 18 20:33:47.762527 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:47.762480    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89" podUID="929e03e5-5147-49af-994e-bd4e797ee2b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 18 20:33:47.932532 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:47.932504    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"
Sep 18 20:33:48.029125 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.029102    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/929e03e5-5147-49af-994e-bd4e797ee2b2-kserve-provision-location\") pod \"929e03e5-5147-49af-994e-bd4e797ee2b2\" (UID: \"929e03e5-5147-49af-994e-bd4e797ee2b2\") "
Sep 18 20:33:48.029298 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.029216    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/929e03e5-5147-49af-994e-bd4e797ee2b2-home\") pod \"929e03e5-5147-49af-994e-bd4e797ee2b2\" (UID: \"929e03e5-5147-49af-994e-bd4e797ee2b2\") "
Sep 18 20:33:48.029298 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.029261    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/929e03e5-5147-49af-994e-bd4e797ee2b2-tls-certs\") pod \"929e03e5-5147-49af-994e-bd4e797ee2b2\" (UID: \"929e03e5-5147-49af-994e-bd4e797ee2b2\") "
Sep 18 20:33:48.029427 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.029321    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/929e03e5-5147-49af-994e-bd4e797ee2b2-cabundle-cert\") pod \"929e03e5-5147-49af-994e-bd4e797ee2b2\" (UID: \"929e03e5-5147-49af-994e-bd4e797ee2b2\") "
Sep 18 20:33:48.029427 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.029356    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnvfz\" (UniqueName: \"kubernetes.io/projected/929e03e5-5147-49af-994e-bd4e797ee2b2-kube-api-access-bnvfz\") pod \"929e03e5-5147-49af-994e-bd4e797ee2b2\" (UID: \"929e03e5-5147-49af-994e-bd4e797ee2b2\") "
Sep 18 20:33:48.029427 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.029389    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/929e03e5-5147-49af-994e-bd4e797ee2b2-tmp-dir\") pod \"929e03e5-5147-49af-994e-bd4e797ee2b2\" (UID: \"929e03e5-5147-49af-994e-bd4e797ee2b2\") "
Sep 18 20:33:48.029427 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.029413    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/929e03e5-5147-49af-994e-bd4e797ee2b2-model-cache\") pod \"929e03e5-5147-49af-994e-bd4e797ee2b2\" (UID: \"929e03e5-5147-49af-994e-bd4e797ee2b2\") "
Sep 18 20:33:48.029623 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.029444    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/929e03e5-5147-49af-994e-bd4e797ee2b2-dshm\") pod \"929e03e5-5147-49af-994e-bd4e797ee2b2\" (UID: \"929e03e5-5147-49af-994e-bd4e797ee2b2\") "
Sep 18 20:33:48.029995 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.029727    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/929e03e5-5147-49af-994e-bd4e797ee2b2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "929e03e5-5147-49af-994e-bd4e797ee2b2" (UID: "929e03e5-5147-49af-994e-bd4e797ee2b2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:33:48.029995 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.029955    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929e03e5-5147-49af-994e-bd4e797ee2b2-model-cache" (OuterVolumeSpecName: "model-cache") pod "929e03e5-5147-49af-994e-bd4e797ee2b2" (UID: "929e03e5-5147-49af-994e-bd4e797ee2b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:33:48.030267 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.030239    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929e03e5-5147-49af-994e-bd4e797ee2b2-home" (OuterVolumeSpecName: "home") pod "929e03e5-5147-49af-994e-bd4e797ee2b2" (UID: "929e03e5-5147-49af-994e-bd4e797ee2b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:33:48.031816 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.031788    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/929e03e5-5147-49af-994e-bd4e797ee2b2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "929e03e5-5147-49af-994e-bd4e797ee2b2" (UID: "929e03e5-5147-49af-994e-bd4e797ee2b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:33:48.032023 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.032002    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/929e03e5-5147-49af-994e-bd4e797ee2b2-kube-api-access-bnvfz" (OuterVolumeSpecName: "kube-api-access-bnvfz") pod "929e03e5-5147-49af-994e-bd4e797ee2b2" (UID: "929e03e5-5147-49af-994e-bd4e797ee2b2"). InnerVolumeSpecName "kube-api-access-bnvfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:33:48.032259 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.032241    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929e03e5-5147-49af-994e-bd4e797ee2b2-dshm" (OuterVolumeSpecName: "dshm") pod "929e03e5-5147-49af-994e-bd4e797ee2b2" (UID: "929e03e5-5147-49af-994e-bd4e797ee2b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:33:48.048956 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.048931    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929e03e5-5147-49af-994e-bd4e797ee2b2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "929e03e5-5147-49af-994e-bd4e797ee2b2" (UID: "929e03e5-5147-49af-994e-bd4e797ee2b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:33:48.054136 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.054114    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929e03e5-5147-49af-994e-bd4e797ee2b2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "929e03e5-5147-49af-994e-bd4e797ee2b2" (UID: "929e03e5-5147-49af-994e-bd4e797ee2b2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:33:48.130069 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.130043    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/929e03e5-5147-49af-994e-bd4e797ee2b2-kserve-provision-location\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:33:48.130069 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.130063    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/929e03e5-5147-49af-994e-bd4e797ee2b2-home\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:33:48.130069 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.130072    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/929e03e5-5147-49af-994e-bd4e797ee2b2-tls-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:33:48.130228 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.130082    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/929e03e5-5147-49af-994e-bd4e797ee2b2-cabundle-cert\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:33:48.130228 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.130090    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnvfz\" (UniqueName: \"kubernetes.io/projected/929e03e5-5147-49af-994e-bd4e797ee2b2-kube-api-access-bnvfz\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:33:48.130228 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.130100    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/929e03e5-5147-49af-994e-bd4e797ee2b2-tmp-dir\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:33:48.130228 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.130108    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/929e03e5-5147-49af-994e-bd4e797ee2b2-model-cache\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:33:48.130228 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.130116    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/929e03e5-5147-49af-994e-bd4e797ee2b2-dshm\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:33:48.619461 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.619415    2581 generic.go:356] "Generic (PLEG): container finished" podID="929e03e5-5147-49af-994e-bd4e797ee2b2" containerID="49f7ab2826011fcf2d35ad85a0d6b77d5cd17706f95ebf438a47c9b5aeed6980" exitCode=0
Sep 18 20:33:48.619987 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.619607    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89" event={"ID":"929e03e5-5147-49af-994e-bd4e797ee2b2","Type":"ContainerDied","Data":"49f7ab2826011fcf2d35ad85a0d6b77d5cd17706f95ebf438a47c9b5aeed6980"}
Sep 18 20:33:48.619987 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.619689    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89" event={"ID":"929e03e5-5147-49af-994e-bd4e797ee2b2","Type":"ContainerDied","Data":"4f4747be9fcea47658e09578408072881ff737480ac70157a4b08e598e4eff2a"}
Sep 18 20:33:48.619987 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.619735    2581 scope.go:117] "RemoveContainer" containerID="49f7ab2826011fcf2d35ad85a0d6b77d5cd17706f95ebf438a47c9b5aeed6980"
Sep 18 20:33:48.620418 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.620395    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"
Sep 18 20:33:48.631201 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.631185    2581 scope.go:117] "RemoveContainer" containerID="a6c7c7c37b9d8d95e964fbf4305a7691696e022b205dea9a7ac1a75b153eca52"
Sep 18 20:33:48.639639 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.639624    2581 scope.go:117] "RemoveContainer" containerID="49f7ab2826011fcf2d35ad85a0d6b77d5cd17706f95ebf438a47c9b5aeed6980"
Sep 18 20:33:48.639915 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:33:48.639896    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49f7ab2826011fcf2d35ad85a0d6b77d5cd17706f95ebf438a47c9b5aeed6980\": container with ID starting with 49f7ab2826011fcf2d35ad85a0d6b77d5cd17706f95ebf438a47c9b5aeed6980 not found: ID does not exist" containerID="49f7ab2826011fcf2d35ad85a0d6b77d5cd17706f95ebf438a47c9b5aeed6980"
Sep 18 20:33:48.639972 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.639921    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49f7ab2826011fcf2d35ad85a0d6b77d5cd17706f95ebf438a47c9b5aeed6980"} err="failed to get container status \"49f7ab2826011fcf2d35ad85a0d6b77d5cd17706f95ebf438a47c9b5aeed6980\": rpc error: code = NotFound desc = could not find container \"49f7ab2826011fcf2d35ad85a0d6b77d5cd17706f95ebf438a47c9b5aeed6980\": container with ID starting with 49f7ab2826011fcf2d35ad85a0d6b77d5cd17706f95ebf438a47c9b5aeed6980 not found: ID does not exist"
Sep 18 20:33:48.639972 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.639937    2581 scope.go:117] "RemoveContainer" containerID="a6c7c7c37b9d8d95e964fbf4305a7691696e022b205dea9a7ac1a75b153eca52"
Sep 18 20:33:48.640165 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:33:48.640150    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6c7c7c37b9d8d95e964fbf4305a7691696e022b205dea9a7ac1a75b153eca52\": container with ID starting with a6c7c7c37b9d8d95e964fbf4305a7691696e022b205dea9a7ac1a75b153eca52 not found: ID does not exist" containerID="a6c7c7c37b9d8d95e964fbf4305a7691696e022b205dea9a7ac1a75b153eca52"
Sep 18 20:33:48.640205 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.640168    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6c7c7c37b9d8d95e964fbf4305a7691696e022b205dea9a7ac1a75b153eca52"} err="failed to get container status \"a6c7c7c37b9d8d95e964fbf4305a7691696e022b205dea9a7ac1a75b153eca52\": rpc error: code = NotFound desc = could not find container \"a6c7c7c37b9d8d95e964fbf4305a7691696e022b205dea9a7ac1a75b153eca52\": container with ID starting with a6c7c7c37b9d8d95e964fbf4305a7691696e022b205dea9a7ac1a75b153eca52 not found: ID does not exist"
Sep 18 20:33:48.648125 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.648102    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"]
Sep 18 20:33:48.652683 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:48.652660    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-85f684d5cc-rgc89"]
Sep 18 20:33:49.433028 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:33:49.432997    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="929e03e5-5147-49af-994e-bd4e797ee2b2" path="/var/lib/kubelet/pods/929e03e5-5147-49af-994e-bd4e797ee2b2/volumes"
Sep 18 20:34:37.116720 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.116672    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"]
Sep 18 20:34:37.117290 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.117227    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="929e03e5-5147-49af-994e-bd4e797ee2b2" containerName="storage-initializer"
Sep 18 20:34:37.117290 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.117248    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="929e03e5-5147-49af-994e-bd4e797ee2b2" containerName="storage-initializer"
Sep 18 20:34:37.117290 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.117271    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="929e03e5-5147-49af-994e-bd4e797ee2b2" containerName="main"
Sep 18 20:34:37.117290 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.117279    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="929e03e5-5147-49af-994e-bd4e797ee2b2" containerName="main"
Sep 18 20:34:37.117514 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.117373    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="929e03e5-5147-49af-994e-bd4e797ee2b2" containerName="main"
Sep 18 20:34:37.120794 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.120773    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"
Sep 18 20:34:37.123370 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.123344    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:34:37.123478 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.123412    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Sep 18 20:34:37.123478 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.123425    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 18 20:34:37.123602 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.123477    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 18 20:34:37.123643 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.123621    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-2v6xm\""
Sep 18 20:34:37.124314 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.124295    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 18 20:34:37.130863 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.130843    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"]
Sep 18 20:34:37.132144 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.132128    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64cb5046-d871-4d00-91a6-752d0fe019c8-cabundle-cert\") pod \"router-with-refs-test-kserve-fbf4cf545-wf49k\" (UID: \"64cb5046-d871-4d00-91a6-752d0fe019c8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"
Sep 18 20:34:37.132193 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.132156    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht9ls\" (UniqueName: \"kubernetes.io/projected/64cb5046-d871-4d00-91a6-752d0fe019c8-kube-api-access-ht9ls\") pod \"router-with-refs-test-kserve-fbf4cf545-wf49k\" (UID: \"64cb5046-d871-4d00-91a6-752d0fe019c8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"
Sep 18 20:34:37.132193 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.132179    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64cb5046-d871-4d00-91a6-752d0fe019c8-kserve-provision-location\") pod \"router-with-refs-test-kserve-fbf4cf545-wf49k\" (UID: \"64cb5046-d871-4d00-91a6-752d0fe019c8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"
Sep 18 20:34:37.132269 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.132225    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64cb5046-d871-4d00-91a6-752d0fe019c8-model-cache\") pod \"router-with-refs-test-kserve-fbf4cf545-wf49k\" (UID: \"64cb5046-d871-4d00-91a6-752d0fe019c8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"
Sep 18 20:34:37.132304 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.132267    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64cb5046-d871-4d00-91a6-752d0fe019c8-home\") pod \"router-with-refs-test-kserve-fbf4cf545-wf49k\" (UID: \"64cb5046-d871-4d00-91a6-752d0fe019c8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"
Sep 18 20:34:37.132358 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.132341    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64cb5046-d871-4d00-91a6-752d0fe019c8-tls-certs\") pod \"router-with-refs-test-kserve-fbf4cf545-wf49k\" (UID: \"64cb5046-d871-4d00-91a6-752d0fe019c8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"
Sep 18 20:34:37.132398 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.132375    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64cb5046-d871-4d00-91a6-752d0fe019c8-tmp-dir\") pod \"router-with-refs-test-kserve-fbf4cf545-wf49k\" (UID: \"64cb5046-d871-4d00-91a6-752d0fe019c8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"
Sep 18 20:34:37.132398 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.132393    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64cb5046-d871-4d00-91a6-752d0fe019c8-dshm\") pod \"router-with-refs-test-kserve-fbf4cf545-wf49k\" (UID: \"64cb5046-d871-4d00-91a6-752d0fe019c8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"
Sep 18 20:34:37.233309 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.233273    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64cb5046-d871-4d00-91a6-752d0fe019c8-model-cache\") pod \"router-with-refs-test-kserve-fbf4cf545-wf49k\" (UID: \"64cb5046-d871-4d00-91a6-752d0fe019c8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"
Sep 18 20:34:37.233478 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.233319    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64cb5046-d871-4d00-91a6-752d0fe019c8-home\") pod \"router-with-refs-test-kserve-fbf4cf545-wf49k\" (UID: \"64cb5046-d871-4d00-91a6-752d0fe019c8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"
Sep 18 20:34:37.233478 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.233379    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64cb5046-d871-4d00-91a6-752d0fe019c8-tls-certs\") pod \"router-with-refs-test-kserve-fbf4cf545-wf49k\" (UID: \"64cb5046-d871-4d00-91a6-752d0fe019c8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"
Sep 18 20:34:37.233478 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.233412    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64cb5046-d871-4d00-91a6-752d0fe019c8-tmp-dir\") pod \"router-with-refs-test-kserve-fbf4cf545-wf49k\" (UID: \"64cb5046-d871-4d00-91a6-752d0fe019c8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"
Sep 18 20:34:37.233478 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.233465    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64cb5046-d871-4d00-91a6-752d0fe019c8-dshm\") pod \"router-with-refs-test-kserve-fbf4cf545-wf49k\" (UID: \"64cb5046-d871-4d00-91a6-752d0fe019c8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"
Sep 18 20:34:37.233724 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.233508    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64cb5046-d871-4d00-91a6-752d0fe019c8-cabundle-cert\") pod \"router-with-refs-test-kserve-fbf4cf545-wf49k\" (UID: \"64cb5046-d871-4d00-91a6-752d0fe019c8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"
Sep 18 20:34:37.233724 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.233568    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht9ls\" (UniqueName: \"kubernetes.io/projected/64cb5046-d871-4d00-91a6-752d0fe019c8-kube-api-access-ht9ls\") pod \"router-with-refs-test-kserve-fbf4cf545-wf49k\" (UID: \"64cb5046-d871-4d00-91a6-752d0fe019c8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"
Sep 18 20:34:37.233724 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.233609    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64cb5046-d871-4d00-91a6-752d0fe019c8-kserve-provision-location\") pod \"router-with-refs-test-kserve-fbf4cf545-wf49k\" (UID: \"64cb5046-d871-4d00-91a6-752d0fe019c8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"
Sep 18 20:34:37.233724 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.233648    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64cb5046-d871-4d00-91a6-752d0fe019c8-model-cache\") pod \"router-with-refs-test-kserve-fbf4cf545-wf49k\" (UID: \"64cb5046-d871-4d00-91a6-752d0fe019c8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"
Sep 18 20:34:37.233724 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.233702    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64cb5046-d871-4d00-91a6-752d0fe019c8-tmp-dir\") pod \"router-with-refs-test-kserve-fbf4cf545-wf49k\" (UID: \"64cb5046-d871-4d00-91a6-752d0fe019c8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"
Sep 18 20:34:37.234020 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.233781    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64cb5046-d871-4d00-91a6-752d0fe019c8-home\") pod \"router-with-refs-test-kserve-fbf4cf545-wf49k\" (UID: \"64cb5046-d871-4d00-91a6-752d0fe019c8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"
Sep 18 20:34:37.234020 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.233951    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64cb5046-d871-4d00-91a6-752d0fe019c8-kserve-provision-location\") pod \"router-with-refs-test-kserve-fbf4cf545-wf49k\" (UID: \"64cb5046-d871-4d00-91a6-752d0fe019c8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"
Sep 18 20:34:37.234303 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.234131    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64cb5046-d871-4d00-91a6-752d0fe019c8-cabundle-cert\") pod \"router-with-refs-test-kserve-fbf4cf545-wf49k\" (UID: \"64cb5046-d871-4d00-91a6-752d0fe019c8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"
Sep 18 20:34:37.235867 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.235848    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64cb5046-d871-4d00-91a6-752d0fe019c8-dshm\") pod \"router-with-refs-test-kserve-fbf4cf545-wf49k\" (UID: \"64cb5046-d871-4d00-91a6-752d0fe019c8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"
Sep 18 20:34:37.236136 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.236117    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64cb5046-d871-4d00-91a6-752d0fe019c8-tls-certs\") pod \"router-with-refs-test-kserve-fbf4cf545-wf49k\" (UID: \"64cb5046-d871-4d00-91a6-752d0fe019c8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"
Sep 18 20:34:37.241446 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.241429    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht9ls\" (UniqueName: \"kubernetes.io/projected/64cb5046-d871-4d00-91a6-752d0fe019c8-kube-api-access-ht9ls\") pod \"router-with-refs-test-kserve-fbf4cf545-wf49k\" (UID: \"64cb5046-d871-4d00-91a6-752d0fe019c8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"
Sep 18 20:34:37.432272 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.432201    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"
Sep 18 20:34:37.708796 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.708771    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"]
Sep 18 20:34:37.710461 ip-10-0-143-134 kubenswrapper[2581]: W0918 20:34:37.710433    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64cb5046_d871_4d00_91a6_752d0fe019c8.slice/crio-486f0bf04b523b456b714444314eed48d13ed454a9a90e09111dd20214755613 WatchSource:0}: Error finding container 486f0bf04b523b456b714444314eed48d13ed454a9a90e09111dd20214755613: Status 404 returned error can't find the container with id 486f0bf04b523b456b714444314eed48d13ed454a9a90e09111dd20214755613
Sep 18 20:34:37.884153 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.884111    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k" event={"ID":"64cb5046-d871-4d00-91a6-752d0fe019c8","Type":"ContainerStarted","Data":"f76ce8baf77ae581e005df67ae301d19c24a5c1a23bec713aec5c7965d8cb018"}
Sep 18 20:34:37.884341 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:37.884178    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k" event={"ID":"64cb5046-d871-4d00-91a6-752d0fe019c8","Type":"ContainerStarted","Data":"486f0bf04b523b456b714444314eed48d13ed454a9a90e09111dd20214755613"}
Sep 18 20:34:39.896505 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:39.896471    2581 generic.go:356] "Generic (PLEG): container finished" podID="64cb5046-d871-4d00-91a6-752d0fe019c8" containerID="f76ce8baf77ae581e005df67ae301d19c24a5c1a23bec713aec5c7965d8cb018" exitCode=0
Sep 18 20:34:39.896868 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:39.896548    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k" event={"ID":"64cb5046-d871-4d00-91a6-752d0fe019c8","Type":"ContainerDied","Data":"f76ce8baf77ae581e005df67ae301d19c24a5c1a23bec713aec5c7965d8cb018"}
Sep 18 20:34:40.904274 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:40.904237    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k" event={"ID":"64cb5046-d871-4d00-91a6-752d0fe019c8","Type":"ContainerStarted","Data":"4322d17d93f73ebc02d15ff55de932ae808081c39bd58ba62338d91a2bca1a1b"}
Sep 18 20:34:40.929591 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:40.929531    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k" podStartSLOduration=1.8399426810000001 podStartE2EDuration="3.929512131s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.08956945s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:34:40.926024306 +0000 UTC m=+3404.028312660" watchObservedRunningTime="2026-09-18 20:34:40.929512131 +0000 UTC m=+3404.031800460"
Sep 18 20:34:41.434435 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:41.434404    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"
Sep 18 20:34:47.434132 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:47.434104    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"
Sep 18 20:34:47.434695 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:47.434655    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k" podUID="64cb5046-d871-4d00-91a6-752d0fe019c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 18 20:34:57.432788 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:34:57.432694    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k" podUID="64cb5046-d871-4d00-91a6-752d0fe019c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 18 20:35:07.433434 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:35:07.433399    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k" podUID="64cb5046-d871-4d00-91a6-752d0fe019c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 18 20:35:17.433163 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:35:17.433117    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k" podUID="64cb5046-d871-4d00-91a6-752d0fe019c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 18 20:35:27.432897 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:35:27.432858    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k" podUID="64cb5046-d871-4d00-91a6-752d0fe019c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 18 20:35:37.432971 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:35:37.432932    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k" podUID="64cb5046-d871-4d00-91a6-752d0fe019c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 18 20:35:47.433058 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:35:47.433013    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k" podUID="64cb5046-d871-4d00-91a6-752d0fe019c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 18 20:35:57.432671 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:35:57.432632    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k" podUID="64cb5046-d871-4d00-91a6-752d0fe019c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 18 20:36:07.438281 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:36:07.433430    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k" podUID="64cb5046-d871-4d00-91a6-752d0fe019c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 18 20:36:17.433261 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:36:17.433209    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k" podUID="64cb5046-d871-4d00-91a6-752d0fe019c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 18 20:36:27.443488 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:36:27.443408    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"
Sep 18 20:36:27.451248 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:36:27.451231    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"
Sep 18 20:40:36.196537 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:36.196502    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"]
Sep 18 20:40:51.219671 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:51.219552    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k" podUID="64cb5046-d871-4d00-91a6-752d0fe019c8" containerName="main" containerID="cri-o://4322d17d93f73ebc02d15ff55de932ae808081c39bd58ba62338d91a2bca1a1b" gracePeriod=45
Sep 18 20:40:51.444384 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:51.444340    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k" podUID="64cb5046-d871-4d00-91a6-752d0fe019c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 18 20:40:51.926086 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:51.926052    2581 generic.go:356] "Generic (PLEG): container finished" podID="64cb5046-d871-4d00-91a6-752d0fe019c8" containerID="4322d17d93f73ebc02d15ff55de932ae808081c39bd58ba62338d91a2bca1a1b" exitCode=0
Sep 18 20:40:51.926229 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:51.926105    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k" event={"ID":"64cb5046-d871-4d00-91a6-752d0fe019c8","Type":"ContainerDied","Data":"4322d17d93f73ebc02d15ff55de932ae808081c39bd58ba62338d91a2bca1a1b"}
Sep 18 20:40:51.987022 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:51.986954    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"
Sep 18 20:40:52.090861 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:52.090808    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64cb5046-d871-4d00-91a6-752d0fe019c8-home\") pod \"64cb5046-d871-4d00-91a6-752d0fe019c8\" (UID: \"64cb5046-d871-4d00-91a6-752d0fe019c8\") "
Sep 18 20:40:52.090861 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:52.090860    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64cb5046-d871-4d00-91a6-752d0fe019c8-tls-certs\") pod \"64cb5046-d871-4d00-91a6-752d0fe019c8\" (UID: \"64cb5046-d871-4d00-91a6-752d0fe019c8\") "
Sep 18 20:40:52.091082 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:52.090890    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64cb5046-d871-4d00-91a6-752d0fe019c8-cabundle-cert\") pod \"64cb5046-d871-4d00-91a6-752d0fe019c8\" (UID: \"64cb5046-d871-4d00-91a6-752d0fe019c8\") "
Sep 18 20:40:52.091082 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:52.090913    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64cb5046-d871-4d00-91a6-752d0fe019c8-kserve-provision-location\") pod \"64cb5046-d871-4d00-91a6-752d0fe019c8\" (UID: \"64cb5046-d871-4d00-91a6-752d0fe019c8\") "
Sep 18 20:40:52.091082 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:52.090939    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ht9ls\" (UniqueName: \"kubernetes.io/projected/64cb5046-d871-4d00-91a6-752d0fe019c8-kube-api-access-ht9ls\") pod \"64cb5046-d871-4d00-91a6-752d0fe019c8\" (UID: \"64cb5046-d871-4d00-91a6-752d0fe019c8\") "
Sep 18 20:40:52.091082 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:52.091004    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64cb5046-d871-4d00-91a6-752d0fe019c8-tmp-dir\") pod \"64cb5046-d871-4d00-91a6-752d0fe019c8\" (UID: \"64cb5046-d871-4d00-91a6-752d0fe019c8\") "
Sep 18 20:40:52.091082 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:52.091037    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64cb5046-d871-4d00-91a6-752d0fe019c8-model-cache\") pod \"64cb5046-d871-4d00-91a6-752d0fe019c8\" (UID: \"64cb5046-d871-4d00-91a6-752d0fe019c8\") "
Sep 18 20:40:52.091082 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:52.091064    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64cb5046-d871-4d00-91a6-752d0fe019c8-dshm\") pod \"64cb5046-d871-4d00-91a6-752d0fe019c8\" (UID: \"64cb5046-d871-4d00-91a6-752d0fe019c8\") "
Sep 18 20:40:52.091397 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:52.091249    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64cb5046-d871-4d00-91a6-752d0fe019c8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "64cb5046-d871-4d00-91a6-752d0fe019c8" (UID: "64cb5046-d871-4d00-91a6-752d0fe019c8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:40:52.091510 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:52.091453    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64cb5046-d871-4d00-91a6-752d0fe019c8-model-cache" (OuterVolumeSpecName: "model-cache") pod "64cb5046-d871-4d00-91a6-752d0fe019c8" (UID: "64cb5046-d871-4d00-91a6-752d0fe019c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:40:52.091510 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:52.091483    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64cb5046-d871-4d00-91a6-752d0fe019c8-cabundle-cert\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:40:52.092172 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:52.091640    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64cb5046-d871-4d00-91a6-752d0fe019c8-home" (OuterVolumeSpecName: "home") pod "64cb5046-d871-4d00-91a6-752d0fe019c8" (UID: "64cb5046-d871-4d00-91a6-752d0fe019c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:40:52.093691 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:52.093660    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64cb5046-d871-4d00-91a6-752d0fe019c8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "64cb5046-d871-4d00-91a6-752d0fe019c8" (UID: "64cb5046-d871-4d00-91a6-752d0fe019c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:40:52.093901 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:52.093879    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64cb5046-d871-4d00-91a6-752d0fe019c8-dshm" (OuterVolumeSpecName: "dshm") pod "64cb5046-d871-4d00-91a6-752d0fe019c8" (UID: "64cb5046-d871-4d00-91a6-752d0fe019c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:40:52.094312 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:52.094284    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64cb5046-d871-4d00-91a6-752d0fe019c8-kube-api-access-ht9ls" (OuterVolumeSpecName: "kube-api-access-ht9ls") pod "64cb5046-d871-4d00-91a6-752d0fe019c8" (UID: "64cb5046-d871-4d00-91a6-752d0fe019c8"). InnerVolumeSpecName "kube-api-access-ht9ls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:40:52.105998 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:52.105959    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64cb5046-d871-4d00-91a6-752d0fe019c8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "64cb5046-d871-4d00-91a6-752d0fe019c8" (UID: "64cb5046-d871-4d00-91a6-752d0fe019c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:40:52.116724 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:52.116698    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64cb5046-d871-4d00-91a6-752d0fe019c8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "64cb5046-d871-4d00-91a6-752d0fe019c8" (UID: "64cb5046-d871-4d00-91a6-752d0fe019c8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:40:52.192119 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:52.192081    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64cb5046-d871-4d00-91a6-752d0fe019c8-kserve-provision-location\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:40:52.192119 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:52.192114    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht9ls\" (UniqueName: \"kubernetes.io/projected/64cb5046-d871-4d00-91a6-752d0fe019c8-kube-api-access-ht9ls\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:40:52.192119 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:52.192124    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64cb5046-d871-4d00-91a6-752d0fe019c8-tmp-dir\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:40:52.192391 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:52.192134    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64cb5046-d871-4d00-91a6-752d0fe019c8-model-cache\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:40:52.192391 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:52.192142    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64cb5046-d871-4d00-91a6-752d0fe019c8-dshm\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:40:52.192391 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:52.192149    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64cb5046-d871-4d00-91a6-752d0fe019c8-home\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:40:52.192391 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:52.192156    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64cb5046-d871-4d00-91a6-752d0fe019c8-tls-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:40:52.933977 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:52.933943    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k" event={"ID":"64cb5046-d871-4d00-91a6-752d0fe019c8","Type":"ContainerDied","Data":"486f0bf04b523b456b714444314eed48d13ed454a9a90e09111dd20214755613"}
Sep 18 20:40:52.934419 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:52.934000    2581 scope.go:117] "RemoveContainer" containerID="4322d17d93f73ebc02d15ff55de932ae808081c39bd58ba62338d91a2bca1a1b"
Sep 18 20:40:52.934419 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:52.933961    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"
Sep 18 20:40:52.944292 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:52.944270    2581 scope.go:117] "RemoveContainer" containerID="f76ce8baf77ae581e005df67ae301d19c24a5c1a23bec713aec5c7965d8cb018"
Sep 18 20:40:52.957772 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:52.957749    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"]
Sep 18 20:40:52.962271 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:52.962246    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-fbf4cf545-wf49k"]
Sep 18 20:40:53.433820 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:40:53.433785    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64cb5046-d871-4d00-91a6-752d0fe019c8" path="/var/lib/kubelet/pods/64cb5046-d871-4d00-91a6-752d0fe019c8/volumes"
Sep 18 20:41:31.320357 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.320320    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"]
Sep 18 20:41:31.320829 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.320673    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64cb5046-d871-4d00-91a6-752d0fe019c8" containerName="storage-initializer"
Sep 18 20:41:31.320829 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.320684    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="64cb5046-d871-4d00-91a6-752d0fe019c8" containerName="storage-initializer"
Sep 18 20:41:31.320829 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.320692    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64cb5046-d871-4d00-91a6-752d0fe019c8" containerName="main"
Sep 18 20:41:31.320829 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.320698    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="64cb5046-d871-4d00-91a6-752d0fe019c8" containerName="main"
Sep 18 20:41:31.320829 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.320755    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="64cb5046-d871-4d00-91a6-752d0fe019c8" containerName="main"
Sep 18 20:41:31.323997 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.323981    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"
Sep 18 20:41:31.328426 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.328403    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Sep 18 20:41:31.328585 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.328460    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-gqtgg\""
Sep 18 20:41:31.329077 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.329058    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:41:31.329173 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.329084    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Sep 18 20:41:31.329173 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.329136    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Sep 18 20:41:31.329173 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.329084    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Sep 18 20:41:31.339784 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.339761    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"]
Sep 18 20:41:31.433043 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.433016    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d658872-2a83-4c96-a39f-e00ebdf0ee91-model-cache\") pod \"lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd\" (UID: \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"
Sep 18 20:41:31.433209 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.433054    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d658872-2a83-4c96-a39f-e00ebdf0ee91-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd\" (UID: \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"
Sep 18 20:41:31.433209 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.433072    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d658872-2a83-4c96-a39f-e00ebdf0ee91-home\") pod \"lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd\" (UID: \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"
Sep 18 20:41:31.433209 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.433158    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d658872-2a83-4c96-a39f-e00ebdf0ee91-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd\" (UID: \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"
Sep 18 20:41:31.433209 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.433193    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d658872-2a83-4c96-a39f-e00ebdf0ee91-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd\" (UID: \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"
Sep 18 20:41:31.433382 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.433216    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9phjf\" (UniqueName: \"kubernetes.io/projected/6d658872-2a83-4c96-a39f-e00ebdf0ee91-kube-api-access-9phjf\") pod \"lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd\" (UID: \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"
Sep 18 20:41:31.433382 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.433253    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d658872-2a83-4c96-a39f-e00ebdf0ee91-dshm\") pod \"lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd\" (UID: \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"
Sep 18 20:41:31.433382 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.433267    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d658872-2a83-4c96-a39f-e00ebdf0ee91-tls-certs\") pod \"lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd\" (UID: \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"
Sep 18 20:41:31.534590 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.534534    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d658872-2a83-4c96-a39f-e00ebdf0ee91-model-cache\") pod \"lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd\" (UID: \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"
Sep 18 20:41:31.534739 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.534716    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d658872-2a83-4c96-a39f-e00ebdf0ee91-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd\" (UID: \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"
Sep 18 20:41:31.534811 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.534754    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d658872-2a83-4c96-a39f-e00ebdf0ee91-home\") pod \"lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd\" (UID: \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"
Sep 18 20:41:31.534904 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.534828    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d658872-2a83-4c96-a39f-e00ebdf0ee91-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd\" (UID: \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"
Sep 18 20:41:31.534961 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.534911    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d658872-2a83-4c96-a39f-e00ebdf0ee91-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd\" (UID: \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"
Sep 18 20:41:31.534961 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.534951    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9phjf\" (UniqueName: \"kubernetes.io/projected/6d658872-2a83-4c96-a39f-e00ebdf0ee91-kube-api-access-9phjf\") pod \"lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd\" (UID: \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"
Sep 18 20:41:31.535035 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.534987    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d658872-2a83-4c96-a39f-e00ebdf0ee91-dshm\") pod \"lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd\" (UID: \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"
Sep 18 20:41:31.535035 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.534998    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d658872-2a83-4c96-a39f-e00ebdf0ee91-model-cache\") pod \"lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd\" (UID: \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"
Sep 18 20:41:31.535035 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.535011    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d658872-2a83-4c96-a39f-e00ebdf0ee91-tls-certs\") pod \"lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd\" (UID: \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"
Sep 18 20:41:31.535183 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.535080    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d658872-2a83-4c96-a39f-e00ebdf0ee91-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd\" (UID: \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"
Sep 18 20:41:31.535183 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.535118    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d658872-2a83-4c96-a39f-e00ebdf0ee91-home\") pod \"lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd\" (UID: \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"
Sep 18 20:41:31.535279 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.535187    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d658872-2a83-4c96-a39f-e00ebdf0ee91-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd\" (UID: \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"
Sep 18 20:41:31.535512 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.535487    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d658872-2a83-4c96-a39f-e00ebdf0ee91-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd\" (UID: \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"
Sep 18 20:41:31.537323 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.537303    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d658872-2a83-4c96-a39f-e00ebdf0ee91-dshm\") pod \"lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd\" (UID: \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"
Sep 18 20:41:31.537692 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.537672    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d658872-2a83-4c96-a39f-e00ebdf0ee91-tls-certs\") pod \"lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd\" (UID: \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"
Sep 18 20:41:31.544059 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.544032    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9phjf\" (UniqueName: \"kubernetes.io/projected/6d658872-2a83-4c96-a39f-e00ebdf0ee91-kube-api-access-9phjf\") pod \"lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd\" (UID: \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"
Sep 18 20:41:31.634743 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.634681    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"
Sep 18 20:41:31.919290 ip-10-0-143-134 kubenswrapper[2581]: W0918 20:41:31.919255    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d658872_2a83_4c96_a39f_e00ebdf0ee91.slice/crio-edf753f2dc0faa56e4a9ab9cf55fc474090cbb3479406182bbfcb4102d50f812 WatchSource:0}: Error finding container edf753f2dc0faa56e4a9ab9cf55fc474090cbb3479406182bbfcb4102d50f812: Status 404 returned error can't find the container with id edf753f2dc0faa56e4a9ab9cf55fc474090cbb3479406182bbfcb4102d50f812
Sep 18 20:41:31.919681 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:31.919658    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"]
Sep 18 20:41:32.146365 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:32.146329    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd" event={"ID":"6d658872-2a83-4c96-a39f-e00ebdf0ee91","Type":"ContainerStarted","Data":"975ff7575c4c7cfb3670213094a70d02ed465d474e542f6fcdfe22f56dcc6c0f"}
Sep 18 20:41:32.146560 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:32.146380    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd" event={"ID":"6d658872-2a83-4c96-a39f-e00ebdf0ee91","Type":"ContainerStarted","Data":"edf753f2dc0faa56e4a9ab9cf55fc474090cbb3479406182bbfcb4102d50f812"}
Sep 18 20:41:35.163990 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:35.163900    2581 generic.go:356] "Generic (PLEG): container finished" podID="6d658872-2a83-4c96-a39f-e00ebdf0ee91" containerID="975ff7575c4c7cfb3670213094a70d02ed465d474e542f6fcdfe22f56dcc6c0f" exitCode=0
Sep 18 20:41:35.164368 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:35.163979    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd" event={"ID":"6d658872-2a83-4c96-a39f-e00ebdf0ee91","Type":"ContainerDied","Data":"975ff7575c4c7cfb3670213094a70d02ed465d474e542f6fcdfe22f56dcc6c0f"}
Sep 18 20:41:36.172910 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:36.172875    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd" event={"ID":"6d658872-2a83-4c96-a39f-e00ebdf0ee91","Type":"ContainerStarted","Data":"22855727c863488f3dda8a1af4187fa0505c1945098f248b66ad5ae4054dff12"}
Sep 18 20:41:36.196141 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:36.196079    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd" podStartSLOduration=2.353260448 podStartE2EDuration="5.19606396s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.842803512s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:41:36.195318828 +0000 UTC m=+3819.297607157" watchObservedRunningTime="2026-09-18 20:41:36.19606396 +0000 UTC m=+3819.298352287"
Sep 18 20:41:36.635796 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:36.635770    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"
Sep 18 20:41:41.635561 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:41.635524    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"
Sep 18 20:41:41.637191 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:41.637164    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd" podUID="6d658872-2a83-4c96-a39f-e00ebdf0ee91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 18 20:41:42.076723 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.076684    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"]
Sep 18 20:41:42.083455 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.083426    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"
Sep 18 20:41:42.086321 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.086298    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 18 20:41:42.086465 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.086321    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 18 20:41:42.086756 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.086737    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-smjj5\""
Sep 18 20:41:42.086885 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.086770    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 18 20:41:42.087226 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.087212    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:41:42.087306 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.087233    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 18 20:41:42.093935 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.093908    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"]
Sep 18 20:41:42.135687 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.135655    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bd350ed-dab5-4380-ba42-8f520fff524b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9\" (UID: \"2bd350ed-dab5-4380-ba42-8f520fff524b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"
Sep 18 20:41:42.135687 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.135685    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prl4d\" (UniqueName: \"kubernetes.io/projected/2bd350ed-dab5-4380-ba42-8f520fff524b-kube-api-access-prl4d\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9\" (UID: \"2bd350ed-dab5-4380-ba42-8f520fff524b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"
Sep 18 20:41:42.135917 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.135720    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bd350ed-dab5-4380-ba42-8f520fff524b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9\" (UID: \"2bd350ed-dab5-4380-ba42-8f520fff524b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"
Sep 18 20:41:42.135917 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.135816    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bd350ed-dab5-4380-ba42-8f520fff524b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9\" (UID: \"2bd350ed-dab5-4380-ba42-8f520fff524b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"
Sep 18 20:41:42.135917 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.135864    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bd350ed-dab5-4380-ba42-8f520fff524b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9\" (UID: \"2bd350ed-dab5-4380-ba42-8f520fff524b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"
Sep 18 20:41:42.135917 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.135892    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bd350ed-dab5-4380-ba42-8f520fff524b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9\" (UID: \"2bd350ed-dab5-4380-ba42-8f520fff524b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"
Sep 18 20:41:42.136053 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.135925    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bd350ed-dab5-4380-ba42-8f520fff524b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9\" (UID: \"2bd350ed-dab5-4380-ba42-8f520fff524b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"
Sep 18 20:41:42.136053 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.135951    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bd350ed-dab5-4380-ba42-8f520fff524b-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9\" (UID: \"2bd350ed-dab5-4380-ba42-8f520fff524b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"
Sep 18 20:41:42.236943 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.236905    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bd350ed-dab5-4380-ba42-8f520fff524b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9\" (UID: \"2bd350ed-dab5-4380-ba42-8f520fff524b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"
Sep 18 20:41:42.237115 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.236951    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bd350ed-dab5-4380-ba42-8f520fff524b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9\" (UID: \"2bd350ed-dab5-4380-ba42-8f520fff524b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"
Sep 18 20:41:42.237115 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.236983    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bd350ed-dab5-4380-ba42-8f520fff524b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9\" (UID: \"2bd350ed-dab5-4380-ba42-8f520fff524b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"
Sep 18 20:41:42.237115 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.237013    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bd350ed-dab5-4380-ba42-8f520fff524b-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9\" (UID: \"2bd350ed-dab5-4380-ba42-8f520fff524b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"
Sep 18 20:41:42.237115 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.237032    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bd350ed-dab5-4380-ba42-8f520fff524b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9\" (UID: \"2bd350ed-dab5-4380-ba42-8f520fff524b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"
Sep 18 20:41:42.237115 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.237047    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prl4d\" (UniqueName: \"kubernetes.io/projected/2bd350ed-dab5-4380-ba42-8f520fff524b-kube-api-access-prl4d\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9\" (UID: \"2bd350ed-dab5-4380-ba42-8f520fff524b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"
Sep 18 20:41:42.237115 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.237073    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bd350ed-dab5-4380-ba42-8f520fff524b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9\" (UID: \"2bd350ed-dab5-4380-ba42-8f520fff524b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"
Sep 18 20:41:42.237405 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.237150    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bd350ed-dab5-4380-ba42-8f520fff524b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9\" (UID: \"2bd350ed-dab5-4380-ba42-8f520fff524b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"
Sep 18 20:41:42.237405 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.237368    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bd350ed-dab5-4380-ba42-8f520fff524b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9\" (UID: \"2bd350ed-dab5-4380-ba42-8f520fff524b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"
Sep 18 20:41:42.237521 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.237450    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bd350ed-dab5-4380-ba42-8f520fff524b-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9\" (UID: \"2bd350ed-dab5-4380-ba42-8f520fff524b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"
Sep 18 20:41:42.237521 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.237484    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bd350ed-dab5-4380-ba42-8f520fff524b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9\" (UID: \"2bd350ed-dab5-4380-ba42-8f520fff524b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"
Sep 18 20:41:42.237687 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.237668    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bd350ed-dab5-4380-ba42-8f520fff524b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9\" (UID: \"2bd350ed-dab5-4380-ba42-8f520fff524b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"
Sep 18 20:41:42.237815 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.237767    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bd350ed-dab5-4380-ba42-8f520fff524b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9\" (UID: \"2bd350ed-dab5-4380-ba42-8f520fff524b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"
Sep 18 20:41:42.239645 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.239618    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bd350ed-dab5-4380-ba42-8f520fff524b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9\" (UID: \"2bd350ed-dab5-4380-ba42-8f520fff524b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"
Sep 18 20:41:42.239764 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.239748    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bd350ed-dab5-4380-ba42-8f520fff524b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9\" (UID: \"2bd350ed-dab5-4380-ba42-8f520fff524b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"
Sep 18 20:41:42.245291 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.245267    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prl4d\" (UniqueName: \"kubernetes.io/projected/2bd350ed-dab5-4380-ba42-8f520fff524b-kube-api-access-prl4d\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9\" (UID: \"2bd350ed-dab5-4380-ba42-8f520fff524b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"
Sep 18 20:41:42.396629 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.396538    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"
Sep 18 20:41:42.673749 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:42.673721    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"]
Sep 18 20:41:42.675268 ip-10-0-143-134 kubenswrapper[2581]: W0918 20:41:42.675238    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd350ed_dab5_4380_ba42_8f520fff524b.slice/crio-ed5251df6dd11f53d433de3c6db7677e0650890ae55943b1ff775e1d04d00f83 WatchSource:0}: Error finding container ed5251df6dd11f53d433de3c6db7677e0650890ae55943b1ff775e1d04d00f83: Status 404 returned error can't find the container with id ed5251df6dd11f53d433de3c6db7677e0650890ae55943b1ff775e1d04d00f83
Sep 18 20:41:43.217185 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:43.217143    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9" event={"ID":"2bd350ed-dab5-4380-ba42-8f520fff524b","Type":"ContainerStarted","Data":"05296a52dd3cd37ae3bd20e950990d5a4e2306114c55d5e90f3febe0eff78d6e"}
Sep 18 20:41:43.217362 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:43.217204    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9" event={"ID":"2bd350ed-dab5-4380-ba42-8f520fff524b","Type":"ContainerStarted","Data":"ed5251df6dd11f53d433de3c6db7677e0650890ae55943b1ff775e1d04d00f83"}
Sep 18 20:41:45.232890 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:45.232852    2581 generic.go:356] "Generic (PLEG): container finished" podID="2bd350ed-dab5-4380-ba42-8f520fff524b" containerID="05296a52dd3cd37ae3bd20e950990d5a4e2306114c55d5e90f3febe0eff78d6e" exitCode=0
Sep 18 20:41:45.232890 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:45.232871    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9" event={"ID":"2bd350ed-dab5-4380-ba42-8f520fff524b","Type":"ContainerDied","Data":"05296a52dd3cd37ae3bd20e950990d5a4e2306114c55d5e90f3febe0eff78d6e"}
Sep 18 20:41:46.241585 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:46.241548    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9" event={"ID":"2bd350ed-dab5-4380-ba42-8f520fff524b","Type":"ContainerStarted","Data":"82c11d3d0bd8c778bd3db37328f30b63bf39f5801b7036df33945cec72bc7993"}
Sep 18 20:41:46.272702 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:46.272637    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9" podStartSLOduration=1.936015601 podStartE2EDuration="4.272617431s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.33660183s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:41:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:41:46.269965186 +0000 UTC m=+3829.372253516" watchObservedRunningTime="2026-09-18 20:41:46.272617431 +0000 UTC m=+3829.374905760"
Sep 18 20:41:51.635477 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:51.635436    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd" podUID="6d658872-2a83-4c96-a39f-e00ebdf0ee91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 18 20:41:52.397244 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:52.397210    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"
Sep 18 20:41:52.397462 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:52.397277    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"
Sep 18 20:41:52.398881 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:41:52.398849    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9" podUID="2bd350ed-dab5-4380-ba42-8f520fff524b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 18 20:42:01.635570 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:42:01.635528    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd" podUID="6d658872-2a83-4c96-a39f-e00ebdf0ee91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 18 20:42:02.397889 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:42:02.397826    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9" podUID="2bd350ed-dab5-4380-ba42-8f520fff524b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 18 20:42:11.635560 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:42:11.635507    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd" podUID="6d658872-2a83-4c96-a39f-e00ebdf0ee91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 18 20:42:12.397622 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:42:12.397576    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9" podUID="2bd350ed-dab5-4380-ba42-8f520fff524b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 18 20:42:21.635723 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:42:21.635667    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd" podUID="6d658872-2a83-4c96-a39f-e00ebdf0ee91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 18 20:42:22.397758 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:42:22.397704    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9" podUID="2bd350ed-dab5-4380-ba42-8f520fff524b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 18 20:42:31.635580 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:42:31.635527    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd" podUID="6d658872-2a83-4c96-a39f-e00ebdf0ee91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 18 20:42:32.397628 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:42:32.397582    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9" podUID="2bd350ed-dab5-4380-ba42-8f520fff524b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 18 20:42:41.635718 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:42:41.635664    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd" podUID="6d658872-2a83-4c96-a39f-e00ebdf0ee91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 18 20:42:42.397622 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:42:42.397575    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9" podUID="2bd350ed-dab5-4380-ba42-8f520fff524b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 18 20:42:51.646107 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:42:51.646071    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"
Sep 18 20:42:51.654255 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:42:51.654228    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"
Sep 18 20:42:52.397179 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:42:52.397131    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9" podUID="2bd350ed-dab5-4380-ba42-8f520fff524b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 18 20:43:02.397883 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:02.397812    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9" podUID="2bd350ed-dab5-4380-ba42-8f520fff524b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 18 20:43:12.397048 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:12.396997    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9" podUID="2bd350ed-dab5-4380-ba42-8f520fff524b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 18 20:43:22.397208 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:22.397162    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9" podUID="2bd350ed-dab5-4380-ba42-8f520fff524b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 18 20:43:32.397430 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:32.397380    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9" podUID="2bd350ed-dab5-4380-ba42-8f520fff524b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 18 20:43:38.555514 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:38.555480    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd_6d658872-2a83-4c96-a39f-e00ebdf0ee91/storage-initializer/0.log"
Sep 18 20:43:38.564721 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:38.564693    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd_6d658872-2a83-4c96-a39f-e00ebdf0ee91/main/0.log"
Sep 18 20:43:39.741933 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:39.741894    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"]
Sep 18 20:43:42.397681 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:42.397627    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9" podUID="2bd350ed-dab5-4380-ba42-8f520fff524b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 18 20:43:52.403310 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:52.403214    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9" podUID="2bd350ed-dab5-4380-ba42-8f520fff524b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 18 20:43:54.769691 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:54.769630    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd" podUID="6d658872-2a83-4c96-a39f-e00ebdf0ee91" containerName="main" containerID="cri-o://22855727c863488f3dda8a1af4187fa0505c1945098f248b66ad5ae4054dff12" gracePeriod=45
Sep 18 20:43:55.667890 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:55.667865    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"
Sep 18 20:43:55.829619 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:55.829575    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d658872-2a83-4c96-a39f-e00ebdf0ee91-cabundle-cert\") pod \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\" (UID: \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\") "
Sep 18 20:43:55.830254 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:55.829648    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d658872-2a83-4c96-a39f-e00ebdf0ee91-dshm\") pod \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\" (UID: \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\") "
Sep 18 20:43:55.830254 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:55.829696    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d658872-2a83-4c96-a39f-e00ebdf0ee91-tmp-dir\") pod \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\" (UID: \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\") "
Sep 18 20:43:55.830254 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:55.829738    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d658872-2a83-4c96-a39f-e00ebdf0ee91-model-cache\") pod \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\" (UID: \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\") "
Sep 18 20:43:55.830254 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:55.829797    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d658872-2a83-4c96-a39f-e00ebdf0ee91-kserve-provision-location\") pod \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\" (UID: \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\") "
Sep 18 20:43:55.830254 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:55.829827    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d658872-2a83-4c96-a39f-e00ebdf0ee91-home\") pod \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\" (UID: \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\") "
Sep 18 20:43:55.830254 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:55.829917    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d658872-2a83-4c96-a39f-e00ebdf0ee91-tls-certs\") pod \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\" (UID: \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\") "
Sep 18 20:43:55.830254 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:55.829953    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9phjf\" (UniqueName: \"kubernetes.io/projected/6d658872-2a83-4c96-a39f-e00ebdf0ee91-kube-api-access-9phjf\") pod \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\" (UID: \"6d658872-2a83-4c96-a39f-e00ebdf0ee91\") "
Sep 18 20:43:55.830254 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:55.830116    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d658872-2a83-4c96-a39f-e00ebdf0ee91-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6d658872-2a83-4c96-a39f-e00ebdf0ee91" (UID: "6d658872-2a83-4c96-a39f-e00ebdf0ee91"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:43:55.830856 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:55.830464    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d658872-2a83-4c96-a39f-e00ebdf0ee91-cabundle-cert\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:43:55.831025 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:55.830993    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d658872-2a83-4c96-a39f-e00ebdf0ee91-home" (OuterVolumeSpecName: "home") pod "6d658872-2a83-4c96-a39f-e00ebdf0ee91" (UID: "6d658872-2a83-4c96-a39f-e00ebdf0ee91"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:43:55.831118 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:55.831080    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d658872-2a83-4c96-a39f-e00ebdf0ee91-model-cache" (OuterVolumeSpecName: "model-cache") pod "6d658872-2a83-4c96-a39f-e00ebdf0ee91" (UID: "6d658872-2a83-4c96-a39f-e00ebdf0ee91"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:43:55.845383 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:55.845130    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d658872-2a83-4c96-a39f-e00ebdf0ee91-dshm" (OuterVolumeSpecName: "dshm") pod "6d658872-2a83-4c96-a39f-e00ebdf0ee91" (UID: "6d658872-2a83-4c96-a39f-e00ebdf0ee91"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:43:55.848540 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:55.848497    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d658872-2a83-4c96-a39f-e00ebdf0ee91-kube-api-access-9phjf" (OuterVolumeSpecName: "kube-api-access-9phjf") pod "6d658872-2a83-4c96-a39f-e00ebdf0ee91" (UID: "6d658872-2a83-4c96-a39f-e00ebdf0ee91"). InnerVolumeSpecName "kube-api-access-9phjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:43:55.850033 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:55.849994    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d658872-2a83-4c96-a39f-e00ebdf0ee91-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6d658872-2a83-4c96-a39f-e00ebdf0ee91" (UID: "6d658872-2a83-4c96-a39f-e00ebdf0ee91"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:43:55.850992 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:55.850970    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d658872-2a83-4c96-a39f-e00ebdf0ee91-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6d658872-2a83-4c96-a39f-e00ebdf0ee91" (UID: "6d658872-2a83-4c96-a39f-e00ebdf0ee91"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:43:55.864260 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:55.864233    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d658872-2a83-4c96-a39f-e00ebdf0ee91-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6d658872-2a83-4c96-a39f-e00ebdf0ee91" (UID: "6d658872-2a83-4c96-a39f-e00ebdf0ee91"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:43:55.931125 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:55.931085    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d658872-2a83-4c96-a39f-e00ebdf0ee91-dshm\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:43:55.931125 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:55.931115    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d658872-2a83-4c96-a39f-e00ebdf0ee91-tmp-dir\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:43:55.931125 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:55.931124    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d658872-2a83-4c96-a39f-e00ebdf0ee91-model-cache\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:43:55.931125 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:55.931134    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d658872-2a83-4c96-a39f-e00ebdf0ee91-kserve-provision-location\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:43:55.931381 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:55.931144    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d658872-2a83-4c96-a39f-e00ebdf0ee91-home\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:43:55.931381 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:55.931153    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d658872-2a83-4c96-a39f-e00ebdf0ee91-tls-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:43:55.931381 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:55.931161    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9phjf\" (UniqueName: \"kubernetes.io/projected/6d658872-2a83-4c96-a39f-e00ebdf0ee91-kube-api-access-9phjf\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:43:56.003774 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:56.003739    2581 generic.go:356] "Generic (PLEG): container finished" podID="6d658872-2a83-4c96-a39f-e00ebdf0ee91" containerID="22855727c863488f3dda8a1af4187fa0505c1945098f248b66ad5ae4054dff12" exitCode=0
Sep 18 20:43:56.003996 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:56.003790    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"
Sep 18 20:43:56.003996 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:56.003822    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd" event={"ID":"6d658872-2a83-4c96-a39f-e00ebdf0ee91","Type":"ContainerDied","Data":"22855727c863488f3dda8a1af4187fa0505c1945098f248b66ad5ae4054dff12"}
Sep 18 20:43:56.003996 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:56.003902    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd" event={"ID":"6d658872-2a83-4c96-a39f-e00ebdf0ee91","Type":"ContainerDied","Data":"edf753f2dc0faa56e4a9ab9cf55fc474090cbb3479406182bbfcb4102d50f812"}
Sep 18 20:43:56.003996 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:56.003933    2581 scope.go:117] "RemoveContainer" containerID="22855727c863488f3dda8a1af4187fa0505c1945098f248b66ad5ae4054dff12"
Sep 18 20:43:56.013927 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:56.013906    2581 scope.go:117] "RemoveContainer" containerID="975ff7575c4c7cfb3670213094a70d02ed465d474e542f6fcdfe22f56dcc6c0f"
Sep 18 20:43:56.022812 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:56.022796    2581 scope.go:117] "RemoveContainer" containerID="22855727c863488f3dda8a1af4187fa0505c1945098f248b66ad5ae4054dff12"
Sep 18 20:43:56.023081 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:43:56.023055    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22855727c863488f3dda8a1af4187fa0505c1945098f248b66ad5ae4054dff12\": container with ID starting with 22855727c863488f3dda8a1af4187fa0505c1945098f248b66ad5ae4054dff12 not found: ID does not exist" containerID="22855727c863488f3dda8a1af4187fa0505c1945098f248b66ad5ae4054dff12"
Sep 18 20:43:56.023146 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:56.023095    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22855727c863488f3dda8a1af4187fa0505c1945098f248b66ad5ae4054dff12"} err="failed to get container status \"22855727c863488f3dda8a1af4187fa0505c1945098f248b66ad5ae4054dff12\": rpc error: code = NotFound desc = could not find container \"22855727c863488f3dda8a1af4187fa0505c1945098f248b66ad5ae4054dff12\": container with ID starting with 22855727c863488f3dda8a1af4187fa0505c1945098f248b66ad5ae4054dff12 not found: ID does not exist"
Sep 18 20:43:56.023146 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:56.023119    2581 scope.go:117] "RemoveContainer" containerID="975ff7575c4c7cfb3670213094a70d02ed465d474e542f6fcdfe22f56dcc6c0f"
Sep 18 20:43:56.023412 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:43:56.023394    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"975ff7575c4c7cfb3670213094a70d02ed465d474e542f6fcdfe22f56dcc6c0f\": container with ID starting with 975ff7575c4c7cfb3670213094a70d02ed465d474e542f6fcdfe22f56dcc6c0f not found: ID does not exist" containerID="975ff7575c4c7cfb3670213094a70d02ed465d474e542f6fcdfe22f56dcc6c0f"
Sep 18 20:43:56.023466 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:56.023418    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"975ff7575c4c7cfb3670213094a70d02ed465d474e542f6fcdfe22f56dcc6c0f"} err="failed to get container status \"975ff7575c4c7cfb3670213094a70d02ed465d474e542f6fcdfe22f56dcc6c0f\": rpc error: code = NotFound desc = could not find container \"975ff7575c4c7cfb3670213094a70d02ed465d474e542f6fcdfe22f56dcc6c0f\": container with ID starting with 975ff7575c4c7cfb3670213094a70d02ed465d474e542f6fcdfe22f56dcc6c0f not found: ID does not exist"
Sep 18 20:43:56.027262 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:56.027242    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"]
Sep 18 20:43:56.030791 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:56.030772    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd"]
Sep 18 20:43:56.647890 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:56.647802    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-8585b4cc67-jjwpd" podUID="6d658872-2a83-4c96-a39f-e00ebdf0ee91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 18 20:43:57.434462 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:43:57.434425    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d658872-2a83-4c96-a39f-e00ebdf0ee91" path="/var/lib/kubelet/pods/6d658872-2a83-4c96-a39f-e00ebdf0ee91/volumes"
Sep 18 20:44:02.407421 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:02.407384    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"
Sep 18 20:44:02.415054 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:02.415033    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"
Sep 18 20:44:09.205795 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.205761    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"]
Sep 18 20:44:09.206213 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.206172    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d658872-2a83-4c96-a39f-e00ebdf0ee91" containerName="storage-initializer"
Sep 18 20:44:09.206213 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.206184    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d658872-2a83-4c96-a39f-e00ebdf0ee91" containerName="storage-initializer"
Sep 18 20:44:09.206213 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.206200    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d658872-2a83-4c96-a39f-e00ebdf0ee91" containerName="main"
Sep 18 20:44:09.206213 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.206206    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d658872-2a83-4c96-a39f-e00ebdf0ee91" containerName="main"
Sep 18 20:44:09.206351 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.206271    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d658872-2a83-4c96-a39f-e00ebdf0ee91" containerName="main"
Sep 18 20:44:09.211208 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.211188    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"
Sep 18 20:44:09.213798 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.213762    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:44:09.213798 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.213785    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-ww85g\""
Sep 18 20:44:09.214151 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.213823    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 18 20:44:09.214151 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.213923    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 18 20:44:09.214151 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.214120    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 18 20:44:09.214612 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.214593    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Sep 18 20:44:09.223136 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.223113    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"]
Sep 18 20:44:09.243278 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.243257    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c63e1f4-187c-47ed-8958-35c268446584-home\") pod \"tls-verification-test-kserve-7957b984ff-kx9h9\" (UID: \"9c63e1f4-187c-47ed-8958-35c268446584\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"
Sep 18 20:44:09.243379 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.243298    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv44s\" (UniqueName: \"kubernetes.io/projected/9c63e1f4-187c-47ed-8958-35c268446584-kube-api-access-rv44s\") pod \"tls-verification-test-kserve-7957b984ff-kx9h9\" (UID: \"9c63e1f4-187c-47ed-8958-35c268446584\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"
Sep 18 20:44:09.243432 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.243390    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c63e1f4-187c-47ed-8958-35c268446584-model-cache\") pod \"tls-verification-test-kserve-7957b984ff-kx9h9\" (UID: \"9c63e1f4-187c-47ed-8958-35c268446584\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"
Sep 18 20:44:09.243432 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.243416    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c63e1f4-187c-47ed-8958-35c268446584-tls-certs\") pod \"tls-verification-test-kserve-7957b984ff-kx9h9\" (UID: \"9c63e1f4-187c-47ed-8958-35c268446584\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"
Sep 18 20:44:09.243528 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.243438    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c63e1f4-187c-47ed-8958-35c268446584-tmp-dir\") pod \"tls-verification-test-kserve-7957b984ff-kx9h9\" (UID: \"9c63e1f4-187c-47ed-8958-35c268446584\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"
Sep 18 20:44:09.243528 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.243504    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c63e1f4-187c-47ed-8958-35c268446584-cabundle-cert\") pod \"tls-verification-test-kserve-7957b984ff-kx9h9\" (UID: \"9c63e1f4-187c-47ed-8958-35c268446584\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"
Sep 18 20:44:09.243528 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.243524    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c63e1f4-187c-47ed-8958-35c268446584-dshm\") pod \"tls-verification-test-kserve-7957b984ff-kx9h9\" (UID: \"9c63e1f4-187c-47ed-8958-35c268446584\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"
Sep 18 20:44:09.243656 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.243538    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c63e1f4-187c-47ed-8958-35c268446584-kserve-provision-location\") pod \"tls-verification-test-kserve-7957b984ff-kx9h9\" (UID: \"9c63e1f4-187c-47ed-8958-35c268446584\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"
Sep 18 20:44:09.344622 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.344592    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c63e1f4-187c-47ed-8958-35c268446584-model-cache\") pod \"tls-verification-test-kserve-7957b984ff-kx9h9\" (UID: \"9c63e1f4-187c-47ed-8958-35c268446584\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"
Sep 18 20:44:09.344622 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.344624    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c63e1f4-187c-47ed-8958-35c268446584-tls-certs\") pod \"tls-verification-test-kserve-7957b984ff-kx9h9\" (UID: \"9c63e1f4-187c-47ed-8958-35c268446584\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"
Sep 18 20:44:09.344886 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.344644    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c63e1f4-187c-47ed-8958-35c268446584-tmp-dir\") pod \"tls-verification-test-kserve-7957b984ff-kx9h9\" (UID: \"9c63e1f4-187c-47ed-8958-35c268446584\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"
Sep 18 20:44:09.344966 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.344913    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c63e1f4-187c-47ed-8958-35c268446584-cabundle-cert\") pod \"tls-verification-test-kserve-7957b984ff-kx9h9\" (UID: \"9c63e1f4-187c-47ed-8958-35c268446584\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"
Sep 18 20:44:09.344966 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.344952    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c63e1f4-187c-47ed-8958-35c268446584-dshm\") pod \"tls-verification-test-kserve-7957b984ff-kx9h9\" (UID: \"9c63e1f4-187c-47ed-8958-35c268446584\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"
Sep 18 20:44:09.345081 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.344980    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c63e1f4-187c-47ed-8958-35c268446584-kserve-provision-location\") pod \"tls-verification-test-kserve-7957b984ff-kx9h9\" (UID: \"9c63e1f4-187c-47ed-8958-35c268446584\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"
Sep 18 20:44:09.345081 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.345039    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c63e1f4-187c-47ed-8958-35c268446584-home\") pod \"tls-verification-test-kserve-7957b984ff-kx9h9\" (UID: \"9c63e1f4-187c-47ed-8958-35c268446584\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"
Sep 18 20:44:09.345081 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.345060    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c63e1f4-187c-47ed-8958-35c268446584-model-cache\") pod \"tls-verification-test-kserve-7957b984ff-kx9h9\" (UID: \"9c63e1f4-187c-47ed-8958-35c268446584\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"
Sep 18 20:44:09.345081 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.345077    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv44s\" (UniqueName: \"kubernetes.io/projected/9c63e1f4-187c-47ed-8958-35c268446584-kube-api-access-rv44s\") pod \"tls-verification-test-kserve-7957b984ff-kx9h9\" (UID: \"9c63e1f4-187c-47ed-8958-35c268446584\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"
Sep 18 20:44:09.345252 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.345098    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c63e1f4-187c-47ed-8958-35c268446584-tmp-dir\") pod \"tls-verification-test-kserve-7957b984ff-kx9h9\" (UID: \"9c63e1f4-187c-47ed-8958-35c268446584\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"
Sep 18 20:44:09.345423 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.345399    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c63e1f4-187c-47ed-8958-35c268446584-home\") pod \"tls-verification-test-kserve-7957b984ff-kx9h9\" (UID: \"9c63e1f4-187c-47ed-8958-35c268446584\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"
Sep 18 20:44:09.345609 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.345578    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c63e1f4-187c-47ed-8958-35c268446584-cabundle-cert\") pod \"tls-verification-test-kserve-7957b984ff-kx9h9\" (UID: \"9c63e1f4-187c-47ed-8958-35c268446584\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"
Sep 18 20:44:09.345792 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.345682    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c63e1f4-187c-47ed-8958-35c268446584-kserve-provision-location\") pod \"tls-verification-test-kserve-7957b984ff-kx9h9\" (UID: \"9c63e1f4-187c-47ed-8958-35c268446584\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"
Sep 18 20:44:09.347293 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.347264    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c63e1f4-187c-47ed-8958-35c268446584-dshm\") pod \"tls-verification-test-kserve-7957b984ff-kx9h9\" (UID: \"9c63e1f4-187c-47ed-8958-35c268446584\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"
Sep 18 20:44:09.347393 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.347334    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c63e1f4-187c-47ed-8958-35c268446584-tls-certs\") pod \"tls-verification-test-kserve-7957b984ff-kx9h9\" (UID: \"9c63e1f4-187c-47ed-8958-35c268446584\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"
Sep 18 20:44:09.353203 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.353179    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv44s\" (UniqueName: \"kubernetes.io/projected/9c63e1f4-187c-47ed-8958-35c268446584-kube-api-access-rv44s\") pod \"tls-verification-test-kserve-7957b984ff-kx9h9\" (UID: \"9c63e1f4-187c-47ed-8958-35c268446584\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"
Sep 18 20:44:09.522670 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.522633    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"
Sep 18 20:44:09.800171 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:09.800062    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"]
Sep 18 20:44:09.802707 ip-10-0-143-134 kubenswrapper[2581]: W0918 20:44:09.802674    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c63e1f4_187c_47ed_8958_35c268446584.slice/crio-16570fa3ce2a434c47fd1088e53a9b125760cdcda0ee44ce04c8fc3ddd95d677 WatchSource:0}: Error finding container 16570fa3ce2a434c47fd1088e53a9b125760cdcda0ee44ce04c8fc3ddd95d677: Status 404 returned error can't find the container with id 16570fa3ce2a434c47fd1088e53a9b125760cdcda0ee44ce04c8fc3ddd95d677
Sep 18 20:44:10.086108 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:10.086019    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9" event={"ID":"9c63e1f4-187c-47ed-8958-35c268446584","Type":"ContainerStarted","Data":"8476bc66c47b78609f2b226e288dd4a44113b4024026150964eae3d6d334812a"}
Sep 18 20:44:10.086108 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:10.086083    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9" event={"ID":"9c63e1f4-187c-47ed-8958-35c268446584","Type":"ContainerStarted","Data":"16570fa3ce2a434c47fd1088e53a9b125760cdcda0ee44ce04c8fc3ddd95d677"}
Sep 18 20:44:13.103749 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:13.103704    2581 generic.go:356] "Generic (PLEG): container finished" podID="9c63e1f4-187c-47ed-8958-35c268446584" containerID="8476bc66c47b78609f2b226e288dd4a44113b4024026150964eae3d6d334812a" exitCode=0
Sep 18 20:44:13.104131 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:13.103781    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9" event={"ID":"9c63e1f4-187c-47ed-8958-35c268446584","Type":"ContainerDied","Data":"8476bc66c47b78609f2b226e288dd4a44113b4024026150964eae3d6d334812a"}
Sep 18 20:44:14.113347 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:14.113302    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9" event={"ID":"9c63e1f4-187c-47ed-8958-35c268446584","Type":"ContainerStarted","Data":"415660d627f79c105dfc0a673fd3db7d175015f72298a324d7d82eac1bf9b8fb"}
Sep 18 20:44:14.153994 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:14.153896    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9" podStartSLOduration=2.918316232 podStartE2EDuration="5.153870777s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.235554545s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:44:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:44:14.152100063 +0000 UTC m=+3977.254388405" watchObservedRunningTime="2026-09-18 20:44:14.153870777 +0000 UTC m=+3977.256159121"
Sep 18 20:44:14.523296 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:14.523268    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"
Sep 18 20:44:19.523066 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:19.523015    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"
Sep 18 20:44:19.525104 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:19.525067    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9" podUID="9c63e1f4-187c-47ed-8958-35c268446584" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 18 20:44:21.567303 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:21.567263    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"]
Sep 18 20:44:29.523228 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:29.523184    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9" podUID="9c63e1f4-187c-47ed-8958-35c268446584" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 18 20:44:36.598273 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:36.598206    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9" podUID="2bd350ed-dab5-4380-ba42-8f520fff524b" containerName="main" containerID="cri-o://82c11d3d0bd8c778bd3db37328f30b63bf39f5801b7036df33945cec72bc7993" gracePeriod=45
Sep 18 20:44:37.257973 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:37.257942    2581 generic.go:356] "Generic (PLEG): container finished" podID="2bd350ed-dab5-4380-ba42-8f520fff524b" containerID="82c11d3d0bd8c778bd3db37328f30b63bf39f5801b7036df33945cec72bc7993" exitCode=0
Sep 18 20:44:37.258151 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:37.258016    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9" event={"ID":"2bd350ed-dab5-4380-ba42-8f520fff524b","Type":"ContainerDied","Data":"82c11d3d0bd8c778bd3db37328f30b63bf39f5801b7036df33945cec72bc7993"}
Sep 18 20:44:37.358528 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:37.358507    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"
Sep 18 20:44:37.513016 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:37.512929    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bd350ed-dab5-4380-ba42-8f520fff524b-kserve-provision-location\") pod \"2bd350ed-dab5-4380-ba42-8f520fff524b\" (UID: \"2bd350ed-dab5-4380-ba42-8f520fff524b\") "
Sep 18 20:44:37.513016 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:37.512964    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bd350ed-dab5-4380-ba42-8f520fff524b-tmp-dir\") pod \"2bd350ed-dab5-4380-ba42-8f520fff524b\" (UID: \"2bd350ed-dab5-4380-ba42-8f520fff524b\") "
Sep 18 20:44:37.513016 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:37.513013    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bd350ed-dab5-4380-ba42-8f520fff524b-dshm\") pod \"2bd350ed-dab5-4380-ba42-8f520fff524b\" (UID: \"2bd350ed-dab5-4380-ba42-8f520fff524b\") "
Sep 18 20:44:37.513306 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:37.513028    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bd350ed-dab5-4380-ba42-8f520fff524b-home\") pod \"2bd350ed-dab5-4380-ba42-8f520fff524b\" (UID: \"2bd350ed-dab5-4380-ba42-8f520fff524b\") "
Sep 18 20:44:37.513306 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:37.513047    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bd350ed-dab5-4380-ba42-8f520fff524b-cabundle-cert\") pod \"2bd350ed-dab5-4380-ba42-8f520fff524b\" (UID: \"2bd350ed-dab5-4380-ba42-8f520fff524b\") "
Sep 18 20:44:37.513306 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:37.513078    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bd350ed-dab5-4380-ba42-8f520fff524b-tls-certs\") pod \"2bd350ed-dab5-4380-ba42-8f520fff524b\" (UID: \"2bd350ed-dab5-4380-ba42-8f520fff524b\") "
Sep 18 20:44:37.513306 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:37.513147    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bd350ed-dab5-4380-ba42-8f520fff524b-model-cache\") pod \"2bd350ed-dab5-4380-ba42-8f520fff524b\" (UID: \"2bd350ed-dab5-4380-ba42-8f520fff524b\") "
Sep 18 20:44:37.513306 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:37.513203    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prl4d\" (UniqueName: \"kubernetes.io/projected/2bd350ed-dab5-4380-ba42-8f520fff524b-kube-api-access-prl4d\") pod \"2bd350ed-dab5-4380-ba42-8f520fff524b\" (UID: \"2bd350ed-dab5-4380-ba42-8f520fff524b\") "
Sep 18 20:44:37.513628 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:37.513489    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd350ed-dab5-4380-ba42-8f520fff524b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2bd350ed-dab5-4380-ba42-8f520fff524b" (UID: "2bd350ed-dab5-4380-ba42-8f520fff524b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:44:37.513796 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:37.513765    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd350ed-dab5-4380-ba42-8f520fff524b-model-cache" (OuterVolumeSpecName: "model-cache") pod "2bd350ed-dab5-4380-ba42-8f520fff524b" (UID: "2bd350ed-dab5-4380-ba42-8f520fff524b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:44:37.513899 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:37.513804    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd350ed-dab5-4380-ba42-8f520fff524b-home" (OuterVolumeSpecName: "home") pod "2bd350ed-dab5-4380-ba42-8f520fff524b" (UID: "2bd350ed-dab5-4380-ba42-8f520fff524b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:44:37.515642 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:37.515614    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd350ed-dab5-4380-ba42-8f520fff524b-dshm" (OuterVolumeSpecName: "dshm") pod "2bd350ed-dab5-4380-ba42-8f520fff524b" (UID: "2bd350ed-dab5-4380-ba42-8f520fff524b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:44:37.515944 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:37.515920    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bd350ed-dab5-4380-ba42-8f520fff524b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2bd350ed-dab5-4380-ba42-8f520fff524b" (UID: "2bd350ed-dab5-4380-ba42-8f520fff524b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:44:37.516205 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:37.516185    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd350ed-dab5-4380-ba42-8f520fff524b-kube-api-access-prl4d" (OuterVolumeSpecName: "kube-api-access-prl4d") pod "2bd350ed-dab5-4380-ba42-8f520fff524b" (UID: "2bd350ed-dab5-4380-ba42-8f520fff524b"). InnerVolumeSpecName "kube-api-access-prl4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:44:37.525664 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:37.525641    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd350ed-dab5-4380-ba42-8f520fff524b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2bd350ed-dab5-4380-ba42-8f520fff524b" (UID: "2bd350ed-dab5-4380-ba42-8f520fff524b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:44:37.551379 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:37.551344    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd350ed-dab5-4380-ba42-8f520fff524b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2bd350ed-dab5-4380-ba42-8f520fff524b" (UID: "2bd350ed-dab5-4380-ba42-8f520fff524b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:44:37.614726 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:37.614685    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bd350ed-dab5-4380-ba42-8f520fff524b-tmp-dir\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:44:37.614726 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:37.614729    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bd350ed-dab5-4380-ba42-8f520fff524b-dshm\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:44:37.615149 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:37.614741    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bd350ed-dab5-4380-ba42-8f520fff524b-home\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:44:37.615149 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:37.614754    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bd350ed-dab5-4380-ba42-8f520fff524b-cabundle-cert\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:44:37.615149 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:37.614766    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bd350ed-dab5-4380-ba42-8f520fff524b-tls-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:44:37.615149 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:37.614778    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bd350ed-dab5-4380-ba42-8f520fff524b-model-cache\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:44:37.615149 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:37.614793    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prl4d\" (UniqueName: \"kubernetes.io/projected/2bd350ed-dab5-4380-ba42-8f520fff524b-kube-api-access-prl4d\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:44:37.615149 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:37.614806    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bd350ed-dab5-4380-ba42-8f520fff524b-kserve-provision-location\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:44:38.265354 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:38.265315    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"
Sep 18 20:44:38.265354 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:38.265345    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9" event={"ID":"2bd350ed-dab5-4380-ba42-8f520fff524b","Type":"ContainerDied","Data":"ed5251df6dd11f53d433de3c6db7677e0650890ae55943b1ff775e1d04d00f83"}
Sep 18 20:44:38.265637 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:38.265406    2581 scope.go:117] "RemoveContainer" containerID="82c11d3d0bd8c778bd3db37328f30b63bf39f5801b7036df33945cec72bc7993"
Sep 18 20:44:38.284308 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:38.284267    2581 scope.go:117] "RemoveContainer" containerID="05296a52dd3cd37ae3bd20e950990d5a4e2306114c55d5e90f3febe0eff78d6e"
Sep 18 20:44:38.296017 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:38.295993    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"]
Sep 18 20:44:38.303496 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:38.303472    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-65tjxt9"]
Sep 18 20:44:39.435098 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:39.435065    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bd350ed-dab5-4380-ba42-8f520fff524b" path="/var/lib/kubelet/pods/2bd350ed-dab5-4380-ba42-8f520fff524b/volumes"
Sep 18 20:44:39.523418 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:39.523377    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9" podUID="9c63e1f4-187c-47ed-8958-35c268446584" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 18 20:44:49.523319 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:49.523269    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9" podUID="9c63e1f4-187c-47ed-8958-35c268446584" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 18 20:44:59.523086 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:44:59.523036    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9" podUID="9c63e1f4-187c-47ed-8958-35c268446584" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 18 20:45:09.523206 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:09.523160    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9" podUID="9c63e1f4-187c-47ed-8958-35c268446584" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 18 20:45:19.523856 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:19.523732    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9" podUID="9c63e1f4-187c-47ed-8958-35c268446584" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 18 20:45:24.500485 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.500446    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"]
Sep 18 20:45:24.500876 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.500821    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bd350ed-dab5-4380-ba42-8f520fff524b" containerName="main"
Sep 18 20:45:24.500876 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.500856    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bd350ed-dab5-4380-ba42-8f520fff524b" containerName="main"
Sep 18 20:45:24.500876 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.500876    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bd350ed-dab5-4380-ba42-8f520fff524b" containerName="storage-initializer"
Sep 18 20:45:24.500990 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.500885    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bd350ed-dab5-4380-ba42-8f520fff524b" containerName="storage-initializer"
Sep 18 20:45:24.500990 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.500956    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bd350ed-dab5-4380-ba42-8f520fff524b" containerName="main"
Sep 18 20:45:24.504277 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.504248    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"
Sep 18 20:45:24.506932 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.506909    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 18 20:45:24.507066 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.506991    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 18 20:45:24.507066 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.506995    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 18 20:45:24.507351 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.507331    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 18 20:45:24.507725 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.507710    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-zt8ws\""
Sep 18 20:45:24.507825 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.507717    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:45:24.525522 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.525497    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"]
Sep 18 20:45:24.633816 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.633786    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7\" (UID: \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"
Sep 18 20:45:24.633816 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.633822    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4hzs\" (UniqueName: \"kubernetes.io/projected/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-kube-api-access-b4hzs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7\" (UID: \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"
Sep 18 20:45:24.634061 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.633870    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7\" (UID: \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"
Sep 18 20:45:24.634061 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.633886    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7\" (UID: \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"
Sep 18 20:45:24.634061 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.633905    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7\" (UID: \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"
Sep 18 20:45:24.634061 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.633925    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7\" (UID: \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"
Sep 18 20:45:24.634061 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.633991    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7\" (UID: \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"
Sep 18 20:45:24.634061 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.634025    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7\" (UID: \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"
Sep 18 20:45:24.734646 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.734616    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7\" (UID: \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"
Sep 18 20:45:24.734817 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.734679    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7\" (UID: \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"
Sep 18 20:45:24.734817 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.734703    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4hzs\" (UniqueName: \"kubernetes.io/projected/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-kube-api-access-b4hzs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7\" (UID: \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"
Sep 18 20:45:24.734817 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.734731    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7\" (UID: \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"
Sep 18 20:45:24.734817 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.734752    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7\" (UID: \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"
Sep 18 20:45:24.734817 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.734773    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7\" (UID: \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"
Sep 18 20:45:24.734817 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.734803    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7\" (UID: \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"
Sep 18 20:45:24.735146 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.734866    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7\" (UID: \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"
Sep 18 20:45:24.735146 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.735092    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7\" (UID: \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"
Sep 18 20:45:24.735146 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.735111    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7\" (UID: \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"
Sep 18 20:45:24.735280 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.735262    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7\" (UID: \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"
Sep 18 20:45:24.735387 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.735365    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7\" (UID: \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"
Sep 18 20:45:24.735680 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.735662    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7\" (UID: \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"
Sep 18 20:45:24.737611 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.737581    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7\" (UID: \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"
Sep 18 20:45:24.737915 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.737895    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7\" (UID: \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"
Sep 18 20:45:24.746697 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.746675    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4hzs\" (UniqueName: \"kubernetes.io/projected/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-kube-api-access-b4hzs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7\" (UID: \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"
Sep 18 20:45:24.816450 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:24.816425    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"
Sep 18 20:45:25.097792 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:25.097760    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"]
Sep 18 20:45:25.099114 ip-10-0-143-134 kubenswrapper[2581]: W0918 20:45:25.099088    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb922cb2d_f55e_4e93_ae25_6b4b646ef4fe.slice/crio-e2a136656f1ffeccb6320282a87e535891530fa3f233a4e68c04780f91f17780 WatchSource:0}: Error finding container e2a136656f1ffeccb6320282a87e535891530fa3f233a4e68c04780f91f17780: Status 404 returned error can't find the container with id e2a136656f1ffeccb6320282a87e535891530fa3f233a4e68c04780f91f17780
Sep 18 20:45:25.527682 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:25.527642    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7" event={"ID":"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe","Type":"ContainerStarted","Data":"9e28d36322b7b0f8bea9d10ac136cac06f45f2a1ef599fcdca36be905c5508c7"}
Sep 18 20:45:25.528197 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:25.527717    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7" event={"ID":"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe","Type":"ContainerStarted","Data":"e2a136656f1ffeccb6320282a87e535891530fa3f233a4e68c04780f91f17780"}
Sep 18 20:45:27.542996 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:27.542958    2581 generic.go:356] "Generic (PLEG): container finished" podID="b922cb2d-f55e-4e93-ae25-6b4b646ef4fe" containerID="9e28d36322b7b0f8bea9d10ac136cac06f45f2a1ef599fcdca36be905c5508c7" exitCode=0
Sep 18 20:45:27.543452 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:27.543019    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7" event={"ID":"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe","Type":"ContainerDied","Data":"9e28d36322b7b0f8bea9d10ac136cac06f45f2a1ef599fcdca36be905c5508c7"}
Sep 18 20:45:28.556720 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:28.556685    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7" event={"ID":"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe","Type":"ContainerStarted","Data":"6513824bfb8c17b882d1815463cd1790e531cf3120eb43a6cc11defa849f9c6d"}
Sep 18 20:45:28.585069 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:28.585015    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7" podStartSLOduration=2.275257039 podStartE2EDuration="4.584999564s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.309742525s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:45:28.581692402 +0000 UTC m=+4051.683980729" watchObservedRunningTime="2026-09-18 20:45:28.584999564 +0000 UTC m=+4051.687287891"
Sep 18 20:45:29.523468 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:29.523422    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9" podUID="9c63e1f4-187c-47ed-8958-35c268446584" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 18 20:45:34.816767 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:34.816724    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"
Sep 18 20:45:34.817260 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:34.816809    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"
Sep 18 20:45:34.818320 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:34.818284    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7" podUID="b922cb2d-f55e-4e93-ae25-6b4b646ef4fe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 18 20:45:39.523240 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:39.523191    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9" podUID="9c63e1f4-187c-47ed-8958-35c268446584" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 18 20:45:44.817436 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:44.817389    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7" podUID="b922cb2d-f55e-4e93-ae25-6b4b646ef4fe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 18 20:45:49.524120 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:49.524071    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9" podUID="9c63e1f4-187c-47ed-8958-35c268446584" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 18 20:45:54.817268 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:54.817223    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7" podUID="b922cb2d-f55e-4e93-ae25-6b4b646ef4fe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 18 20:45:59.523163 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:45:59.523115    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9" podUID="9c63e1f4-187c-47ed-8958-35c268446584" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 18 20:46:04.817423 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:04.817369    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7" podUID="b922cb2d-f55e-4e93-ae25-6b4b646ef4fe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 18 20:46:09.533846 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:09.533811    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"
Sep 18 20:46:09.541713 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:09.541686    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"
Sep 18 20:46:14.554269 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:14.554232    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"]
Sep 18 20:46:14.817501 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:14.817388    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7" podUID="b922cb2d-f55e-4e93-ae25-6b4b646ef4fe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 18 20:46:24.816885 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:24.816819    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7" podUID="b922cb2d-f55e-4e93-ae25-6b4b646ef4fe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 18 20:46:29.577386 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:29.577340    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9" podUID="9c63e1f4-187c-47ed-8958-35c268446584" containerName="main" containerID="cri-o://415660d627f79c105dfc0a673fd3db7d175015f72298a324d7d82eac1bf9b8fb" gracePeriod=45
Sep 18 20:46:30.323164 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.323132    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"
Sep 18 20:46:30.350579 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.350552    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rv44s\" (UniqueName: \"kubernetes.io/projected/9c63e1f4-187c-47ed-8958-35c268446584-kube-api-access-rv44s\") pod \"9c63e1f4-187c-47ed-8958-35c268446584\" (UID: \"9c63e1f4-187c-47ed-8958-35c268446584\") "
Sep 18 20:46:30.350703 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.350613    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c63e1f4-187c-47ed-8958-35c268446584-cabundle-cert\") pod \"9c63e1f4-187c-47ed-8958-35c268446584\" (UID: \"9c63e1f4-187c-47ed-8958-35c268446584\") "
Sep 18 20:46:30.350703 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.350637    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c63e1f4-187c-47ed-8958-35c268446584-dshm\") pod \"9c63e1f4-187c-47ed-8958-35c268446584\" (UID: \"9c63e1f4-187c-47ed-8958-35c268446584\") "
Sep 18 20:46:30.350805 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.350757    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c63e1f4-187c-47ed-8958-35c268446584-model-cache\") pod \"9c63e1f4-187c-47ed-8958-35c268446584\" (UID: \"9c63e1f4-187c-47ed-8958-35c268446584\") "
Sep 18 20:46:30.350877 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.350813    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c63e1f4-187c-47ed-8958-35c268446584-tls-certs\") pod \"9c63e1f4-187c-47ed-8958-35c268446584\" (UID: \"9c63e1f4-187c-47ed-8958-35c268446584\") "
Sep 18 20:46:30.350939 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.350876    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c63e1f4-187c-47ed-8958-35c268446584-tmp-dir\") pod \"9c63e1f4-187c-47ed-8958-35c268446584\" (UID: \"9c63e1f4-187c-47ed-8958-35c268446584\") "
Sep 18 20:46:30.350939 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.350917    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c63e1f4-187c-47ed-8958-35c268446584-kserve-provision-location\") pod \"9c63e1f4-187c-47ed-8958-35c268446584\" (UID: \"9c63e1f4-187c-47ed-8958-35c268446584\") "
Sep 18 20:46:30.351052 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.350955    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c63e1f4-187c-47ed-8958-35c268446584-home\") pod \"9c63e1f4-187c-47ed-8958-35c268446584\" (UID: \"9c63e1f4-187c-47ed-8958-35c268446584\") "
Sep 18 20:46:30.353140 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.351101    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c63e1f4-187c-47ed-8958-35c268446584-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9c63e1f4-187c-47ed-8958-35c268446584" (UID: "9c63e1f4-187c-47ed-8958-35c268446584"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:46:30.353140 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.351495    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c63e1f4-187c-47ed-8958-35c268446584-cabundle-cert\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:46:30.353140 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.351696    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c63e1f4-187c-47ed-8958-35c268446584-home" (OuterVolumeSpecName: "home") pod "9c63e1f4-187c-47ed-8958-35c268446584" (UID: "9c63e1f4-187c-47ed-8958-35c268446584"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:46:30.353140 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.352202    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c63e1f4-187c-47ed-8958-35c268446584-model-cache" (OuterVolumeSpecName: "model-cache") pod "9c63e1f4-187c-47ed-8958-35c268446584" (UID: "9c63e1f4-187c-47ed-8958-35c268446584"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:46:30.354295 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.353769    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c63e1f4-187c-47ed-8958-35c268446584-dshm" (OuterVolumeSpecName: "dshm") pod "9c63e1f4-187c-47ed-8958-35c268446584" (UID: "9c63e1f4-187c-47ed-8958-35c268446584"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:46:30.354295 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.353871    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c63e1f4-187c-47ed-8958-35c268446584-kube-api-access-rv44s" (OuterVolumeSpecName: "kube-api-access-rv44s") pod "9c63e1f4-187c-47ed-8958-35c268446584" (UID: "9c63e1f4-187c-47ed-8958-35c268446584"). InnerVolumeSpecName "kube-api-access-rv44s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:46:30.354295 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.354167    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c63e1f4-187c-47ed-8958-35c268446584-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c63e1f4-187c-47ed-8958-35c268446584" (UID: "9c63e1f4-187c-47ed-8958-35c268446584"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:46:30.373515 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.373487    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c63e1f4-187c-47ed-8958-35c268446584-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9c63e1f4-187c-47ed-8958-35c268446584" (UID: "9c63e1f4-187c-47ed-8958-35c268446584"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:46:30.377959 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.377935    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c63e1f4-187c-47ed-8958-35c268446584-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9c63e1f4-187c-47ed-8958-35c268446584" (UID: "9c63e1f4-187c-47ed-8958-35c268446584"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:46:30.452607 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.452578    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c63e1f4-187c-47ed-8958-35c268446584-dshm\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:46:30.452607 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.452604    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c63e1f4-187c-47ed-8958-35c268446584-model-cache\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:46:30.452752 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.452614    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c63e1f4-187c-47ed-8958-35c268446584-tls-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:46:30.452752 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.452621    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c63e1f4-187c-47ed-8958-35c268446584-tmp-dir\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:46:30.452752 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.452630    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c63e1f4-187c-47ed-8958-35c268446584-kserve-provision-location\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:46:30.452752 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.452638    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c63e1f4-187c-47ed-8958-35c268446584-home\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:46:30.452752 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.452646    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv44s\" (UniqueName: \"kubernetes.io/projected/9c63e1f4-187c-47ed-8958-35c268446584-kube-api-access-rv44s\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:46:30.937201 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.937161    2581 generic.go:356] "Generic (PLEG): container finished" podID="9c63e1f4-187c-47ed-8958-35c268446584" containerID="415660d627f79c105dfc0a673fd3db7d175015f72298a324d7d82eac1bf9b8fb" exitCode=0
Sep 18 20:46:30.937201 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.937211    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"
Sep 18 20:46:30.937766 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.937256    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9" event={"ID":"9c63e1f4-187c-47ed-8958-35c268446584","Type":"ContainerDied","Data":"415660d627f79c105dfc0a673fd3db7d175015f72298a324d7d82eac1bf9b8fb"}
Sep 18 20:46:30.937766 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.937305    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9" event={"ID":"9c63e1f4-187c-47ed-8958-35c268446584","Type":"ContainerDied","Data":"16570fa3ce2a434c47fd1088e53a9b125760cdcda0ee44ce04c8fc3ddd95d677"}
Sep 18 20:46:30.937766 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.937334    2581 scope.go:117] "RemoveContainer" containerID="415660d627f79c105dfc0a673fd3db7d175015f72298a324d7d82eac1bf9b8fb"
Sep 18 20:46:30.947294 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.947273    2581 scope.go:117] "RemoveContainer" containerID="8476bc66c47b78609f2b226e288dd4a44113b4024026150964eae3d6d334812a"
Sep 18 20:46:30.963246 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.963210    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"]
Sep 18 20:46:30.964212 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.964189    2581 scope.go:117] "RemoveContainer" containerID="415660d627f79c105dfc0a673fd3db7d175015f72298a324d7d82eac1bf9b8fb"
Sep 18 20:46:30.964523 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:46:30.964503    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"415660d627f79c105dfc0a673fd3db7d175015f72298a324d7d82eac1bf9b8fb\": container with ID starting with 415660d627f79c105dfc0a673fd3db7d175015f72298a324d7d82eac1bf9b8fb not found: ID does not exist" containerID="415660d627f79c105dfc0a673fd3db7d175015f72298a324d7d82eac1bf9b8fb"
Sep 18 20:46:30.964604 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.964530    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"415660d627f79c105dfc0a673fd3db7d175015f72298a324d7d82eac1bf9b8fb"} err="failed to get container status \"415660d627f79c105dfc0a673fd3db7d175015f72298a324d7d82eac1bf9b8fb\": rpc error: code = NotFound desc = could not find container \"415660d627f79c105dfc0a673fd3db7d175015f72298a324d7d82eac1bf9b8fb\": container with ID starting with 415660d627f79c105dfc0a673fd3db7d175015f72298a324d7d82eac1bf9b8fb not found: ID does not exist"
Sep 18 20:46:30.964604 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.964547    2581 scope.go:117] "RemoveContainer" containerID="8476bc66c47b78609f2b226e288dd4a44113b4024026150964eae3d6d334812a"
Sep 18 20:46:30.964854 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:46:30.964813    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8476bc66c47b78609f2b226e288dd4a44113b4024026150964eae3d6d334812a\": container with ID starting with 8476bc66c47b78609f2b226e288dd4a44113b4024026150964eae3d6d334812a not found: ID does not exist" containerID="8476bc66c47b78609f2b226e288dd4a44113b4024026150964eae3d6d334812a"
Sep 18 20:46:30.964929 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.964857    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8476bc66c47b78609f2b226e288dd4a44113b4024026150964eae3d6d334812a"} err="failed to get container status \"8476bc66c47b78609f2b226e288dd4a44113b4024026150964eae3d6d334812a\": rpc error: code = NotFound desc = could not find container \"8476bc66c47b78609f2b226e288dd4a44113b4024026150964eae3d6d334812a\": container with ID starting with 8476bc66c47b78609f2b226e288dd4a44113b4024026150964eae3d6d334812a not found: ID does not exist"
Sep 18 20:46:30.965804 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:30.965779    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7957b984ff-kx9h9"]
Sep 18 20:46:31.433630 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:31.433583    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c63e1f4-187c-47ed-8958-35c268446584" path="/var/lib/kubelet/pods/9c63e1f4-187c-47ed-8958-35c268446584/volumes"
Sep 18 20:46:34.817375 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:34.817323    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7" podUID="b922cb2d-f55e-4e93-ae25-6b4b646ef4fe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 18 20:46:44.817545 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:44.817500    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7" podUID="b922cb2d-f55e-4e93-ae25-6b4b646ef4fe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 18 20:46:54.816933 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:46:54.816821    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7" podUID="b922cb2d-f55e-4e93-ae25-6b4b646ef4fe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 18 20:47:04.817137 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:04.817073    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7" podUID="b922cb2d-f55e-4e93-ae25-6b4b646ef4fe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 18 20:47:14.817539 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:14.817487    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7" podUID="b922cb2d-f55e-4e93-ae25-6b4b646ef4fe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 18 20:47:24.826539 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:24.826506    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"
Sep 18 20:47:24.834234 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:24.834213    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"
Sep 18 20:47:39.281898 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:39.281865    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"]
Sep 18 20:47:54.305314 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:54.305268    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7" podUID="b922cb2d-f55e-4e93-ae25-6b4b646ef4fe" containerName="main" containerID="cri-o://6513824bfb8c17b882d1815463cd1790e531cf3120eb43a6cc11defa849f9c6d" gracePeriod=45
Sep 18 20:47:54.827505 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:54.827460    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7" podUID="b922cb2d-f55e-4e93-ae25-6b4b646ef4fe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 18 20:47:54.991628 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:54.991605    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"
Sep 18 20:47:55.007366 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.007331    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-cabundle-cert\") pod \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\" (UID: \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\") "
Sep 18 20:47:55.007513 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.007396    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-tls-certs\") pod \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\" (UID: \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\") "
Sep 18 20:47:55.007513 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.007414    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-kserve-provision-location\") pod \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\" (UID: \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\") "
Sep 18 20:47:55.007513 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.007447    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-model-cache\") pod \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\" (UID: \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\") "
Sep 18 20:47:55.007513 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.007466    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-dshm\") pod \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\" (UID: \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\") "
Sep 18 20:47:55.007513 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.007486    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-tmp-dir\") pod \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\" (UID: \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\") "
Sep 18 20:47:55.007788 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.007519    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4hzs\" (UniqueName: \"kubernetes.io/projected/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-kube-api-access-b4hzs\") pod \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\" (UID: \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\") "
Sep 18 20:47:55.007788 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.007541    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-home\") pod \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\" (UID: \"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe\") "
Sep 18 20:47:55.007788 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.007736    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-model-cache" (OuterVolumeSpecName: "model-cache") pod "b922cb2d-f55e-4e93-ae25-6b4b646ef4fe" (UID: "b922cb2d-f55e-4e93-ae25-6b4b646ef4fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:47:55.007788 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.007756    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b922cb2d-f55e-4e93-ae25-6b4b646ef4fe" (UID: "b922cb2d-f55e-4e93-ae25-6b4b646ef4fe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:47:55.008023 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.007807    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-model-cache\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:47:55.008554 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.008528    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-home" (OuterVolumeSpecName: "home") pod "b922cb2d-f55e-4e93-ae25-6b4b646ef4fe" (UID: "b922cb2d-f55e-4e93-ae25-6b4b646ef4fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:47:55.010134 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.010091    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-dshm" (OuterVolumeSpecName: "dshm") pod "b922cb2d-f55e-4e93-ae25-6b4b646ef4fe" (UID: "b922cb2d-f55e-4e93-ae25-6b4b646ef4fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:47:55.010344 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.010311    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b922cb2d-f55e-4e93-ae25-6b4b646ef4fe" (UID: "b922cb2d-f55e-4e93-ae25-6b4b646ef4fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:47:55.010585 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.010558    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-kube-api-access-b4hzs" (OuterVolumeSpecName: "kube-api-access-b4hzs") pod "b922cb2d-f55e-4e93-ae25-6b4b646ef4fe" (UID: "b922cb2d-f55e-4e93-ae25-6b4b646ef4fe"). InnerVolumeSpecName "kube-api-access-b4hzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:47:55.023990 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.023962    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b922cb2d-f55e-4e93-ae25-6b4b646ef4fe" (UID: "b922cb2d-f55e-4e93-ae25-6b4b646ef4fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:47:55.038973 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.038916    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b922cb2d-f55e-4e93-ae25-6b4b646ef4fe" (UID: "b922cb2d-f55e-4e93-ae25-6b4b646ef4fe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:47:55.109133 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.109048    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-tls-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:47:55.109133 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.109077    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-kserve-provision-location\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:47:55.109133 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.109091    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-dshm\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:47:55.109133 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.109104    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-tmp-dir\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:47:55.109133 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.109114    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4hzs\" (UniqueName: \"kubernetes.io/projected/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-kube-api-access-b4hzs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:47:55.109133 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.109122    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-home\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:47:55.109133 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.109131    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe-cabundle-cert\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:47:55.396762 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.396664    2581 generic.go:356] "Generic (PLEG): container finished" podID="b922cb2d-f55e-4e93-ae25-6b4b646ef4fe" containerID="6513824bfb8c17b882d1815463cd1790e531cf3120eb43a6cc11defa849f9c6d" exitCode=0
Sep 18 20:47:55.396762 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.396710    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"
Sep 18 20:47:55.396762 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.396748    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7" event={"ID":"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe","Type":"ContainerDied","Data":"6513824bfb8c17b882d1815463cd1790e531cf3120eb43a6cc11defa849f9c6d"}
Sep 18 20:47:55.397416 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.396813    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7" event={"ID":"b922cb2d-f55e-4e93-ae25-6b4b646ef4fe","Type":"ContainerDied","Data":"e2a136656f1ffeccb6320282a87e535891530fa3f233a4e68c04780f91f17780"}
Sep 18 20:47:55.397416 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.396870    2581 scope.go:117] "RemoveContainer" containerID="6513824bfb8c17b882d1815463cd1790e531cf3120eb43a6cc11defa849f9c6d"
Sep 18 20:47:55.407059 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.407041    2581 scope.go:117] "RemoveContainer" containerID="9e28d36322b7b0f8bea9d10ac136cac06f45f2a1ef599fcdca36be905c5508c7"
Sep 18 20:47:55.415255 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.415234    2581 scope.go:117] "RemoveContainer" containerID="6513824bfb8c17b882d1815463cd1790e531cf3120eb43a6cc11defa849f9c6d"
Sep 18 20:47:55.415528 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:47:55.415511    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6513824bfb8c17b882d1815463cd1790e531cf3120eb43a6cc11defa849f9c6d\": container with ID starting with 6513824bfb8c17b882d1815463cd1790e531cf3120eb43a6cc11defa849f9c6d not found: ID does not exist" containerID="6513824bfb8c17b882d1815463cd1790e531cf3120eb43a6cc11defa849f9c6d"
Sep 18 20:47:55.415567 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.415536    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6513824bfb8c17b882d1815463cd1790e531cf3120eb43a6cc11defa849f9c6d"} err="failed to get container status \"6513824bfb8c17b882d1815463cd1790e531cf3120eb43a6cc11defa849f9c6d\": rpc error: code = NotFound desc = could not find container \"6513824bfb8c17b882d1815463cd1790e531cf3120eb43a6cc11defa849f9c6d\": container with ID starting with 6513824bfb8c17b882d1815463cd1790e531cf3120eb43a6cc11defa849f9c6d not found: ID does not exist"
Sep 18 20:47:55.415567 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.415550    2581 scope.go:117] "RemoveContainer" containerID="9e28d36322b7b0f8bea9d10ac136cac06f45f2a1ef599fcdca36be905c5508c7"
Sep 18 20:47:55.415797 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:47:55.415780    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e28d36322b7b0f8bea9d10ac136cac06f45f2a1ef599fcdca36be905c5508c7\": container with ID starting with 9e28d36322b7b0f8bea9d10ac136cac06f45f2a1ef599fcdca36be905c5508c7 not found: ID does not exist" containerID="9e28d36322b7b0f8bea9d10ac136cac06f45f2a1ef599fcdca36be905c5508c7"
Sep 18 20:47:55.415861 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.415804    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e28d36322b7b0f8bea9d10ac136cac06f45f2a1ef599fcdca36be905c5508c7"} err="failed to get container status \"9e28d36322b7b0f8bea9d10ac136cac06f45f2a1ef599fcdca36be905c5508c7\": rpc error: code = NotFound desc = could not find container \"9e28d36322b7b0f8bea9d10ac136cac06f45f2a1ef599fcdca36be905c5508c7\": container with ID starting with 9e28d36322b7b0f8bea9d10ac136cac06f45f2a1ef599fcdca36be905c5508c7 not found: ID does not exist"
Sep 18 20:47:55.420155 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.420134    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"]
Sep 18 20:47:55.428568 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.425669    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7cb4ffc88c-24px7"]
Sep 18 20:47:55.435129 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:47:55.435104    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b922cb2d-f55e-4e93-ae25-6b4b646ef4fe" path="/var/lib/kubelet/pods/b922cb2d-f55e-4e93-ae25-6b4b646ef4fe/volumes"
Sep 18 20:48:45.122348 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.122311    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"]
Sep 18 20:48:45.122942 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.122699    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c63e1f4-187c-47ed-8958-35c268446584" containerName="storage-initializer"
Sep 18 20:48:45.122942 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.122712    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c63e1f4-187c-47ed-8958-35c268446584" containerName="storage-initializer"
Sep 18 20:48:45.122942 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.122723    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c63e1f4-187c-47ed-8958-35c268446584" containerName="main"
Sep 18 20:48:45.122942 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.122729    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c63e1f4-187c-47ed-8958-35c268446584" containerName="main"
Sep 18 20:48:45.122942 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.122738    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b922cb2d-f55e-4e93-ae25-6b4b646ef4fe" containerName="storage-initializer"
Sep 18 20:48:45.122942 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.122744    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b922cb2d-f55e-4e93-ae25-6b4b646ef4fe" containerName="storage-initializer"
Sep 18 20:48:45.122942 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.122761    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b922cb2d-f55e-4e93-ae25-6b4b646ef4fe" containerName="main"
Sep 18 20:48:45.122942 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.122767    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b922cb2d-f55e-4e93-ae25-6b4b646ef4fe" containerName="main"
Sep 18 20:48:45.122942 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.122827    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b922cb2d-f55e-4e93-ae25-6b4b646ef4fe" containerName="main"
Sep 18 20:48:45.122942 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.122848    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c63e1f4-187c-47ed-8958-35c268446584" containerName="main"
Sep 18 20:48:45.125249 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.125232    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:48:45.127733 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.127702    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 18 20:48:45.128221 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.128207    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-4lw4c\""
Sep 18 20:48:45.128702 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.128682    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 18 20:48:45.128702 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.128696    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-x598c\""
Sep 18 20:48:45.128891 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.128766    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 18 20:48:45.128987 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.128969    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:48:45.129077 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.128974    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 18 20:48:45.145485 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.145462    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"]
Sep 18 20:48:45.149207 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.149174    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41765192-e877-4448-b735-4ca110ed6a7f-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-7958899789-gp9g7\" (UID: \"41765192-e877-4448-b735-4ca110ed6a7f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:48:45.157434 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.157407    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x92n\" (UniqueName: \"kubernetes.io/projected/41765192-e877-4448-b735-4ca110ed6a7f-kube-api-access-5x92n\") pod \"router-with-refs-pd-test-kserve-7958899789-gp9g7\" (UID: \"41765192-e877-4448-b735-4ca110ed6a7f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:48:45.157561 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.157481    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41765192-e877-4448-b735-4ca110ed6a7f-dshm\") pod \"router-with-refs-pd-test-kserve-7958899789-gp9g7\" (UID: \"41765192-e877-4448-b735-4ca110ed6a7f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:48:45.157561 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.157520    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41765192-e877-4448-b735-4ca110ed6a7f-tmp-dir\") pod \"router-with-refs-pd-test-kserve-7958899789-gp9g7\" (UID: \"41765192-e877-4448-b735-4ca110ed6a7f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:48:45.157685 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.157558    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41765192-e877-4448-b735-4ca110ed6a7f-tls-certs\") pod \"router-with-refs-pd-test-kserve-7958899789-gp9g7\" (UID: \"41765192-e877-4448-b735-4ca110ed6a7f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:48:45.157913 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.157887    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41765192-e877-4448-b735-4ca110ed6a7f-model-cache\") pod \"router-with-refs-pd-test-kserve-7958899789-gp9g7\" (UID: \"41765192-e877-4448-b735-4ca110ed6a7f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:48:45.158022 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.157920    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41765192-e877-4448-b735-4ca110ed6a7f-home\") pod \"router-with-refs-pd-test-kserve-7958899789-gp9g7\" (UID: \"41765192-e877-4448-b735-4ca110ed6a7f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:48:45.158022 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.157952    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41765192-e877-4448-b735-4ca110ed6a7f-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-7958899789-gp9g7\" (UID: \"41765192-e877-4448-b735-4ca110ed6a7f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:48:45.259251 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.259217    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41765192-e877-4448-b735-4ca110ed6a7f-model-cache\") pod \"router-with-refs-pd-test-kserve-7958899789-gp9g7\" (UID: \"41765192-e877-4448-b735-4ca110ed6a7f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:48:45.259251 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.259251    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41765192-e877-4448-b735-4ca110ed6a7f-home\") pod \"router-with-refs-pd-test-kserve-7958899789-gp9g7\" (UID: \"41765192-e877-4448-b735-4ca110ed6a7f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:48:45.259478 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.259271    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41765192-e877-4448-b735-4ca110ed6a7f-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-7958899789-gp9g7\" (UID: \"41765192-e877-4448-b735-4ca110ed6a7f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:48:45.259478 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.259294    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41765192-e877-4448-b735-4ca110ed6a7f-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-7958899789-gp9g7\" (UID: \"41765192-e877-4448-b735-4ca110ed6a7f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:48:45.259478 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.259315    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x92n\" (UniqueName: \"kubernetes.io/projected/41765192-e877-4448-b735-4ca110ed6a7f-kube-api-access-5x92n\") pod \"router-with-refs-pd-test-kserve-7958899789-gp9g7\" (UID: \"41765192-e877-4448-b735-4ca110ed6a7f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:48:45.259478 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.259341    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41765192-e877-4448-b735-4ca110ed6a7f-dshm\") pod \"router-with-refs-pd-test-kserve-7958899789-gp9g7\" (UID: \"41765192-e877-4448-b735-4ca110ed6a7f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:48:45.259478 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.259358    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41765192-e877-4448-b735-4ca110ed6a7f-tmp-dir\") pod \"router-with-refs-pd-test-kserve-7958899789-gp9g7\" (UID: \"41765192-e877-4448-b735-4ca110ed6a7f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:48:45.259478 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.259376    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41765192-e877-4448-b735-4ca110ed6a7f-tls-certs\") pod \"router-with-refs-pd-test-kserve-7958899789-gp9g7\" (UID: \"41765192-e877-4448-b735-4ca110ed6a7f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:48:45.259796 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.259740    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41765192-e877-4448-b735-4ca110ed6a7f-model-cache\") pod \"router-with-refs-pd-test-kserve-7958899789-gp9g7\" (UID: \"41765192-e877-4448-b735-4ca110ed6a7f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:48:45.259875 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.259798    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41765192-e877-4448-b735-4ca110ed6a7f-home\") pod \"router-with-refs-pd-test-kserve-7958899789-gp9g7\" (UID: \"41765192-e877-4448-b735-4ca110ed6a7f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:48:45.259875 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.259862    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41765192-e877-4448-b735-4ca110ed6a7f-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-7958899789-gp9g7\" (UID: \"41765192-e877-4448-b735-4ca110ed6a7f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:48:45.259954 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.259897    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41765192-e877-4448-b735-4ca110ed6a7f-tmp-dir\") pod \"router-with-refs-pd-test-kserve-7958899789-gp9g7\" (UID: \"41765192-e877-4448-b735-4ca110ed6a7f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:48:45.260023 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.260004    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41765192-e877-4448-b735-4ca110ed6a7f-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-7958899789-gp9g7\" (UID: \"41765192-e877-4448-b735-4ca110ed6a7f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:48:45.261817 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.261798    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41765192-e877-4448-b735-4ca110ed6a7f-dshm\") pod \"router-with-refs-pd-test-kserve-7958899789-gp9g7\" (UID: \"41765192-e877-4448-b735-4ca110ed6a7f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:48:45.262012 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.261996    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41765192-e877-4448-b735-4ca110ed6a7f-tls-certs\") pod \"router-with-refs-pd-test-kserve-7958899789-gp9g7\" (UID: \"41765192-e877-4448-b735-4ca110ed6a7f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:48:45.268211 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.268190    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x92n\" (UniqueName: \"kubernetes.io/projected/41765192-e877-4448-b735-4ca110ed6a7f-kube-api-access-5x92n\") pod \"router-with-refs-pd-test-kserve-7958899789-gp9g7\" (UID: \"41765192-e877-4448-b735-4ca110ed6a7f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:48:45.435008 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.434926    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:48:45.727214 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:45.727187    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"]
Sep 18 20:48:45.727898 ip-10-0-143-134 kubenswrapper[2581]: W0918 20:48:45.727867    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41765192_e877_4448_b735_4ca110ed6a7f.slice/crio-e5bc08f3fbb8ddc28a81c46d2b38834a048e6814dcf9254f561aee54d4117ca2 WatchSource:0}: Error finding container e5bc08f3fbb8ddc28a81c46d2b38834a048e6814dcf9254f561aee54d4117ca2: Status 404 returned error can't find the container with id e5bc08f3fbb8ddc28a81c46d2b38834a048e6814dcf9254f561aee54d4117ca2
Sep 18 20:48:46.665448 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:46.665416    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7" event={"ID":"41765192-e877-4448-b735-4ca110ed6a7f","Type":"ContainerStarted","Data":"3f8123a54ab5227b50de3af9fe66d57ea948020659351fd720dd2fcb3367014c"}
Sep 18 20:48:46.665879 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:46.665470    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7" event={"ID":"41765192-e877-4448-b735-4ca110ed6a7f","Type":"ContainerStarted","Data":"e5bc08f3fbb8ddc28a81c46d2b38834a048e6814dcf9254f561aee54d4117ca2"}
Sep 18 20:48:46.665879 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:46.665530    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:48:47.675216 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:47.675162    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7" event={"ID":"41765192-e877-4448-b735-4ca110ed6a7f","Type":"ContainerStarted","Data":"8b4e3804002c0c57dbe45b0acb4cb1f8c386054b51b24ea1943503f3b65a5993"}
Sep 18 20:48:49.687896 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:49.687860    2581 generic.go:356] "Generic (PLEG): container finished" podID="41765192-e877-4448-b735-4ca110ed6a7f" containerID="8b4e3804002c0c57dbe45b0acb4cb1f8c386054b51b24ea1943503f3b65a5993" exitCode=0
Sep 18 20:48:49.688291 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:49.687922    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7" event={"ID":"41765192-e877-4448-b735-4ca110ed6a7f","Type":"ContainerDied","Data":"8b4e3804002c0c57dbe45b0acb4cb1f8c386054b51b24ea1943503f3b65a5993"}
Sep 18 20:48:50.695631 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:50.695593    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7" event={"ID":"41765192-e877-4448-b735-4ca110ed6a7f","Type":"ContainerStarted","Data":"523371fdba4b9559d64a001407a627bdaa9536ca11d89d04d959bc17480f706e"}
Sep 18 20:48:50.721262 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:50.721206    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7" podStartSLOduration=3.587606864 podStartE2EDuration="5.721190653s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.133583789s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:48:50.718705642 +0000 UTC m=+4253.820993971" watchObservedRunningTime="2026-09-18 20:48:50.721190653 +0000 UTC m=+4253.823478981"
Sep 18 20:48:55.435880 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:55.435829    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:48:55.436376 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:55.435912    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:48:55.437225 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:55.437181    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7" podUID="41765192-e877-4448-b735-4ca110ed6a7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8001/health\": dial tcp 10.132.0.73:8001: connect: connection refused"
Sep 18 20:48:55.449457 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:48:55.449435    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:49:05.435586 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:05.435540    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7" podUID="41765192-e877-4448-b735-4ca110ed6a7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8001/health\": dial tcp 10.132.0.73:8001: connect: connection refused"
Sep 18 20:49:15.435531 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:15.435490    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7" podUID="41765192-e877-4448-b735-4ca110ed6a7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8001/health\": dial tcp 10.132.0.73:8001: connect: connection refused"
Sep 18 20:49:20.526222 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.526185    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"]
Sep 18 20:49:20.529882 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.529857    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"
Sep 18 20:49:20.532605 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.532571    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:49:20.532605 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.532587    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-gfj9n\""
Sep 18 20:49:20.532813 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.532589    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 18 20:49:20.532813 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.532581    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 18 20:49:20.532813 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.532690    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 18 20:49:20.532813 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.532700    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 18 20:49:20.549176 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.549146    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"]
Sep 18 20:49:20.688360 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.688322    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/416a7b52-9dbf-4091-a38b-a7a49a860c8d-kserve-provision-location\") pod \"prestop-hook-test-kserve-787cdcb6df-65qgs\" (UID: \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"
Sep 18 20:49:20.688561 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.688383    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/416a7b52-9dbf-4091-a38b-a7a49a860c8d-home\") pod \"prestop-hook-test-kserve-787cdcb6df-65qgs\" (UID: \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"
Sep 18 20:49:20.688561 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.688500    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/416a7b52-9dbf-4091-a38b-a7a49a860c8d-model-cache\") pod \"prestop-hook-test-kserve-787cdcb6df-65qgs\" (UID: \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"
Sep 18 20:49:20.688561 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.688543    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/416a7b52-9dbf-4091-a38b-a7a49a860c8d-dshm\") pod \"prestop-hook-test-kserve-787cdcb6df-65qgs\" (UID: \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"
Sep 18 20:49:20.688736 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.688567    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/416a7b52-9dbf-4091-a38b-a7a49a860c8d-tmp-dir\") pod \"prestop-hook-test-kserve-787cdcb6df-65qgs\" (UID: \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"
Sep 18 20:49:20.688736 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.688632    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/416a7b52-9dbf-4091-a38b-a7a49a860c8d-cabundle-cert\") pod \"prestop-hook-test-kserve-787cdcb6df-65qgs\" (UID: \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"
Sep 18 20:49:20.688736 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.688664    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/416a7b52-9dbf-4091-a38b-a7a49a860c8d-tls-certs\") pod \"prestop-hook-test-kserve-787cdcb6df-65qgs\" (UID: \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"
Sep 18 20:49:20.688736 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.688696    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh44b\" (UniqueName: \"kubernetes.io/projected/416a7b52-9dbf-4091-a38b-a7a49a860c8d-kube-api-access-xh44b\") pod \"prestop-hook-test-kserve-787cdcb6df-65qgs\" (UID: \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"
Sep 18 20:49:20.790032 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.789921    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/416a7b52-9dbf-4091-a38b-a7a49a860c8d-kserve-provision-location\") pod \"prestop-hook-test-kserve-787cdcb6df-65qgs\" (UID: \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"
Sep 18 20:49:20.790032 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.789992    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/416a7b52-9dbf-4091-a38b-a7a49a860c8d-home\") pod \"prestop-hook-test-kserve-787cdcb6df-65qgs\" (UID: \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"
Sep 18 20:49:20.790272 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.790060    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/416a7b52-9dbf-4091-a38b-a7a49a860c8d-model-cache\") pod \"prestop-hook-test-kserve-787cdcb6df-65qgs\" (UID: \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"
Sep 18 20:49:20.790272 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.790092    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/416a7b52-9dbf-4091-a38b-a7a49a860c8d-dshm\") pod \"prestop-hook-test-kserve-787cdcb6df-65qgs\" (UID: \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"
Sep 18 20:49:20.790272 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.790113    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/416a7b52-9dbf-4091-a38b-a7a49a860c8d-tmp-dir\") pod \"prestop-hook-test-kserve-787cdcb6df-65qgs\" (UID: \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"
Sep 18 20:49:20.790272 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.790151    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/416a7b52-9dbf-4091-a38b-a7a49a860c8d-cabundle-cert\") pod \"prestop-hook-test-kserve-787cdcb6df-65qgs\" (UID: \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"
Sep 18 20:49:20.790485 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.790381    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/416a7b52-9dbf-4091-a38b-a7a49a860c8d-tls-certs\") pod \"prestop-hook-test-kserve-787cdcb6df-65qgs\" (UID: \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"
Sep 18 20:49:20.790485 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.790405    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/416a7b52-9dbf-4091-a38b-a7a49a860c8d-home\") pod \"prestop-hook-test-kserve-787cdcb6df-65qgs\" (UID: \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"
Sep 18 20:49:20.790485 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.790420    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh44b\" (UniqueName: \"kubernetes.io/projected/416a7b52-9dbf-4091-a38b-a7a49a860c8d-kube-api-access-xh44b\") pod \"prestop-hook-test-kserve-787cdcb6df-65qgs\" (UID: \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"
Sep 18 20:49:20.790646 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.790512    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/416a7b52-9dbf-4091-a38b-a7a49a860c8d-kserve-provision-location\") pod \"prestop-hook-test-kserve-787cdcb6df-65qgs\" (UID: \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"
Sep 18 20:49:20.790646 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.790571    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/416a7b52-9dbf-4091-a38b-a7a49a860c8d-tmp-dir\") pod \"prestop-hook-test-kserve-787cdcb6df-65qgs\" (UID: \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"
Sep 18 20:49:20.790754 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.790692    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/416a7b52-9dbf-4091-a38b-a7a49a860c8d-model-cache\") pod \"prestop-hook-test-kserve-787cdcb6df-65qgs\" (UID: \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"
Sep 18 20:49:20.791189 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.791170    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/416a7b52-9dbf-4091-a38b-a7a49a860c8d-cabundle-cert\") pod \"prestop-hook-test-kserve-787cdcb6df-65qgs\" (UID: \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"
Sep 18 20:49:20.792765 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.792744    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/416a7b52-9dbf-4091-a38b-a7a49a860c8d-dshm\") pod \"prestop-hook-test-kserve-787cdcb6df-65qgs\" (UID: \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"
Sep 18 20:49:20.793084 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.793065    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/416a7b52-9dbf-4091-a38b-a7a49a860c8d-tls-certs\") pod \"prestop-hook-test-kserve-787cdcb6df-65qgs\" (UID: \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"
Sep 18 20:49:20.798945 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.798924    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh44b\" (UniqueName: \"kubernetes.io/projected/416a7b52-9dbf-4091-a38b-a7a49a860c8d-kube-api-access-xh44b\") pod \"prestop-hook-test-kserve-787cdcb6df-65qgs\" (UID: \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"
Sep 18 20:49:20.840798 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:20.840770    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"
Sep 18 20:49:21.121217 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:21.121185    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"]
Sep 18 20:49:21.122131 ip-10-0-143-134 kubenswrapper[2581]: W0918 20:49:21.122099    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416a7b52_9dbf_4091_a38b_a7a49a860c8d.slice/crio-100fedc96cd10ba8de7edc78fd7427ec0ffffe0bf047a64b2dfc5db98c1ed735 WatchSource:0}: Error finding container 100fedc96cd10ba8de7edc78fd7427ec0ffffe0bf047a64b2dfc5db98c1ed735: Status 404 returned error can't find the container with id 100fedc96cd10ba8de7edc78fd7427ec0ffffe0bf047a64b2dfc5db98c1ed735
Sep 18 20:49:21.876364 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:21.876314    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs" event={"ID":"416a7b52-9dbf-4091-a38b-a7a49a860c8d","Type":"ContainerStarted","Data":"36bfadc707bc26c303054fc78996c01b81382a0cddb4b4e17eb2eb459300d3c4"}
Sep 18 20:49:21.876786 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:21.876430    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs" event={"ID":"416a7b52-9dbf-4091-a38b-a7a49a860c8d","Type":"ContainerStarted","Data":"100fedc96cd10ba8de7edc78fd7427ec0ffffe0bf047a64b2dfc5db98c1ed735"}
Sep 18 20:49:23.889012 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:23.888978    2581 generic.go:356] "Generic (PLEG): container finished" podID="416a7b52-9dbf-4091-a38b-a7a49a860c8d" containerID="36bfadc707bc26c303054fc78996c01b81382a0cddb4b4e17eb2eb459300d3c4" exitCode=0
Sep 18 20:49:23.889422 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:23.889054    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs" event={"ID":"416a7b52-9dbf-4091-a38b-a7a49a860c8d","Type":"ContainerDied","Data":"36bfadc707bc26c303054fc78996c01b81382a0cddb4b4e17eb2eb459300d3c4"}
Sep 18 20:49:24.896902 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:24.896861    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs" event={"ID":"416a7b52-9dbf-4091-a38b-a7a49a860c8d","Type":"ContainerStarted","Data":"8db3641ec0cf68a971b346120583e5235e8424f01f055848bb30a3f454a6b739"}
Sep 18 20:49:24.919902 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:24.919822    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs" podStartSLOduration=2.682203573 podStartE2EDuration="4.919803114s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.237599541s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:49:24.917623278 +0000 UTC m=+4288.019911616" watchObservedRunningTime="2026-09-18 20:49:24.919803114 +0000 UTC m=+4288.022091442"
Sep 18 20:49:25.435512 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:25.435454    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7" podUID="41765192-e877-4448-b735-4ca110ed6a7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8001/health\": dial tcp 10.132.0.73:8001: connect: connection refused"
Sep 18 20:49:25.841036 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:25.841003    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"
Sep 18 20:49:30.841685 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:30.841634    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"
Sep 18 20:49:30.843549 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:30.843516    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs" podUID="416a7b52-9dbf-4091-a38b-a7a49a860c8d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 18 20:49:35.435826 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:35.435777    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7" podUID="41765192-e877-4448-b735-4ca110ed6a7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8001/health\": dial tcp 10.132.0.73:8001: connect: connection refused"
Sep 18 20:49:39.031538 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:39.031503    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"]
Sep 18 20:49:45.435312 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:45.435265    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7" podUID="41765192-e877-4448-b735-4ca110ed6a7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8001/health\": dial tcp 10.132.0.73:8001: connect: connection refused"
Sep 18 20:49:54.065960 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:54.065871    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs" podUID="416a7b52-9dbf-4091-a38b-a7a49a860c8d" containerName="main" containerID="cri-o://8db3641ec0cf68a971b346120583e5235e8424f01f055848bb30a3f454a6b739" gracePeriod=45
Sep 18 20:49:55.436137 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:49:55.436080    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7" podUID="41765192-e877-4448-b735-4ca110ed6a7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8001/health\": dial tcp 10.132.0.73:8001: connect: connection refused"
Sep 18 20:50:05.435957 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:05.435908    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7" podUID="41765192-e877-4448-b735-4ca110ed6a7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8001/health\": dial tcp 10.132.0.73:8001: connect: connection refused"
Sep 18 20:50:15.435301 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:15.435258    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7" podUID="41765192-e877-4448-b735-4ca110ed6a7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8001/health\": dial tcp 10.132.0.73:8001: connect: connection refused"
Sep 18 20:50:25.436213 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:25.436153    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7" podUID="41765192-e877-4448-b735-4ca110ed6a7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8001/health\": dial tcp 10.132.0.73:8001: connect: connection refused"
Sep 18 20:50:35.435412 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:35.435373    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7" podUID="41765192-e877-4448-b735-4ca110ed6a7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8001/health\": dial tcp 10.132.0.73:8001: connect: connection refused"
Sep 18 20:50:39.354608 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:39.354529    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-787cdcb6df-65qgs_416a7b52-9dbf-4091-a38b-a7a49a860c8d/main/0.log"
Sep 18 20:50:39.355068 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:39.354957    2581 generic.go:356] "Generic (PLEG): container finished" podID="416a7b52-9dbf-4091-a38b-a7a49a860c8d" containerID="8db3641ec0cf68a971b346120583e5235e8424f01f055848bb30a3f454a6b739" exitCode=137
Sep 18 20:50:39.355068 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:39.355015    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs" event={"ID":"416a7b52-9dbf-4091-a38b-a7a49a860c8d","Type":"ContainerDied","Data":"8db3641ec0cf68a971b346120583e5235e8424f01f055848bb30a3f454a6b739"}
Sep 18 20:50:39.754084 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:39.754059    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-787cdcb6df-65qgs_416a7b52-9dbf-4091-a38b-a7a49a860c8d/main/0.log"
Sep 18 20:50:39.757743 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:39.757717    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"
Sep 18 20:50:39.951719 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:39.951592    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/416a7b52-9dbf-4091-a38b-a7a49a860c8d-tls-certs\") pod \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\" (UID: \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\") "
Sep 18 20:50:39.951719 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:39.951675    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/416a7b52-9dbf-4091-a38b-a7a49a860c8d-cabundle-cert\") pod \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\" (UID: \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\") "
Sep 18 20:50:39.952038 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:39.951778    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/416a7b52-9dbf-4091-a38b-a7a49a860c8d-home\") pod \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\" (UID: \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\") "
Sep 18 20:50:39.952038 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:39.951809    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/416a7b52-9dbf-4091-a38b-a7a49a860c8d-model-cache\") pod \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\" (UID: \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\") "
Sep 18 20:50:39.952038 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:39.951878    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/416a7b52-9dbf-4091-a38b-a7a49a860c8d-dshm\") pod \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\" (UID: \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\") "
Sep 18 20:50:39.952038 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:39.951926    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xh44b\" (UniqueName: \"kubernetes.io/projected/416a7b52-9dbf-4091-a38b-a7a49a860c8d-kube-api-access-xh44b\") pod \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\" (UID: \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\") "
Sep 18 20:50:39.952038 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:39.951988    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/416a7b52-9dbf-4091-a38b-a7a49a860c8d-kserve-provision-location\") pod \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\" (UID: \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\") "
Sep 18 20:50:39.952038 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:39.952018    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/416a7b52-9dbf-4091-a38b-a7a49a860c8d-tmp-dir\") pod \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\" (UID: \"416a7b52-9dbf-4091-a38b-a7a49a860c8d\") "
Sep 18 20:50:39.952405 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:39.952333    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/416a7b52-9dbf-4091-a38b-a7a49a860c8d-home" (OuterVolumeSpecName: "home") pod "416a7b52-9dbf-4091-a38b-a7a49a860c8d" (UID: "416a7b52-9dbf-4091-a38b-a7a49a860c8d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:50:39.952656 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:39.952582    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/416a7b52-9dbf-4091-a38b-a7a49a860c8d-model-cache" (OuterVolumeSpecName: "model-cache") pod "416a7b52-9dbf-4091-a38b-a7a49a860c8d" (UID: "416a7b52-9dbf-4091-a38b-a7a49a860c8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:50:39.952796 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:39.952770    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/416a7b52-9dbf-4091-a38b-a7a49a860c8d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "416a7b52-9dbf-4091-a38b-a7a49a860c8d" (UID: "416a7b52-9dbf-4091-a38b-a7a49a860c8d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:50:39.953423 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:39.953397    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/416a7b52-9dbf-4091-a38b-a7a49a860c8d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "416a7b52-9dbf-4091-a38b-a7a49a860c8d" (UID: "416a7b52-9dbf-4091-a38b-a7a49a860c8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:50:39.954641 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:39.954608    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/416a7b52-9dbf-4091-a38b-a7a49a860c8d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "416a7b52-9dbf-4091-a38b-a7a49a860c8d" (UID: "416a7b52-9dbf-4091-a38b-a7a49a860c8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:50:39.955235 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:39.955214    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/416a7b52-9dbf-4091-a38b-a7a49a860c8d-dshm" (OuterVolumeSpecName: "dshm") pod "416a7b52-9dbf-4091-a38b-a7a49a860c8d" (UID: "416a7b52-9dbf-4091-a38b-a7a49a860c8d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:50:39.955556 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:39.955541    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/416a7b52-9dbf-4091-a38b-a7a49a860c8d-kube-api-access-xh44b" (OuterVolumeSpecName: "kube-api-access-xh44b") pod "416a7b52-9dbf-4091-a38b-a7a49a860c8d" (UID: "416a7b52-9dbf-4091-a38b-a7a49a860c8d"). InnerVolumeSpecName "kube-api-access-xh44b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:50:39.975888 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:39.975862    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/416a7b52-9dbf-4091-a38b-a7a49a860c8d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "416a7b52-9dbf-4091-a38b-a7a49a860c8d" (UID: "416a7b52-9dbf-4091-a38b-a7a49a860c8d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:50:40.053053 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:40.053015    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/416a7b52-9dbf-4091-a38b-a7a49a860c8d-kserve-provision-location\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:50:40.053254 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:40.053057    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/416a7b52-9dbf-4091-a38b-a7a49a860c8d-tmp-dir\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:50:40.053254 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:40.053078    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/416a7b52-9dbf-4091-a38b-a7a49a860c8d-tls-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:50:40.053254 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:40.053095    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/416a7b52-9dbf-4091-a38b-a7a49a860c8d-cabundle-cert\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:50:40.053254 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:40.053110    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/416a7b52-9dbf-4091-a38b-a7a49a860c8d-home\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:50:40.053254 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:40.053126    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/416a7b52-9dbf-4091-a38b-a7a49a860c8d-model-cache\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:50:40.053254 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:40.053141    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/416a7b52-9dbf-4091-a38b-a7a49a860c8d-dshm\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:50:40.053254 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:40.053156    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh44b\" (UniqueName: \"kubernetes.io/projected/416a7b52-9dbf-4091-a38b-a7a49a860c8d-kube-api-access-xh44b\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:50:40.362750 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:40.362711    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-787cdcb6df-65qgs_416a7b52-9dbf-4091-a38b-a7a49a860c8d/main/0.log"
Sep 18 20:50:40.363213 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:40.363194    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"
Sep 18 20:50:40.363285 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:40.363197    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs" event={"ID":"416a7b52-9dbf-4091-a38b-a7a49a860c8d","Type":"ContainerDied","Data":"100fedc96cd10ba8de7edc78fd7427ec0ffffe0bf047a64b2dfc5db98c1ed735"}
Sep 18 20:50:40.363329 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:40.363321    2581 scope.go:117] "RemoveContainer" containerID="8db3641ec0cf68a971b346120583e5235e8424f01f055848bb30a3f454a6b739"
Sep 18 20:50:40.373199 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:40.373184    2581 scope.go:117] "RemoveContainer" containerID="36bfadc707bc26c303054fc78996c01b81382a0cddb4b4e17eb2eb459300d3c4"
Sep 18 20:50:40.386311 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:40.386288    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"]
Sep 18 20:50:40.389364 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:40.389345    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-787cdcb6df-65qgs"]
Sep 18 20:50:41.434780 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:41.434742    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="416a7b52-9dbf-4091-a38b-a7a49a860c8d" path="/var/lib/kubelet/pods/416a7b52-9dbf-4091-a38b-a7a49a860c8d/volumes"
Sep 18 20:50:45.450716 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:45.450684    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:50:45.463613 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:50:45.463585    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:51:21.330604 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:21.330510    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"]
Sep 18 20:51:36.353663 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:36.353613    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7" podUID="41765192-e877-4448-b735-4ca110ed6a7f" containerName="main" containerID="cri-o://523371fdba4b9559d64a001407a627bdaa9536ca11d89d04d959bc17480f706e" gracePeriod=45
Sep 18 20:51:36.970891 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:36.970853    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7" podUID="41765192-e877-4448-b735-4ca110ed6a7f" containerName="llm-d-routing-sidecar" containerID="cri-o://3f8123a54ab5227b50de3af9fe66d57ea948020659351fd720dd2fcb3367014c" gracePeriod=45
Sep 18 20:51:37.210830 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.210803    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:51:37.239298 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.239268    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41765192-e877-4448-b735-4ca110ed6a7f-tmp-dir\") pod \"41765192-e877-4448-b735-4ca110ed6a7f\" (UID: \"41765192-e877-4448-b735-4ca110ed6a7f\") "
Sep 18 20:51:37.239436 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.239334    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41765192-e877-4448-b735-4ca110ed6a7f-tls-certs\") pod \"41765192-e877-4448-b735-4ca110ed6a7f\" (UID: \"41765192-e877-4448-b735-4ca110ed6a7f\") "
Sep 18 20:51:37.239436 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.239370    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41765192-e877-4448-b735-4ca110ed6a7f-model-cache\") pod \"41765192-e877-4448-b735-4ca110ed6a7f\" (UID: \"41765192-e877-4448-b735-4ca110ed6a7f\") "
Sep 18 20:51:37.239436 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.239396    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41765192-e877-4448-b735-4ca110ed6a7f-home\") pod \"41765192-e877-4448-b735-4ca110ed6a7f\" (UID: \"41765192-e877-4448-b735-4ca110ed6a7f\") "
Sep 18 20:51:37.239436 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.239431    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41765192-e877-4448-b735-4ca110ed6a7f-dshm\") pod \"41765192-e877-4448-b735-4ca110ed6a7f\" (UID: \"41765192-e877-4448-b735-4ca110ed6a7f\") "
Sep 18 20:51:37.239658 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.239483    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5x92n\" (UniqueName: \"kubernetes.io/projected/41765192-e877-4448-b735-4ca110ed6a7f-kube-api-access-5x92n\") pod \"41765192-e877-4448-b735-4ca110ed6a7f\" (UID: \"41765192-e877-4448-b735-4ca110ed6a7f\") "
Sep 18 20:51:37.239658 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.239514    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41765192-e877-4448-b735-4ca110ed6a7f-kserve-provision-location\") pod \"41765192-e877-4448-b735-4ca110ed6a7f\" (UID: \"41765192-e877-4448-b735-4ca110ed6a7f\") "
Sep 18 20:51:37.239658 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.239562    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41765192-e877-4448-b735-4ca110ed6a7f-cabundle-cert\") pod \"41765192-e877-4448-b735-4ca110ed6a7f\" (UID: \"41765192-e877-4448-b735-4ca110ed6a7f\") "
Sep 18 20:51:37.240241 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.240074    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41765192-e877-4448-b735-4ca110ed6a7f-model-cache" (OuterVolumeSpecName: "model-cache") pod "41765192-e877-4448-b735-4ca110ed6a7f" (UID: "41765192-e877-4448-b735-4ca110ed6a7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:51:37.240241 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.240168    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41765192-e877-4448-b735-4ca110ed6a7f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "41765192-e877-4448-b735-4ca110ed6a7f" (UID: "41765192-e877-4448-b735-4ca110ed6a7f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:51:37.240906 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.240873    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41765192-e877-4448-b735-4ca110ed6a7f-home" (OuterVolumeSpecName: "home") pod "41765192-e877-4448-b735-4ca110ed6a7f" (UID: "41765192-e877-4448-b735-4ca110ed6a7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:51:37.242878 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.242681    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41765192-e877-4448-b735-4ca110ed6a7f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "41765192-e877-4448-b735-4ca110ed6a7f" (UID: "41765192-e877-4448-b735-4ca110ed6a7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:51:37.242878 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.242794    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41765192-e877-4448-b735-4ca110ed6a7f-dshm" (OuterVolumeSpecName: "dshm") pod "41765192-e877-4448-b735-4ca110ed6a7f" (UID: "41765192-e877-4448-b735-4ca110ed6a7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:51:37.242878 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.242804    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41765192-e877-4448-b735-4ca110ed6a7f-kube-api-access-5x92n" (OuterVolumeSpecName: "kube-api-access-5x92n") pod "41765192-e877-4448-b735-4ca110ed6a7f" (UID: "41765192-e877-4448-b735-4ca110ed6a7f"). InnerVolumeSpecName "kube-api-access-5x92n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:51:37.251359 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.251334    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41765192-e877-4448-b735-4ca110ed6a7f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "41765192-e877-4448-b735-4ca110ed6a7f" (UID: "41765192-e877-4448-b735-4ca110ed6a7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:51:37.269000 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.268972    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41765192-e877-4448-b735-4ca110ed6a7f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "41765192-e877-4448-b735-4ca110ed6a7f" (UID: "41765192-e877-4448-b735-4ca110ed6a7f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:51:37.340784 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.340752    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41765192-e877-4448-b735-4ca110ed6a7f-tls-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:51:37.340784 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.340780    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41765192-e877-4448-b735-4ca110ed6a7f-model-cache\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:51:37.340784 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.340788    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41765192-e877-4448-b735-4ca110ed6a7f-home\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:51:37.340991 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.340795    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41765192-e877-4448-b735-4ca110ed6a7f-dshm\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:51:37.340991 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.340804    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x92n\" (UniqueName: \"kubernetes.io/projected/41765192-e877-4448-b735-4ca110ed6a7f-kube-api-access-5x92n\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:51:37.340991 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.340813    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41765192-e877-4448-b735-4ca110ed6a7f-kserve-provision-location\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:51:37.340991 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.340821    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41765192-e877-4448-b735-4ca110ed6a7f-cabundle-cert\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:51:37.340991 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.340870    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41765192-e877-4448-b735-4ca110ed6a7f-tmp-dir\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:51:37.662531 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.662494    2581 generic.go:356] "Generic (PLEG): container finished" podID="41765192-e877-4448-b735-4ca110ed6a7f" containerID="523371fdba4b9559d64a001407a627bdaa9536ca11d89d04d959bc17480f706e" exitCode=0
Sep 18 20:51:37.662531 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.662523    2581 generic.go:356] "Generic (PLEG): container finished" podID="41765192-e877-4448-b735-4ca110ed6a7f" containerID="3f8123a54ab5227b50de3af9fe66d57ea948020659351fd720dd2fcb3367014c" exitCode=0
Sep 18 20:51:37.663007 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.662549    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"
Sep 18 20:51:37.663007 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.662581    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7" event={"ID":"41765192-e877-4448-b735-4ca110ed6a7f","Type":"ContainerDied","Data":"523371fdba4b9559d64a001407a627bdaa9536ca11d89d04d959bc17480f706e"}
Sep 18 20:51:37.663007 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.662643    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7" event={"ID":"41765192-e877-4448-b735-4ca110ed6a7f","Type":"ContainerDied","Data":"3f8123a54ab5227b50de3af9fe66d57ea948020659351fd720dd2fcb3367014c"}
Sep 18 20:51:37.663007 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.662662    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7" event={"ID":"41765192-e877-4448-b735-4ca110ed6a7f","Type":"ContainerDied","Data":"e5bc08f3fbb8ddc28a81c46d2b38834a048e6814dcf9254f561aee54d4117ca2"}
Sep 18 20:51:37.663007 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.662683    2581 scope.go:117] "RemoveContainer" containerID="523371fdba4b9559d64a001407a627bdaa9536ca11d89d04d959bc17480f706e"
Sep 18 20:51:37.671827 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.671806    2581 scope.go:117] "RemoveContainer" containerID="8b4e3804002c0c57dbe45b0acb4cb1f8c386054b51b24ea1943503f3b65a5993"
Sep 18 20:51:37.680359 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.680337    2581 scope.go:117] "RemoveContainer" containerID="3f8123a54ab5227b50de3af9fe66d57ea948020659351fd720dd2fcb3367014c"
Sep 18 20:51:37.681289 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.681209    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"]
Sep 18 20:51:37.686276 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.686254    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7958899789-gp9g7"]
Sep 18 20:51:37.688607 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.688593    2581 scope.go:117] "RemoveContainer" containerID="523371fdba4b9559d64a001407a627bdaa9536ca11d89d04d959bc17480f706e"
Sep 18 20:51:37.688904 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:51:37.688882    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"523371fdba4b9559d64a001407a627bdaa9536ca11d89d04d959bc17480f706e\": container with ID starting with 523371fdba4b9559d64a001407a627bdaa9536ca11d89d04d959bc17480f706e not found: ID does not exist" containerID="523371fdba4b9559d64a001407a627bdaa9536ca11d89d04d959bc17480f706e"
Sep 18 20:51:37.689005 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.688911    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"523371fdba4b9559d64a001407a627bdaa9536ca11d89d04d959bc17480f706e"} err="failed to get container status \"523371fdba4b9559d64a001407a627bdaa9536ca11d89d04d959bc17480f706e\": rpc error: code = NotFound desc = could not find container \"523371fdba4b9559d64a001407a627bdaa9536ca11d89d04d959bc17480f706e\": container with ID starting with 523371fdba4b9559d64a001407a627bdaa9536ca11d89d04d959bc17480f706e not found: ID does not exist"
Sep 18 20:51:37.689005 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.688928    2581 scope.go:117] "RemoveContainer" containerID="8b4e3804002c0c57dbe45b0acb4cb1f8c386054b51b24ea1943503f3b65a5993"
Sep 18 20:51:37.689141 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:51:37.689127    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b4e3804002c0c57dbe45b0acb4cb1f8c386054b51b24ea1943503f3b65a5993\": container with ID starting with 8b4e3804002c0c57dbe45b0acb4cb1f8c386054b51b24ea1943503f3b65a5993 not found: ID does not exist" containerID="8b4e3804002c0c57dbe45b0acb4cb1f8c386054b51b24ea1943503f3b65a5993"
Sep 18 20:51:37.689190 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.689143    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b4e3804002c0c57dbe45b0acb4cb1f8c386054b51b24ea1943503f3b65a5993"} err="failed to get container status \"8b4e3804002c0c57dbe45b0acb4cb1f8c386054b51b24ea1943503f3b65a5993\": rpc error: code = NotFound desc = could not find container \"8b4e3804002c0c57dbe45b0acb4cb1f8c386054b51b24ea1943503f3b65a5993\": container with ID starting with 8b4e3804002c0c57dbe45b0acb4cb1f8c386054b51b24ea1943503f3b65a5993 not found: ID does not exist"
Sep 18 20:51:37.689190 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.689154    2581 scope.go:117] "RemoveContainer" containerID="3f8123a54ab5227b50de3af9fe66d57ea948020659351fd720dd2fcb3367014c"
Sep 18 20:51:37.689394 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:51:37.689372    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f8123a54ab5227b50de3af9fe66d57ea948020659351fd720dd2fcb3367014c\": container with ID starting with 3f8123a54ab5227b50de3af9fe66d57ea948020659351fd720dd2fcb3367014c not found: ID does not exist" containerID="3f8123a54ab5227b50de3af9fe66d57ea948020659351fd720dd2fcb3367014c"
Sep 18 20:51:37.689430 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.689400    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f8123a54ab5227b50de3af9fe66d57ea948020659351fd720dd2fcb3367014c"} err="failed to get container status \"3f8123a54ab5227b50de3af9fe66d57ea948020659351fd720dd2fcb3367014c\": rpc error: code = NotFound desc = could not find container \"3f8123a54ab5227b50de3af9fe66d57ea948020659351fd720dd2fcb3367014c\": container with ID starting with 3f8123a54ab5227b50de3af9fe66d57ea948020659351fd720dd2fcb3367014c not found: ID does not exist"
Sep 18 20:51:37.689430 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.689420    2581 scope.go:117] "RemoveContainer" containerID="523371fdba4b9559d64a001407a627bdaa9536ca11d89d04d959bc17480f706e"
Sep 18 20:51:37.689648 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.689633    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"523371fdba4b9559d64a001407a627bdaa9536ca11d89d04d959bc17480f706e"} err="failed to get container status \"523371fdba4b9559d64a001407a627bdaa9536ca11d89d04d959bc17480f706e\": rpc error: code = NotFound desc = could not find container \"523371fdba4b9559d64a001407a627bdaa9536ca11d89d04d959bc17480f706e\": container with ID starting with 523371fdba4b9559d64a001407a627bdaa9536ca11d89d04d959bc17480f706e not found: ID does not exist"
Sep 18 20:51:37.689710 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.689648    2581 scope.go:117] "RemoveContainer" containerID="8b4e3804002c0c57dbe45b0acb4cb1f8c386054b51b24ea1943503f3b65a5993"
Sep 18 20:51:37.689822 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.689807    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b4e3804002c0c57dbe45b0acb4cb1f8c386054b51b24ea1943503f3b65a5993"} err="failed to get container status \"8b4e3804002c0c57dbe45b0acb4cb1f8c386054b51b24ea1943503f3b65a5993\": rpc error: code = NotFound desc = could not find container \"8b4e3804002c0c57dbe45b0acb4cb1f8c386054b51b24ea1943503f3b65a5993\": container with ID starting with 8b4e3804002c0c57dbe45b0acb4cb1f8c386054b51b24ea1943503f3b65a5993 not found: ID does not exist"
Sep 18 20:51:37.689937 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.689821    2581 scope.go:117] "RemoveContainer" containerID="3f8123a54ab5227b50de3af9fe66d57ea948020659351fd720dd2fcb3367014c"
Sep 18 20:51:37.690100 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:37.690079    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f8123a54ab5227b50de3af9fe66d57ea948020659351fd720dd2fcb3367014c"} err="failed to get container status \"3f8123a54ab5227b50de3af9fe66d57ea948020659351fd720dd2fcb3367014c\": rpc error: code = NotFound desc = could not find container \"3f8123a54ab5227b50de3af9fe66d57ea948020659351fd720dd2fcb3367014c\": container with ID starting with 3f8123a54ab5227b50de3af9fe66d57ea948020659351fd720dd2fcb3367014c not found: ID does not exist"
Sep 18 20:51:39.433141 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:39.433098    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41765192-e877-4448-b735-4ca110ed6a7f" path="/var/lib/kubelet/pods/41765192-e877-4448-b735-4ca110ed6a7f/volumes"
Sep 18 20:51:41.454459 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.454425    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6"]
Sep 18 20:51:41.454869 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.454747    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41765192-e877-4448-b735-4ca110ed6a7f" containerName="storage-initializer"
Sep 18 20:51:41.454869 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.454757    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="41765192-e877-4448-b735-4ca110ed6a7f" containerName="storage-initializer"
Sep 18 20:51:41.454869 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.454768    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="416a7b52-9dbf-4091-a38b-a7a49a860c8d" containerName="storage-initializer"
Sep 18 20:51:41.454869 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.454775    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="416a7b52-9dbf-4091-a38b-a7a49a860c8d" containerName="storage-initializer"
Sep 18 20:51:41.454869 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.454796    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41765192-e877-4448-b735-4ca110ed6a7f" containerName="llm-d-routing-sidecar"
Sep 18 20:51:41.454869 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.454802    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="41765192-e877-4448-b735-4ca110ed6a7f" containerName="llm-d-routing-sidecar"
Sep 18 20:51:41.454869 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.454817    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41765192-e877-4448-b735-4ca110ed6a7f" containerName="main"
Sep 18 20:51:41.454869 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.454822    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="41765192-e877-4448-b735-4ca110ed6a7f" containerName="main"
Sep 18 20:51:41.454869 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.454848    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="416a7b52-9dbf-4091-a38b-a7a49a860c8d" containerName="main"
Sep 18 20:51:41.454869 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.454856    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="416a7b52-9dbf-4091-a38b-a7a49a860c8d" containerName="main"
Sep 18 20:51:41.455193 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.454928    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="41765192-e877-4448-b735-4ca110ed6a7f" containerName="llm-d-routing-sidecar"
Sep 18 20:51:41.455193 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.454940    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="41765192-e877-4448-b735-4ca110ed6a7f" containerName="main"
Sep 18 20:51:41.455193 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.454946    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="416a7b52-9dbf-4091-a38b-a7a49a860c8d" containerName="main"
Sep 18 20:51:41.459635 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.459617    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6"
Sep 18 20:51:41.462183 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.462162    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 18 20:51:41.463215 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.463199    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 18 20:51:41.463268 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.463249    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 18 20:51:41.463484 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.463471    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-ff7kp\""
Sep 18 20:51:41.466980 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.466960    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6"]
Sep 18 20:51:41.475826 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.475800    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a26ac5-5f72-4012-831c-1d3084b258ba-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wvjc6\" (UID: \"71a26ac5-5f72-4012-831c-1d3084b258ba\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6"
Sep 18 20:51:41.475941 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.475889    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a26ac5-5f72-4012-831c-1d3084b258ba-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wvjc6\" (UID: \"71a26ac5-5f72-4012-831c-1d3084b258ba\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6"
Sep 18 20:51:41.475941 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.475918    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a26ac5-5f72-4012-831c-1d3084b258ba-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wvjc6\" (UID: \"71a26ac5-5f72-4012-831c-1d3084b258ba\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6"
Sep 18 20:51:41.476021 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.476008    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a26ac5-5f72-4012-831c-1d3084b258ba-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wvjc6\" (UID: \"71a26ac5-5f72-4012-831c-1d3084b258ba\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6"
Sep 18 20:51:41.476054 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.476039    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a26ac5-5f72-4012-831c-1d3084b258ba-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wvjc6\" (UID: \"71a26ac5-5f72-4012-831c-1d3084b258ba\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6"
Sep 18 20:51:41.476118 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.476101    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpwps\" (UniqueName: \"kubernetes.io/projected/71a26ac5-5f72-4012-831c-1d3084b258ba-kube-api-access-fpwps\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wvjc6\" (UID: \"71a26ac5-5f72-4012-831c-1d3084b258ba\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6"
Sep 18 20:51:41.576994 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.576943    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpwps\" (UniqueName: \"kubernetes.io/projected/71a26ac5-5f72-4012-831c-1d3084b258ba-kube-api-access-fpwps\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wvjc6\" (UID: \"71a26ac5-5f72-4012-831c-1d3084b258ba\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6"
Sep 18 20:51:41.577208 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.577010    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a26ac5-5f72-4012-831c-1d3084b258ba-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wvjc6\" (UID: \"71a26ac5-5f72-4012-831c-1d3084b258ba\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6"
Sep 18 20:51:41.577208 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.577057    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a26ac5-5f72-4012-831c-1d3084b258ba-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wvjc6\" (UID: \"71a26ac5-5f72-4012-831c-1d3084b258ba\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6"
Sep 18 20:51:41.577208 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.577079    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a26ac5-5f72-4012-831c-1d3084b258ba-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wvjc6\" (UID: \"71a26ac5-5f72-4012-831c-1d3084b258ba\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6"
Sep 18 20:51:41.577208 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.577145    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a26ac5-5f72-4012-831c-1d3084b258ba-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wvjc6\" (UID: \"71a26ac5-5f72-4012-831c-1d3084b258ba\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6"
Sep 18 20:51:41.577208 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.577180    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a26ac5-5f72-4012-831c-1d3084b258ba-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wvjc6\" (UID: \"71a26ac5-5f72-4012-831c-1d3084b258ba\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6"
Sep 18 20:51:41.577488 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.577351    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a26ac5-5f72-4012-831c-1d3084b258ba-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wvjc6\" (UID: \"71a26ac5-5f72-4012-831c-1d3084b258ba\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6"
Sep 18 20:51:41.577549 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.577532    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a26ac5-5f72-4012-831c-1d3084b258ba-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wvjc6\" (UID: \"71a26ac5-5f72-4012-831c-1d3084b258ba\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6"
Sep 18 20:51:41.577594 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.577559    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a26ac5-5f72-4012-831c-1d3084b258ba-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wvjc6\" (UID: \"71a26ac5-5f72-4012-831c-1d3084b258ba\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6"
Sep 18 20:51:41.579403 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.579380    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a26ac5-5f72-4012-831c-1d3084b258ba-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wvjc6\" (UID: \"71a26ac5-5f72-4012-831c-1d3084b258ba\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6"
Sep 18 20:51:41.579805 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.579789    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a26ac5-5f72-4012-831c-1d3084b258ba-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wvjc6\" (UID: \"71a26ac5-5f72-4012-831c-1d3084b258ba\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6"
Sep 18 20:51:41.585310 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.585290    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpwps\" (UniqueName: \"kubernetes.io/projected/71a26ac5-5f72-4012-831c-1d3084b258ba-kube-api-access-fpwps\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-wvjc6\" (UID: \"71a26ac5-5f72-4012-831c-1d3084b258ba\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6"
Sep 18 20:51:41.770958 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:41.770925    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6"
Sep 18 20:51:42.051425 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:42.051328    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6"]
Sep 18 20:51:42.057207 ip-10-0-143-134 kubenswrapper[2581]: W0918 20:51:42.057181    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a26ac5_5f72_4012_831c_1d3084b258ba.slice/crio-0142f42168cbe0089c7ef425f42754594be33996170445c2b12a3123adc01853 WatchSource:0}: Error finding container 0142f42168cbe0089c7ef425f42754594be33996170445c2b12a3123adc01853: Status 404 returned error can't find the container with id 0142f42168cbe0089c7ef425f42754594be33996170445c2b12a3123adc01853
Sep 18 20:51:42.692460 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:42.692422    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6" event={"ID":"71a26ac5-5f72-4012-831c-1d3084b258ba","Type":"ContainerStarted","Data":"ca6a9588851eae49cf81010a871a5b5b05d179262a6b0facf19c19ff60f2d6b4"}
Sep 18 20:51:42.693006 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:42.692472    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6" event={"ID":"71a26ac5-5f72-4012-831c-1d3084b258ba","Type":"ContainerStarted","Data":"0142f42168cbe0089c7ef425f42754594be33996170445c2b12a3123adc01853"}
Sep 18 20:51:42.711102 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:42.711054    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6" podStartSLOduration=1.711038593 podStartE2EDuration="1.711038593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:51:42.710030592 +0000 UTC m=+4425.812318920" watchObservedRunningTime="2026-09-18 20:51:42.711038593 +0000 UTC m=+4425.813326920"
Sep 18 20:51:42.771783 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:42.771748    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6"
Sep 18 20:51:51.771682 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:51.771644    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6"
Sep 18 20:51:51.784056 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:51.784022    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6"
Sep 18 20:51:52.756129 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:51:52.756089    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6"
Sep 18 20:52:07.149582 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:52:07.149544    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6"]
Sep 18 20:52:07.160262 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:52:07.160222    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 20:52:07.160262 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:52:07.160262 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 20:52:07.160262 ip-10-0-143-134 kubenswrapper[2581]:  > containerID="ca6a9588851eae49cf81010a871a5b5b05d179262a6b0facf19c19ff60f2d6b4" cmd=["/bin/sleep","15"]
Sep 18 20:52:07.160603 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:52:07.160284    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6" message="" err=<
Sep 18 20:52:07.160603 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:52:07.160603 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 20:52:07.160603 ip-10-0-143-134 kubenswrapper[2581]:  >
Sep 18 20:52:07.160603 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:52:07.160303    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 20:52:07.160603 ip-10-0-143-134 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 20:52:07.160603 ip-10-0-143-134 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 18 20:52:07.160603 ip-10-0-143-134 kubenswrapper[2581]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6" podUID="71a26ac5-5f72-4012-831c-1d3084b258ba" containerName="main" containerID="cri-o://ca6a9588851eae49cf81010a871a5b5b05d179262a6b0facf19c19ff60f2d6b4"
Sep 18 20:52:07.160603 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:52:07.160348    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6" podUID="71a26ac5-5f72-4012-831c-1d3084b258ba" containerName="main" containerID="cri-o://ca6a9588851eae49cf81010a871a5b5b05d179262a6b0facf19c19ff60f2d6b4" gracePeriod=60
Sep 18 20:52:07.512685 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:52:07.512659    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6"
Sep 18 20:52:07.608098 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:52:07.608056    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a26ac5-5f72-4012-831c-1d3084b258ba-dshm\") pod \"71a26ac5-5f72-4012-831c-1d3084b258ba\" (UID: \"71a26ac5-5f72-4012-831c-1d3084b258ba\") "
Sep 18 20:52:07.608296 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:52:07.608113    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a26ac5-5f72-4012-831c-1d3084b258ba-tmp-dir\") pod \"71a26ac5-5f72-4012-831c-1d3084b258ba\" (UID: \"71a26ac5-5f72-4012-831c-1d3084b258ba\") "
Sep 18 20:52:07.608296 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:52:07.608134    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpwps\" (UniqueName: \"kubernetes.io/projected/71a26ac5-5f72-4012-831c-1d3084b258ba-kube-api-access-fpwps\") pod \"71a26ac5-5f72-4012-831c-1d3084b258ba\" (UID: \"71a26ac5-5f72-4012-831c-1d3084b258ba\") "
Sep 18 20:52:07.608296 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:52:07.608156    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a26ac5-5f72-4012-831c-1d3084b258ba-model-cache\") pod \"71a26ac5-5f72-4012-831c-1d3084b258ba\" (UID: \"71a26ac5-5f72-4012-831c-1d3084b258ba\") "
Sep 18 20:52:07.608296 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:52:07.608205    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a26ac5-5f72-4012-831c-1d3084b258ba-tls-certs\") pod \"71a26ac5-5f72-4012-831c-1d3084b258ba\" (UID: \"71a26ac5-5f72-4012-831c-1d3084b258ba\") "
Sep 18 20:52:07.608296 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:52:07.608265    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a26ac5-5f72-4012-831c-1d3084b258ba-home\") pod \"71a26ac5-5f72-4012-831c-1d3084b258ba\" (UID: \"71a26ac5-5f72-4012-831c-1d3084b258ba\") "
Sep 18 20:52:07.608576 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:52:07.608438    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a26ac5-5f72-4012-831c-1d3084b258ba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "71a26ac5-5f72-4012-831c-1d3084b258ba" (UID: "71a26ac5-5f72-4012-831c-1d3084b258ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:52:07.608576 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:52:07.608489    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a26ac5-5f72-4012-831c-1d3084b258ba-model-cache" (OuterVolumeSpecName: "model-cache") pod "71a26ac5-5f72-4012-831c-1d3084b258ba" (UID: "71a26ac5-5f72-4012-831c-1d3084b258ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:52:07.608576 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:52:07.608535    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a26ac5-5f72-4012-831c-1d3084b258ba-tmp-dir\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:52:07.608735 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:52:07.608593    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a26ac5-5f72-4012-831c-1d3084b258ba-home" (OuterVolumeSpecName: "home") pod "71a26ac5-5f72-4012-831c-1d3084b258ba" (UID: "71a26ac5-5f72-4012-831c-1d3084b258ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:52:07.610711 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:52:07.610681    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a26ac5-5f72-4012-831c-1d3084b258ba-kube-api-access-fpwps" (OuterVolumeSpecName: "kube-api-access-fpwps") pod "71a26ac5-5f72-4012-831c-1d3084b258ba" (UID: "71a26ac5-5f72-4012-831c-1d3084b258ba"). InnerVolumeSpecName "kube-api-access-fpwps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:52:07.610711 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:52:07.610700    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a26ac5-5f72-4012-831c-1d3084b258ba-dshm" (OuterVolumeSpecName: "dshm") pod "71a26ac5-5f72-4012-831c-1d3084b258ba" (UID: "71a26ac5-5f72-4012-831c-1d3084b258ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:52:07.610874 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:52:07.610733    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71a26ac5-5f72-4012-831c-1d3084b258ba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "71a26ac5-5f72-4012-831c-1d3084b258ba" (UID: "71a26ac5-5f72-4012-831c-1d3084b258ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:52:07.709403 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:52:07.709296    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a26ac5-5f72-4012-831c-1d3084b258ba-tls-certs\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:52:07.709403 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:52:07.709343    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a26ac5-5f72-4012-831c-1d3084b258ba-home\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:52:07.709403 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:52:07.709351    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a26ac5-5f72-4012-831c-1d3084b258ba-dshm\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:52:07.709403 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:52:07.709360    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpwps\" (UniqueName: \"kubernetes.io/projected/71a26ac5-5f72-4012-831c-1d3084b258ba-kube-api-access-fpwps\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:52:07.709403 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:52:07.709370    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a26ac5-5f72-4012-831c-1d3084b258ba-model-cache\") on node \"ip-10-0-143-134.ec2.internal\" DevicePath \"\""
Sep 18 20:52:07.824125 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:52:07.824090    2581 generic.go:356] "Generic (PLEG): container finished" podID="71a26ac5-5f72-4012-831c-1d3084b258ba" containerID="ca6a9588851eae49cf81010a871a5b5b05d179262a6b0facf19c19ff60f2d6b4" exitCode=0
Sep 18 20:52:07.824330 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:52:07.824134    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6"
Sep 18 20:52:07.824330 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:52:07.824182    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6" event={"ID":"71a26ac5-5f72-4012-831c-1d3084b258ba","Type":"ContainerDied","Data":"ca6a9588851eae49cf81010a871a5b5b05d179262a6b0facf19c19ff60f2d6b4"}
Sep 18 20:52:07.824330 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:52:07.824241    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6" event={"ID":"71a26ac5-5f72-4012-831c-1d3084b258ba","Type":"ContainerDied","Data":"0142f42168cbe0089c7ef425f42754594be33996170445c2b12a3123adc01853"}
Sep 18 20:52:07.824330 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:52:07.824275    2581 scope.go:117] "RemoveContainer" containerID="ca6a9588851eae49cf81010a871a5b5b05d179262a6b0facf19c19ff60f2d6b4"
Sep 18 20:52:07.833982 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:52:07.833964    2581 scope.go:117] "RemoveContainer" containerID="ca6a9588851eae49cf81010a871a5b5b05d179262a6b0facf19c19ff60f2d6b4"
Sep 18 20:52:07.834229 ip-10-0-143-134 kubenswrapper[2581]: E0918 20:52:07.834209    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca6a9588851eae49cf81010a871a5b5b05d179262a6b0facf19c19ff60f2d6b4\": container with ID starting with ca6a9588851eae49cf81010a871a5b5b05d179262a6b0facf19c19ff60f2d6b4 not found: ID does not exist" containerID="ca6a9588851eae49cf81010a871a5b5b05d179262a6b0facf19c19ff60f2d6b4"
Sep 18 20:52:07.834274 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:52:07.834237    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca6a9588851eae49cf81010a871a5b5b05d179262a6b0facf19c19ff60f2d6b4"} err="failed to get container status \"ca6a9588851eae49cf81010a871a5b5b05d179262a6b0facf19c19ff60f2d6b4\": rpc error: code = NotFound desc = could not find container \"ca6a9588851eae49cf81010a871a5b5b05d179262a6b0facf19c19ff60f2d6b4\": container with ID starting with ca6a9588851eae49cf81010a871a5b5b05d179262a6b0facf19c19ff60f2d6b4 not found: ID does not exist"
Sep 18 20:52:07.846387 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:52:07.846366    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6"]
Sep 18 20:52:07.858343 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:52:07.858320    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-wvjc6"]
Sep 18 20:52:09.432823 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:52:09.432787    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71a26ac5-5f72-4012-831c-1d3084b258ba" path="/var/lib/kubelet/pods/71a26ac5-5f72-4012-831c-1d3084b258ba/volumes"
Sep 18 20:53:19.333149 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:19.333095    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-gfm4b_5a7f347f-5fd5-4b95-a61d-fac88872ac60/authorino/0.log"
Sep 18 20:53:19.372520 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:19.372493    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-bq46m_d8340c00-5917-4482-9af1-3085b2315c7d/kuadrant-console-plugin/0.log"
Sep 18 20:53:22.388278 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:22.388239    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kcrhg/must-gather-f9fmk"]
Sep 18 20:53:22.388698 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:22.388618    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71a26ac5-5f72-4012-831c-1d3084b258ba" containerName="main"
Sep 18 20:53:22.388698 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:22.388631    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="71a26ac5-5f72-4012-831c-1d3084b258ba" containerName="main"
Sep 18 20:53:22.388785 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:22.388701    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="71a26ac5-5f72-4012-831c-1d3084b258ba" containerName="main"
Sep 18 20:53:22.391810 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:22.391793    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kcrhg/must-gather-f9fmk"
Sep 18 20:53:22.395344 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:22.395321    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kcrhg\"/\"kube-root-ca.crt\""
Sep 18 20:53:22.395480 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:22.395342    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kcrhg\"/\"default-dockercfg-2sg28\""
Sep 18 20:53:22.396252 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:22.396238    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kcrhg\"/\"openshift-service-ca.crt\""
Sep 18 20:53:22.402135 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:22.402115    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kcrhg/must-gather-f9fmk"]
Sep 18 20:53:22.446279 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:22.446240    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5d2dd3ad-3b8d-4efb-8c5a-cfef1d2871dd-must-gather-output\") pod \"must-gather-f9fmk\" (UID: \"5d2dd3ad-3b8d-4efb-8c5a-cfef1d2871dd\") " pod="openshift-must-gather-kcrhg/must-gather-f9fmk"
Sep 18 20:53:22.446451 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:22.446348    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt85x\" (UniqueName: \"kubernetes.io/projected/5d2dd3ad-3b8d-4efb-8c5a-cfef1d2871dd-kube-api-access-tt85x\") pod \"must-gather-f9fmk\" (UID: \"5d2dd3ad-3b8d-4efb-8c5a-cfef1d2871dd\") " pod="openshift-must-gather-kcrhg/must-gather-f9fmk"
Sep 18 20:53:22.547537 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:22.547496    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5d2dd3ad-3b8d-4efb-8c5a-cfef1d2871dd-must-gather-output\") pod \"must-gather-f9fmk\" (UID: \"5d2dd3ad-3b8d-4efb-8c5a-cfef1d2871dd\") " pod="openshift-must-gather-kcrhg/must-gather-f9fmk"
Sep 18 20:53:22.547724 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:22.547603    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt85x\" (UniqueName: \"kubernetes.io/projected/5d2dd3ad-3b8d-4efb-8c5a-cfef1d2871dd-kube-api-access-tt85x\") pod \"must-gather-f9fmk\" (UID: \"5d2dd3ad-3b8d-4efb-8c5a-cfef1d2871dd\") " pod="openshift-must-gather-kcrhg/must-gather-f9fmk"
Sep 18 20:53:22.547890 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:22.547868    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5d2dd3ad-3b8d-4efb-8c5a-cfef1d2871dd-must-gather-output\") pod \"must-gather-f9fmk\" (UID: \"5d2dd3ad-3b8d-4efb-8c5a-cfef1d2871dd\") " pod="openshift-must-gather-kcrhg/must-gather-f9fmk"
Sep 18 20:53:22.556253 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:22.556231    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt85x\" (UniqueName: \"kubernetes.io/projected/5d2dd3ad-3b8d-4efb-8c5a-cfef1d2871dd-kube-api-access-tt85x\") pod \"must-gather-f9fmk\" (UID: \"5d2dd3ad-3b8d-4efb-8c5a-cfef1d2871dd\") " pod="openshift-must-gather-kcrhg/must-gather-f9fmk"
Sep 18 20:53:22.702430 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:22.702350    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kcrhg/must-gather-f9fmk"
Sep 18 20:53:22.974178 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:22.974150    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kcrhg/must-gather-f9fmk"]
Sep 18 20:53:22.976013 ip-10-0-143-134 kubenswrapper[2581]: W0918 20:53:22.975985    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d2dd3ad_3b8d_4efb_8c5a_cfef1d2871dd.slice/crio-7ae72d22fed089252cec04a815d2e70d1a17b02f22ece5d88a32ce83e73f5985 WatchSource:0}: Error finding container 7ae72d22fed089252cec04a815d2e70d1a17b02f22ece5d88a32ce83e73f5985: Status 404 returned error can't find the container with id 7ae72d22fed089252cec04a815d2e70d1a17b02f22ece5d88a32ce83e73f5985
Sep 18 20:53:22.977913 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:22.977894    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:53:23.213755 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:23.213721    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kcrhg/must-gather-f9fmk" event={"ID":"5d2dd3ad-3b8d-4efb-8c5a-cfef1d2871dd","Type":"ContainerStarted","Data":"7ae72d22fed089252cec04a815d2e70d1a17b02f22ece5d88a32ce83e73f5985"}
Sep 18 20:53:24.222008 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:24.221973    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kcrhg/must-gather-f9fmk" event={"ID":"5d2dd3ad-3b8d-4efb-8c5a-cfef1d2871dd","Type":"ContainerStarted","Data":"2fc12f380985dc4c91d6d177297116bb95cdbc6ac63fa141755b70136c273dea"}
Sep 18 20:53:24.222449 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:24.222028    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kcrhg/must-gather-f9fmk" event={"ID":"5d2dd3ad-3b8d-4efb-8c5a-cfef1d2871dd","Type":"ContainerStarted","Data":"3aecf67d4b068a59e91a0a641c97462aeb539e44c9953ca8488174f94a2fb950"}
Sep 18 20:53:24.240160 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:24.240113    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kcrhg/must-gather-f9fmk" podStartSLOduration=1.4892369140000001 podStartE2EDuration="2.240097089s" totalImagesPullingTime="750.860175ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:53:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:53:24.237197012 +0000 UTC m=+4527.339485340" watchObservedRunningTime="2026-09-18 20:53:24.240097089 +0000 UTC m=+4527.342385417"
Sep 18 20:53:25.245386 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:25.245346    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2p2bp_5d3541d3-fb52-42a5-b977-9cc44ac396bc/global-pull-secret-syncer/0.log"
Sep 18 20:53:25.424396 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:25.424368    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cg6mk_6ff14ef8-6eac-448b-b297-b1fb15d59eac/konnectivity-agent/0.log"
Sep 18 20:53:25.566149 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:25.566121    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-134.ec2.internal_dff242b51bfc62e6a56c5d2723611cd5/haproxy/0.log"
Sep 18 20:53:30.651466 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:30.650983    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-gfm4b_5a7f347f-5fd5-4b95-a61d-fac88872ac60/authorino/0.log"
Sep 18 20:53:30.749183 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:30.749147    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-bq46m_d8340c00-5917-4482-9af1-3085b2315c7d/kuadrant-console-plugin/0.log"
Sep 18 20:53:32.266940 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:32.266905    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ee7a81a5-d588-4320-8b6f-5dbb28fa5b87/alertmanager/0.log"
Sep 18 20:53:32.293298 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:32.293266    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ee7a81a5-d588-4320-8b6f-5dbb28fa5b87/config-reloader/0.log"
Sep 18 20:53:32.318960 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:32.318927    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ee7a81a5-d588-4320-8b6f-5dbb28fa5b87/kube-rbac-proxy-web/0.log"
Sep 18 20:53:32.342593 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:32.342560    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ee7a81a5-d588-4320-8b6f-5dbb28fa5b87/kube-rbac-proxy/0.log"
Sep 18 20:53:32.366131 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:32.366099    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ee7a81a5-d588-4320-8b6f-5dbb28fa5b87/kube-rbac-proxy-metric/0.log"
Sep 18 20:53:32.401353 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:32.401325    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ee7a81a5-d588-4320-8b6f-5dbb28fa5b87/prom-label-proxy/0.log"
Sep 18 20:53:32.434227 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:32.434191    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ee7a81a5-d588-4320-8b6f-5dbb28fa5b87/init-config-reloader/0.log"
Sep 18 20:53:32.485856 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:32.485718    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-rm6st_efe551fc-5e21-4174-bfaf-95e53c300ae2/cluster-monitoring-operator/0.log"
Sep 18 20:53:32.658474 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:32.658444    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-r4twg_44301539-88b6-4be8-8475-a1e987e81370/monitoring-plugin/0.log"
Sep 18 20:53:32.807285 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:32.807200    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bdftw_9c2ad484-0f94-4f69-84a8-4fb65d4e8d53/node-exporter/0.log"
Sep 18 20:53:32.836442 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:32.836371    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bdftw_9c2ad484-0f94-4f69-84a8-4fb65d4e8d53/kube-rbac-proxy/0.log"
Sep 18 20:53:32.865363 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:32.865329    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bdftw_9c2ad484-0f94-4f69-84a8-4fb65d4e8d53/init-textfile/0.log"
Sep 18 20:53:33.351740 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:33.351671    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-57b7d86db6-pbz6t_0eddddab-ac2f-4633-bb6a-01a896804307/telemeter-client/0.log"
Sep 18 20:53:33.376723 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:33.376673    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-57b7d86db6-pbz6t_0eddddab-ac2f-4633-bb6a-01a896804307/reload/0.log"
Sep 18 20:53:33.399275 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:33.399247    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-57b7d86db6-pbz6t_0eddddab-ac2f-4633-bb6a-01a896804307/kube-rbac-proxy/0.log"
Sep 18 20:53:33.429693 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:33.429660    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78d686bcb7-gmvm2_75f125aa-6a99-40d4-914d-67c8efd07d39/thanos-query/0.log"
Sep 18 20:53:33.453454 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:33.453412    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78d686bcb7-gmvm2_75f125aa-6a99-40d4-914d-67c8efd07d39/kube-rbac-proxy-web/0.log"
Sep 18 20:53:33.477357 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:33.477332    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78d686bcb7-gmvm2_75f125aa-6a99-40d4-914d-67c8efd07d39/kube-rbac-proxy/0.log"
Sep 18 20:53:33.502396 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:33.502369    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78d686bcb7-gmvm2_75f125aa-6a99-40d4-914d-67c8efd07d39/prom-label-proxy/0.log"
Sep 18 20:53:33.523696 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:33.523646    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78d686bcb7-gmvm2_75f125aa-6a99-40d4-914d-67c8efd07d39/kube-rbac-proxy-rules/0.log"
Sep 18 20:53:33.544631 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:33.544597    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-78d686bcb7-gmvm2_75f125aa-6a99-40d4-914d-67c8efd07d39/kube-rbac-proxy-metrics/0.log"
Sep 18 20:53:34.620276 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:34.620241    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-f7sc6_b1138025-f527-48a3-baa5-499c4bfbe124/networking-console-plugin/0.log"
Sep 18 20:53:35.305303 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:35.305268    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kcrhg/perf-node-gather-daemonset-jtkzv"]
Sep 18 20:53:35.311398 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:35.311366    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-jtkzv"
Sep 18 20:53:35.316569 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:35.316544    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kcrhg/perf-node-gather-daemonset-jtkzv"]
Sep 18 20:53:35.372525 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:35.372330    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87ee9e40-99a9-45d6-8535-e886545d46e9-sys\") pod \"perf-node-gather-daemonset-jtkzv\" (UID: \"87ee9e40-99a9-45d6-8535-e886545d46e9\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-jtkzv"
Sep 18 20:53:35.372525 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:35.372382    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87ee9e40-99a9-45d6-8535-e886545d46e9-lib-modules\") pod \"perf-node-gather-daemonset-jtkzv\" (UID: \"87ee9e40-99a9-45d6-8535-e886545d46e9\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-jtkzv"
Sep 18 20:53:35.372525 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:35.372447    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggffv\" (UniqueName: \"kubernetes.io/projected/87ee9e40-99a9-45d6-8535-e886545d46e9-kube-api-access-ggffv\") pod \"perf-node-gather-daemonset-jtkzv\" (UID: \"87ee9e40-99a9-45d6-8535-e886545d46e9\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-jtkzv"
Sep 18 20:53:35.372525 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:35.372496    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/87ee9e40-99a9-45d6-8535-e886545d46e9-proc\") pod \"perf-node-gather-daemonset-jtkzv\" (UID: \"87ee9e40-99a9-45d6-8535-e886545d46e9\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-jtkzv"
Sep 18 20:53:35.372525 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:35.372525    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/87ee9e40-99a9-45d6-8535-e886545d46e9-podres\") pod \"perf-node-gather-daemonset-jtkzv\" (UID: \"87ee9e40-99a9-45d6-8535-e886545d46e9\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-jtkzv"
Sep 18 20:53:35.473921 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:35.473889    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/87ee9e40-99a9-45d6-8535-e886545d46e9-proc\") pod \"perf-node-gather-daemonset-jtkzv\" (UID: \"87ee9e40-99a9-45d6-8535-e886545d46e9\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-jtkzv"
Sep 18 20:53:35.473921 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:35.473926    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/87ee9e40-99a9-45d6-8535-e886545d46e9-podres\") pod \"perf-node-gather-daemonset-jtkzv\" (UID: \"87ee9e40-99a9-45d6-8535-e886545d46e9\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-jtkzv"
Sep 18 20:53:35.474119 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:35.473979    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87ee9e40-99a9-45d6-8535-e886545d46e9-sys\") pod \"perf-node-gather-daemonset-jtkzv\" (UID: \"87ee9e40-99a9-45d6-8535-e886545d46e9\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-jtkzv"
Sep 18 20:53:35.474119 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:35.473996    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87ee9e40-99a9-45d6-8535-e886545d46e9-lib-modules\") pod \"perf-node-gather-daemonset-jtkzv\" (UID: \"87ee9e40-99a9-45d6-8535-e886545d46e9\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-jtkzv"
Sep 18 20:53:35.474119 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:35.474032    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggffv\" (UniqueName: \"kubernetes.io/projected/87ee9e40-99a9-45d6-8535-e886545d46e9-kube-api-access-ggffv\") pod \"perf-node-gather-daemonset-jtkzv\" (UID: \"87ee9e40-99a9-45d6-8535-e886545d46e9\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-jtkzv"
Sep 18 20:53:35.474119 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:35.474040    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/87ee9e40-99a9-45d6-8535-e886545d46e9-proc\") pod \"perf-node-gather-daemonset-jtkzv\" (UID: \"87ee9e40-99a9-45d6-8535-e886545d46e9\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-jtkzv"
Sep 18 20:53:35.474119 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:35.474091    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87ee9e40-99a9-45d6-8535-e886545d46e9-sys\") pod \"perf-node-gather-daemonset-jtkzv\" (UID: \"87ee9e40-99a9-45d6-8535-e886545d46e9\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-jtkzv"
Sep 18 20:53:35.474283 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:35.474136    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/87ee9e40-99a9-45d6-8535-e886545d46e9-podres\") pod \"perf-node-gather-daemonset-jtkzv\" (UID: \"87ee9e40-99a9-45d6-8535-e886545d46e9\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-jtkzv"
Sep 18 20:53:35.474283 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:35.474164    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87ee9e40-99a9-45d6-8535-e886545d46e9-lib-modules\") pod \"perf-node-gather-daemonset-jtkzv\" (UID: \"87ee9e40-99a9-45d6-8535-e886545d46e9\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-jtkzv"
Sep 18 20:53:35.484731 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:35.484705    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggffv\" (UniqueName: \"kubernetes.io/projected/87ee9e40-99a9-45d6-8535-e886545d46e9-kube-api-access-ggffv\") pod \"perf-node-gather-daemonset-jtkzv\" (UID: \"87ee9e40-99a9-45d6-8535-e886545d46e9\") " pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-jtkzv"
Sep 18 20:53:35.580681 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:35.580611    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6cc68c547f-kmlfx_df95825b-4cda-4434-8962-457e21e7d633/console/0.log"
Sep 18 20:53:35.625460 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:35.625429    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-jtkzv"
Sep 18 20:53:35.925114 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:35.925086    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kcrhg/perf-node-gather-daemonset-jtkzv"]
Sep 18 20:53:35.927024 ip-10-0-143-134 kubenswrapper[2581]: W0918 20:53:35.926995    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod87ee9e40_99a9_45d6_8535_e886545d46e9.slice/crio-5a0d6b9d7eee0c493e1ad394665b22606f7e27741727816a16e69530c3ab4e73 WatchSource:0}: Error finding container 5a0d6b9d7eee0c493e1ad394665b22606f7e27741727816a16e69530c3ab4e73: Status 404 returned error can't find the container with id 5a0d6b9d7eee0c493e1ad394665b22606f7e27741727816a16e69530c3ab4e73
Sep 18 20:53:36.237661 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:36.237593    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-9h56d_bf93b809-edab-49b2-8217-9e5ab4d42789/volume-data-source-validator/0.log"
Sep 18 20:53:36.334917 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:36.334879    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-jtkzv" event={"ID":"87ee9e40-99a9-45d6-8535-e886545d46e9","Type":"ContainerStarted","Data":"072fd1f411f4de519378a72a5b48c4a99a3252beb5536afba46790868440abfc"}
Sep 18 20:53:36.335100 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:36.334943    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-jtkzv" event={"ID":"87ee9e40-99a9-45d6-8535-e886545d46e9","Type":"ContainerStarted","Data":"5a0d6b9d7eee0c493e1ad394665b22606f7e27741727816a16e69530c3ab4e73"}
Sep 18 20:53:36.335100 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:36.334989    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-jtkzv"
Sep 18 20:53:36.353919 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:36.353868    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-jtkzv" podStartSLOduration=1.35385315 podStartE2EDuration="1.35385315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:53:36.35253886 +0000 UTC m=+4539.454827189" watchObservedRunningTime="2026-09-18 20:53:36.35385315 +0000 UTC m=+4539.456141470"
Sep 18 20:53:37.306625 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:37.306587    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dwtpk_82551de1-f8ab-432e-8ff6-1da685c07fa9/dns/0.log"
Sep 18 20:53:37.328796 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:37.328769    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dwtpk_82551de1-f8ab-432e-8ff6-1da685c07fa9/kube-rbac-proxy/0.log"
Sep 18 20:53:37.448073 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:37.448042    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w9dlm_58fa049b-20bb-4466-bb38-8b46448db93b/dns-node-resolver/0.log"
Sep 18 20:53:38.020010 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:38.019972    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7nrfp_68f35ba9-52cf-4a53-bc81-b2863eeb0930/node-ca/0.log"
Sep 18 20:53:39.890709 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:39.890680    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fn944_e7a69286-c751-476a-a2fc-631e26030284/serve-healthcheck-canary/0.log"
Sep 18 20:53:40.359653 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:40.359620    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-ffj79_e764d634-1d03-429d-8454-d725e6be456f/insights-operator/0.log"
Sep 18 20:53:40.361044 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:40.361012    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-ffj79_e764d634-1d03-429d-8454-d725e6be456f/insights-operator/1.log"
Sep 18 20:53:40.530139 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:40.530108    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qv2hc_9cd29832-b16f-48c7-a830-fd10ff432df8/kube-rbac-proxy/0.log"
Sep 18 20:53:40.551353 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:40.551323    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qv2hc_9cd29832-b16f-48c7-a830-fd10ff432df8/exporter/0.log"
Sep 18 20:53:40.574366 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:40.574341    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qv2hc_9cd29832-b16f-48c7-a830-fd10ff432df8/extractor/0.log"
Sep 18 20:53:42.351005 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:42.350977    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kcrhg/perf-node-gather-daemonset-jtkzv"
Sep 18 20:53:43.664785 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:43.664751    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-d69b774b8-h2kfq_bc342203-c44d-4cb8-84bb-2b70c1f759ad/manager/0.log"
Sep 18 20:53:43.690933 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:43.690893    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-fjbfv_0ce1a023-ce9b-441d-ad6d-7c5790179b51/openshift-lws-operator/0.log"
Sep 18 20:53:45.126411 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:45.126377    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-f2j8z_2aa87bc1-647b-4681-b4d2-3e221d8d2967/server/0.log"
Sep 18 20:53:47.022980 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:47.022945    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-t2pj2_46b58988-c6e2-4517-94dd-231064e5bf82/seaweedfs/0.log"
Sep 18 20:53:51.894610 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:51.894576    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-8l864_85c9bace-8688-4f4a-ad42-dc32911c6263/kube-storage-version-migrator-operator/1.log"
Sep 18 20:53:51.895634 ip-10-0-143-134 kubenswrapper[2581]: I0918 20:53:51.895614    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-8l864_85c9bace-8688-4f4a-ad42-dc32911c6263/kube-storage-version-migrator-operator/0.log"