Jul 28 17:21:10.975874 ip-10-0-135-177 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 28 17:21:10.975888 ip-10-0-135-177 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 28 17:21:10.975897 ip-10-0-135-177 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 28 17:21:10.976229 ip-10-0-135-177 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 28 17:21:21.201497 ip-10-0-135-177 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 28 17:21:21.201511 ip-10-0-135-177 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot b8937ff155e24a4ea3cfd2206fac3e33 --
Jul 28 17:23:16.495426 ip-10-0-135-177 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 17:23:16.827676 ip-10-0-135-177 kubenswrapper[2543]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:23:16.827676 ip-10-0-135-177 kubenswrapper[2543]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 17:23:16.827676 ip-10-0-135-177 kubenswrapper[2543]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:23:16.827676 ip-10-0-135-177 kubenswrapper[2543]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 17:23:16.827676 ip-10-0-135-177 kubenswrapper[2543]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:23:16.828584 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.828449    2543 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 17:23:16.834739 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834724    2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:23:16.834739 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834739    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:23:16.834806 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834743    2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:23:16.834806 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834747    2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:23:16.834806 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834750    2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:23:16.834806 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834753    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:23:16.834806 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834756    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:23:16.834806 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834759    2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:23:16.834806 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834762    2543 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:23:16.834806 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834765    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:23:16.834806 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834768    2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:23:16.834806 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834771    2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:23:16.834806 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834773    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:23:16.834806 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834776    2543 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:23:16.834806 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834778    2543 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:23:16.834806 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834781    2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:23:16.834806 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834784    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:23:16.834806 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834786    2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:23:16.834806 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834789    2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:23:16.834806 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834791    2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:23:16.834806 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834794    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:23:16.834806 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834796    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:23:16.835280 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834799    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:23:16.835280 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834802    2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:23:16.835280 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834804    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:23:16.835280 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834807    2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:23:16.835280 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834810    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:23:16.835280 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834813    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:23:16.835280 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834816    2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:23:16.835280 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834819    2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:23:16.835280 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834821    2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:23:16.835280 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834824    2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:23:16.835280 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834827    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:23:16.835280 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834829    2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:23:16.835280 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834832    2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:23:16.835280 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834834    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:23:16.835280 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834837    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:23:16.835280 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834840    2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:23:16.835280 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834843    2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:23:16.835280 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834845    2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:23:16.835280 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834848    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:23:16.835280 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834851    2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:23:16.835755 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834853    2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:23:16.835755 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834856    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:23:16.835755 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834858    2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:23:16.835755 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834860    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:23:16.835755 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834863    2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:23:16.835755 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834865    2543 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:23:16.835755 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834868    2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:23:16.835755 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834870    2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:23:16.835755 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834873    2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:23:16.835755 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834875    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:23:16.835755 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834878    2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:23:16.835755 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834880    2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:23:16.835755 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834883    2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:23:16.835755 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834888    2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:23:16.835755 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834892    2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:23:16.835755 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834895    2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:23:16.835755 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834900    2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:23:16.835755 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834904    2543 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:23:16.835755 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834906    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:23:16.836224 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834909    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:23:16.836224 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834912    2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:23:16.836224 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834915    2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:23:16.836224 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834917    2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:23:16.836224 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834920    2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:23:16.836224 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834922    2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:23:16.836224 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834925    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:23:16.836224 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834929    2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:23:16.836224 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834932    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:23:16.836224 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834935    2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:23:16.836224 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834939    2543 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:23:16.836224 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834942    2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:23:16.836224 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834944    2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:23:16.836224 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834947    2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:23:16.836224 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834951    2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:23:16.836224 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834954    2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:23:16.836224 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834956    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:23:16.836224 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834959    2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:23:16.836224 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834961    2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:23:16.836224 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834964    2543 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:23:16.836778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834966    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:23:16.836778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834969    2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:23:16.836778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834971    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:23:16.836778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834974    2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:23:16.836778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834976    2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:23:16.836778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834979    2543 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:23:16.836778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834982    2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:23:16.836778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834985    2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:23:16.836778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834988    2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:23:16.836778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834990    2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:23:16.836778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834993    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:23:16.836778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834996    2543 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:23:16.836778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.834998    2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:23:16.836778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835001    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:23:16.836778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835004    2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:23:16.836778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835006    2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:23:16.836778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835009    2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:23:16.836778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835012    2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:23:16.836778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835712    2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:23:16.836778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835718    2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:23:16.837269 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835721    2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:23:16.837269 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835724    2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:23:16.837269 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835726    2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:23:16.837269 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835730    2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:23:16.837269 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835733    2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:23:16.837269 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835736    2543 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:23:16.837269 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835739    2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:23:16.837269 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835741    2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:23:16.837269 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835744    2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:23:16.837269 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835747    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:23:16.837269 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835749    2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:23:16.837269 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835752    2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:23:16.837269 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835755    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:23:16.837269 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835757    2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:23:16.837269 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835760    2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:23:16.837269 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835763    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:23:16.837269 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835766    2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:23:16.837269 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835769    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:23:16.837269 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835773    2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:23:16.837759 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835776    2543 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:23:16.837759 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835781    2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:23:16.837759 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835785    2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:23:16.837759 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835788    2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:23:16.837759 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835791    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:23:16.837759 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835794    2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:23:16.837759 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835797    2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:23:16.837759 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835800    2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:23:16.837759 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835802    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:23:16.837759 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835805    2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:23:16.837759 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835808    2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:23:16.837759 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835810    2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:23:16.837759 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835813    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:23:16.837759 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835816    2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:23:16.837759 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835819    2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:23:16.837759 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835822    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:23:16.837759 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835825    2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:23:16.837759 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835828    2543 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:23:16.837759 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835831    2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:23:16.837759 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835834    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:23:16.838279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835836    2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:23:16.838279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835839    2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:23:16.838279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835842    2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:23:16.838279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835844    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:23:16.838279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835847    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:23:16.838279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835849    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:23:16.838279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835852    2543 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:23:16.838279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835855    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:23:16.838279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835857    2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:23:16.838279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835860    2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:23:16.838279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835862    2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:23:16.838279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835866    2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:23:16.838279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835868    2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:23:16.838279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835871    2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:23:16.838279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835874    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:23:16.838279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835876    2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:23:16.838279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835879    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:23:16.838279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835882    2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:23:16.838279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835884    2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:23:16.838279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835887    2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:23:16.838816 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835889    2543 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:23:16.838816 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835892    2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:23:16.838816 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835895    2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:23:16.838816 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835898    2543 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:23:16.838816 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835901    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:23:16.838816 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835903    2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:23:16.838816 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835906    2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:23:16.838816 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835908    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:23:16.838816 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835911    2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:23:16.838816 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835913    2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:23:16.838816 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835916    2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:23:16.838816 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835919    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:23:16.838816 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835922    2543 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:23:16.838816 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835924    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:23:16.838816 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835927    2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:23:16.838816 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835930    2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:23:16.838816 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835932    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:23:16.838816 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835935    2543 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:23:16.838816 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835937    2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:23:16.838816 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835941    2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:23:16.839314 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835944    2543 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:23:16.839314 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835947    2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:23:16.839314 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835951    2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:23:16.839314 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835955    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:23:16.839314 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835958    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:23:16.839314 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835961    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:23:16.839314 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835963    2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:23:16.839314 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835966    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:23:16.839314 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835968    2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:23:16.839314 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835971    2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:23:16.839314 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835974    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:23:16.839314 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835976    2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:23:16.839314 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835979    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:23:16.839314 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835981    2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:23:16.839314 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835984    2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:23:16.839314 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835987    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:23:16.839314 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835989    2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:23:16.839314 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.835992    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:23:16.839314 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836362    2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:23:16.839776 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836367    2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:23:16.839776 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836370    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:23:16.839776 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836373    2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:23:16.839776 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836376    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:23:16.839776 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836379    2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:23:16.839776 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836382    2543 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:23:16.839776 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836385    2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:23:16.839776 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836388    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:23:16.839776 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836390    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:23:16.839776 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836393    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:23:16.839776 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836396    2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:23:16.839776 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836399    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:23:16.839776 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836401    2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:23:16.839776 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836404    2543 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:23:16.839776 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836406    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:23:16.839776 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836409    2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:23:16.839776 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836412    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:23:16.839776 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836415    2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:23:16.839776 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836418    2543 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:23:16.839776 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836420    2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:23:16.840279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836423    2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:23:16.840279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836427    2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:23:16.840279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836430    2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:23:16.840279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836433    2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:23:16.840279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836436    2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:23:16.840279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836438    2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:23:16.840279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836441    2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:23:16.840279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836443    2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:23:16.840279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836446    2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:23:16.840279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836449    2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:23:16.840279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836453    2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:23:16.840279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836457    2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:23:16.840279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836460    2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:23:16.840279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836462    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:23:16.840279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836465    2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:23:16.840279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836468    2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:23:16.840279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836471    2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:23:16.840279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836474    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:23:16.840279 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836477    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:23:16.840752 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836479    2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:23:16.840752 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836482    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:23:16.840752 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836484    2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:23:16.840752 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836487    2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:23:16.840752 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836491    2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:23:16.840752 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836494    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:23:16.840752 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836497    2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:23:16.840752 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836500    2543 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:23:16.840752 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836502    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:23:16.840752 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836506    2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:23:16.840752 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836509    2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:23:16.840752 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836511    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:23:16.840752 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836514    2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:23:16.840752 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836516    2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:23:16.840752 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836519    2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:23:16.840752 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836521    2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:23:16.840752 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836524    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:23:16.840752 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836527    2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:23:16.840752 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836530    2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:23:16.841226 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836532    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:23:16.841226 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836535    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:23:16.841226 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836537    2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:23:16.841226 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836540    2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:23:16.841226 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836543    2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:23:16.841226 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836545    2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:23:16.841226 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836548    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:23:16.841226 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836550    2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:23:16.841226 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836553    2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:23:16.841226 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836555    2543 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:23:16.841226 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836560    2543 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:23:16.841226 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836563    2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:23:16.841226 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836566    2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:23:16.841226 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836569    2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:23:16.841226 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836571    2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:23:16.841226 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836574    2543 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:23:16.841226 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836577    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:23:16.841226 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836579    2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:23:16.841226 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836582    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:23:16.841226 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836585    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:23:16.841699 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836588    2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:23:16.841699 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836591    2543 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:23:16.841699 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836594    2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:23:16.841699 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836597    2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:23:16.841699 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836600    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:23:16.841699 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836602    2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:23:16.841699 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836605    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:23:16.841699 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836608    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:23:16.841699 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836611    2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:23:16.841699 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836613    2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:23:16.841699 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836616    2543 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:23:16.841699 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836618    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:23:16.841699 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836621    2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:23:16.841699 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836624    2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:23:16.841699 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836627    2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:23:16.841699 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836629    2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:23:16.841699 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836632    2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:23:16.841699 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836634    2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:23:16.841699 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836637    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:23:16.841699 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.836639    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:23:16.842216 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836705    2543 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 17:23:16.842216 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836716    2543 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 17:23:16.842216 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836729    2543 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 17:23:16.842216 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836733    2543 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 17:23:16.842216 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836738    2543 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 17:23:16.842216 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836741    2543 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 17:23:16.842216 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836745    2543 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 17:23:16.842216 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836749    2543 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 17:23:16.842216 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836752    2543 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 17:23:16.842216 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836755    2543 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 17:23:16.842216 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836759    2543 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 17:23:16.842216 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836762    2543 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 17:23:16.842216 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836766    2543 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 17:23:16.842216 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836770    2543 flags.go:64] FLAG: --cgroup-root=""
Jul 28 17:23:16.842216 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836773    2543 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 17:23:16.842216 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836777    2543 flags.go:64] FLAG: --client-ca-file=""
Jul 28 17:23:16.842216 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836780    2543 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 17:23:16.842216 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836783    2543 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 17:23:16.842216 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836787    2543 flags.go:64] FLAG: --cluster-domain=""
Jul 28 17:23:16.842216 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836791    2543 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 17:23:16.842216 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836794    2543 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 17:23:16.842216 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836797    2543 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 17:23:16.842216 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836801    2543 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 17:23:16.842216 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836804    2543 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 17:23:16.842788 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836807    2543 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 17:23:16.842788 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836811    2543 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 17:23:16.842788 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836814    2543 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 17:23:16.842788 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836817    2543 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 17:23:16.842788 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836820    2543 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 17:23:16.842788 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836823    2543 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 17:23:16.842788 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836826    2543 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 17:23:16.842788 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836829    2543 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 17:23:16.842788 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836833    2543 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 17:23:16.842788 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836836    2543 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 17:23:16.842788 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836840    2543 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 17:23:16.842788 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836843    2543 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 17:23:16.842788 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836846    2543 flags.go:64] FLAG: --enable-server="true"
Jul 28 17:23:16.842788 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836849    2543 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 17:23:16.842788 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836854    2543 flags.go:64] FLAG: --event-burst="100"
Jul 28 17:23:16.842788 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836857    2543 flags.go:64] FLAG: --event-qps="50"
Jul 28 17:23:16.842788 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836860    2543 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 17:23:16.842788 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836863    2543 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 17:23:16.842788 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836866    2543 flags.go:64] FLAG: --eviction-hard=""
Jul 28 17:23:16.842788 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836870    2543 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 17:23:16.842788 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836873    2543 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 17:23:16.842788 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836878    2543 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 17:23:16.842788 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836881    2543 flags.go:64] FLAG: --eviction-soft=""
Jul 28 17:23:16.842788 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836884    2543 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 17:23:16.842788 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836887    2543 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 17:23:16.843720 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836890    2543 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 17:23:16.843720 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836893    2543 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 17:23:16.843720 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836896    2543 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 17:23:16.843720 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836899    2543 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 17:23:16.843720 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836902    2543 flags.go:64] FLAG: --feature-gates=""
Jul 28 17:23:16.843720 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836906    2543 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 17:23:16.843720 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836910    2543 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 17:23:16.843720 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836913    2543 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 17:23:16.843720 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836916    2543 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 17:23:16.843720 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836919    2543 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 17:23:16.843720 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836922    2543 flags.go:64] FLAG: --help="false"
Jul 28 17:23:16.843720 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836925    2543 flags.go:64] FLAG: --hostname-override="ip-10-0-135-177.ec2.internal"
Jul 28 17:23:16.843720 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836928    2543 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 17:23:16.843720 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836931    2543 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 17:23:16.843720 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836934    2543 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 17:23:16.843720 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836938    2543 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 17:23:16.843720 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836942    2543 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 17:23:16.843720 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836947    2543 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 17:23:16.843720 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836950    2543 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 17:23:16.843720 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836953    2543 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 17:23:16.843720 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836956    2543 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 17:23:16.843720 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836962    2543 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 17:23:16.843720 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836965    2543 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 17:23:16.843720 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836968    2543 flags.go:64] FLAG: --kube-reserved=""
Jul 28 17:23:16.844752 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836971    2543 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 17:23:16.844752 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836974    2543 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 17:23:16.844752 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836977    2543 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 17:23:16.844752 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836980    2543 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 17:23:16.844752 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836985    2543 flags.go:64] FLAG: --lock-file=""
Jul 28 17:23:16.844752 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836989    2543 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 17:23:16.844752 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836992    2543 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 17:23:16.844752 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.836995    2543 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 17:23:16.844752 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837000    2543 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 17:23:16.844752 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837003    2543 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 17:23:16.844752 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837006    2543 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 17:23:16.844752 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837009    2543 flags.go:64] FLAG: --logging-format="text"
Jul 28 17:23:16.844752 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837012    2543 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 17:23:16.844752 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837015    2543 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 17:23:16.844752 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837018    2543 flags.go:64] FLAG: --manifest-url=""
Jul 28 17:23:16.844752 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837021    2543 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 17:23:16.844752 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837025    2543 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 17:23:16.844752 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837028    2543 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 17:23:16.844752 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837031    2543 flags.go:64] FLAG: --max-pods="110"
Jul 28 17:23:16.844752 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837035    2543 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 17:23:16.844752 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837038    2543 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 17:23:16.844752 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837040    2543 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 17:23:16.844752 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837044    2543 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 17:23:16.844752 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837047    2543 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 17:23:16.844752 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837050    2543 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 17:23:16.845818 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837054    2543 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 17:23:16.845818 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837061    2543 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 17:23:16.845818 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837064    2543 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 17:23:16.845818 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837067    2543 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 17:23:16.845818 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837070    2543 flags.go:64] FLAG: --pod-cidr=""
Jul 28 17:23:16.845818 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837073    2543 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 17:23:16.845818 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837078    2543 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 17:23:16.845818 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837082    2543 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 17:23:16.845818 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837085    2543 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 17:23:16.845818 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837088    2543 flags.go:64] FLAG: --port="10250"
Jul 28 17:23:16.845818 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837091    2543 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 17:23:16.845818 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837095    2543 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f2f20244a6206183"
Jul 28 17:23:16.845818 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837098    2543 flags.go:64] FLAG: --qos-reserved=""
Jul 28 17:23:16.845818 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837101    2543 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 17:23:16.845818 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837105    2543 flags.go:64] FLAG: --register-node="true"
Jul 28 17:23:16.845818 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837108    2543 flags.go:64] FLAG: --register-with-taints=""
Jul 28 17:23:16.845818 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837111    2543 flags.go:64] FLAG: --registry-burst="10"
Jul 28 17:23:16.845818 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837114    2543 flags.go:64] FLAG: --registry-qps="5"
Jul 28 17:23:16.845818 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837117    2543 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 17:23:16.845818 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837119    2543 flags.go:64] FLAG: --reserved-memory=""
Jul 28 17:23:16.845818 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837123    2543 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 17:23:16.845818 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837126    2543 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 17:23:16.845818 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837141    2543 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 17:23:16.845818 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837145    2543 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 17:23:16.845818 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837148    2543 flags.go:64] FLAG: --runonce="false"
Jul 28 17:23:16.846913 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837150    2543 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 17:23:16.846913 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837154    2543 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 17:23:16.846913 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837157    2543 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 17:23:16.846913 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837160    2543 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 17:23:16.846913 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837163    2543 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 17:23:16.846913 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837166    2543 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 17:23:16.846913 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837169    2543 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 17:23:16.846913 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837172    2543 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 17:23:16.846913 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837176    2543 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 17:23:16.846913 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837179    2543 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 17:23:16.846913 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837182    2543 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 17:23:16.846913 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837185    2543 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 17:23:16.846913 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837188    2543 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 17:23:16.846913 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837192    2543 flags.go:64] FLAG: --system-cgroups=""
Jul 28 17:23:16.846913 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837195    2543 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 17:23:16.846913 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837201    2543 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 17:23:16.846913 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837203    2543 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 17:23:16.846913 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837206    2543 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 17:23:16.846913 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837211    2543 flags.go:64] FLAG: --tls-min-version=""
Jul 28 17:23:16.846913 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837214    2543 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 17:23:16.846913 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837217    2543 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 17:23:16.846913 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837220    2543 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 17:23:16.846913 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837222    2543 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 17:23:16.846913 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837225    2543 flags.go:64] FLAG: --v="2"
Jul 28 17:23:16.846913 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837229    2543 flags.go:64] FLAG: --version="false"
Jul 28 17:23:16.847739 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837233    2543 flags.go:64] FLAG: --vmodule=""
Jul 28 17:23:16.847739 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837237    2543 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 17:23:16.847739 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.837240    2543 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 17:23:16.847739 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837348    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:23:16.847739 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837352    2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:23:16.847739 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837355    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:23:16.847739 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837358    2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:23:16.847739 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837361    2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:23:16.847739 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837364    2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:23:16.847739 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837367    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:23:16.847739 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837369    2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:23:16.847739 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837372    2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:23:16.847739 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837375    2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:23:16.847739 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837378    2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:23:16.847739 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837380    2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:23:16.847739 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837385    2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:23:16.847739 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837387    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:23:16.847739 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837390    2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:23:16.847739 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837393    2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:23:16.847739 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837395    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:23:16.848595 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837398    2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:23:16.848595 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837401    2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:23:16.848595 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837403    2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:23:16.848595 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837406    2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:23:16.848595 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837409    2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:23:16.848595 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837413    2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:23:16.848595 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837416    2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:23:16.848595 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837418    2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:23:16.848595 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837421    2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:23:16.848595 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837423    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:23:16.848595 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837426    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:23:16.848595 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837429    2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:23:16.848595 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837431    2543 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:23:16.848595 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837434    2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:23:16.848595 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837436    2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:23:16.848595 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837439    2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:23:16.848595 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837441    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:23:16.848595 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837444    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:23:16.848595 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837447    2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:23:16.849238 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837449    2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:23:16.849238 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837452    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:23:16.849238 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837455    2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:23:16.849238 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837457    2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:23:16.849238 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837460    2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:23:16.849238 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837462    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:23:16.849238 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837467    2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:23:16.849238 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837470    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:23:16.849238 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837474    2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:23:16.849238 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837477    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:23:16.849238 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837480    2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:23:16.849238 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837483    2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:23:16.849238 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837486    2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:23:16.849238 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837489    2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:23:16.849238 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837492    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:23:16.849238 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837495    2543 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:23:16.849238 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837498    2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:23:16.849238 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837501    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:23:16.849238 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837504    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:23:16.849829 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837507    2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:23:16.849829 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837510    2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:23:16.849829 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837512    2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:23:16.849829 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837515    2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:23:16.849829 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837518    2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:23:16.849829 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837520    2543 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:23:16.849829 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837523    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:23:16.849829 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837525    2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:23:16.849829 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837528    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:23:16.849829 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837530    2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:23:16.849829 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837533    2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:23:16.849829 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837536    2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:23:16.849829 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837538    2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:23:16.849829 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837541    2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:23:16.849829 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837543    2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:23:16.849829 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837546    2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:23:16.849829 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837549    2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:23:16.849829 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837553    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:23:16.849829 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837555    2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:23:16.850381 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837558    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:23:16.850381 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837561    2543 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:23:16.850381 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837565    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:23:16.850381 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837568    2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:23:16.850381 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837570    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:23:16.850381 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837573    2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:23:16.850381 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837576    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:23:16.850381 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837578    2543 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:23:16.850381 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837581    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:23:16.850381 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837583    2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:23:16.850381 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837586    2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:23:16.850381 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837589    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:23:16.850381 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837592    2543 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:23:16.850381 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837595    2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:23:16.850381 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837597    2543 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:23:16.850381 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837600    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:23:16.850381 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837603    2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:23:16.850381 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837605    2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:23:16.850381 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837608    2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:23:16.850381 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837610    2543 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:23:16.850865 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837613    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:23:16.850865 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837617    2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:23:16.850865 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837620    2543 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:23:16.850865 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837623    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:23:16.850865 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.837626    2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:23:16.850865 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.838156    2543 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:23:16.850865 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.845546    2543 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 17:23:16.850865 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.845564    2543 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 17:23:16.850865 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.845590    2543 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 17:23:16.850865 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.845599    2543 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 17:23:16.850865 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845693    2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:23:16.850865 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845701    2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:23:16.850865 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845706    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:23:16.850865 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845711    2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:23:16.851289 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845715    2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:23:16.851289 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845720    2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:23:16.851289 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845725    2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:23:16.851289 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845729    2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:23:16.851289 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845733    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:23:16.851289 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845738    2543 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:23:16.851289 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845742    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:23:16.851289 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845747    2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:23:16.851289 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845751    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:23:16.851289 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845756    2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:23:16.851289 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845760    2543 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:23:16.851289 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845765    2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:23:16.851289 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845770    2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:23:16.851289 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845774    2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:23:16.851289 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845779    2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:23:16.851289 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845783    2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:23:16.851289 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845787    2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:23:16.851289 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845792    2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:23:16.851289 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845796    2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:23:16.851289 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845801    2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:23:16.851831 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845805    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:23:16.851831 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845809    2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:23:16.851831 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845813    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:23:16.851831 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845817    2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:23:16.851831 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845821    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:23:16.851831 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845826    2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:23:16.851831 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845830    2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:23:16.851831 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845835    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:23:16.851831 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845839    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:23:16.851831 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845842    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:23:16.851831 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845847    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:23:16.851831 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845851    2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:23:16.851831 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845855    2543 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:23:16.851831 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845860    2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:23:16.851831 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845864    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:23:16.851831 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845869    2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:23:16.851831 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845873    2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:23:16.851831 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845877    2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:23:16.851831 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845881    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:23:16.852344 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845885    2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:23:16.852344 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845890    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:23:16.852344 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845894    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:23:16.852344 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845899    2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:23:16.852344 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845904    2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:23:16.852344 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845908    2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:23:16.852344 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845912    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:23:16.852344 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845916    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:23:16.852344 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845921    2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:23:16.852344 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845925    2543 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:23:16.852344 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845937    2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:23:16.852344 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845941    2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:23:16.852344 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845945    2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:23:16.852344 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845950    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:23:16.852344 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845954    2543 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:23:16.852344 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845958    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:23:16.852344 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845963    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:23:16.852344 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845967    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:23:16.852344 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845972    2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:23:16.852344 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845977    2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:23:16.852899 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845982    2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:23:16.852899 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845986    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:23:16.852899 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845991    2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:23:16.852899 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845995    2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:23:16.852899 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.845999    2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:23:16.852899 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846006    2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:23:16.852899 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846012    2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:23:16.852899 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846017    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:23:16.852899 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846021    2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:23:16.852899 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846026    2543 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:23:16.852899 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846030    2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:23:16.852899 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846034    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:23:16.852899 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846038    2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:23:16.852899 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846043    2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:23:16.852899 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846047    2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:23:16.852899 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846052    2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:23:16.852899 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846057    2543 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:23:16.852899 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846061    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:23:16.852899 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846065    2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:23:16.853605 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846069    2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:23:16.853605 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846076    2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:23:16.853605 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846082    2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:23:16.853605 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846087    2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:23:16.853605 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846092    2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:23:16.853605 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846096    2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:23:16.853605 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846100    2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:23:16.853605 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846105    2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:23:16.853605 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846109    2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:23:16.853605 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846113    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:23:16.853605 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846118    2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:23:16.853605 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846122    2543 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:23:16.853605 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846127    2543 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:23:16.853605 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846149    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:23:16.853605 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846154    2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:23:16.853605 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846159    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:23:16.853605 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846165    2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:23:16.854079 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.846175    2543 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:23:16.854079 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846367    2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:23:16.854079 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846375    2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:23:16.854079 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846381    2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:23:16.854079 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846385    2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:23:16.854079 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846390    2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:23:16.854079 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846394    2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:23:16.854079 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846398    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:23:16.854079 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846402    2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:23:16.854079 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846406    2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:23:16.854079 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846411    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:23:16.854079 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846415    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:23:16.854079 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846420    2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:23:16.854079 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846424    2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:23:16.854079 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846429    2543 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:23:16.854551 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846433    2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:23:16.854551 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846437    2543 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:23:16.854551 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846442    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:23:16.854551 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846447    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:23:16.854551 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846451    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:23:16.854551 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846455    2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:23:16.854551 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846459    2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:23:16.854551 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846464    2543 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:23:16.854551 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846468    2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:23:16.854551 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846472    2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:23:16.854551 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846476    2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:23:16.854551 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846481    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:23:16.854551 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846485    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:23:16.854551 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846490    2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:23:16.854551 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846495    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:23:16.854551 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846500    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:23:16.854551 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846504    2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:23:16.854551 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846508    2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:23:16.854551 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846512    2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:23:16.854551 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846516    2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:23:16.855066 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846521    2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:23:16.855066 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846525    2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:23:16.855066 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846529    2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:23:16.855066 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846533    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:23:16.855066 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846537    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:23:16.855066 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846541    2543 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:23:16.855066 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846545    2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:23:16.855066 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846550    2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:23:16.855066 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846554    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:23:16.855066 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846559    2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:23:16.855066 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846563    2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:23:16.855066 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846567    2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:23:16.855066 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846572    2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:23:16.855066 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846576    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:23:16.855066 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846580    2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:23:16.855066 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846583    2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:23:16.855066 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846588    2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:23:16.855066 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846592    2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:23:16.855066 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846596    2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:23:16.855066 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846600    2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:23:16.855778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846604    2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:23:16.855778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846609    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:23:16.855778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846615    2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:23:16.855778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846621    2543 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:23:16.855778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846626    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:23:16.855778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846630    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:23:16.855778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846635    2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:23:16.855778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846641    2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:23:16.855778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846645    2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:23:16.855778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846651    2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:23:16.855778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846657    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:23:16.855778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846662    2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:23:16.855778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846666    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:23:16.855778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846671    2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:23:16.855778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846676    2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:23:16.855778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846680    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:23:16.855778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846684    2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:23:16.855778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846688    2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:23:16.855778 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846692    2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:23:16.856470 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846697    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:23:16.856470 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846701    2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:23:16.856470 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846707    2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:23:16.856470 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846712    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:23:16.856470 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846717    2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:23:16.856470 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846722    2543 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:23:16.856470 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846727    2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:23:16.856470 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846732    2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:23:16.856470 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846736    2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:23:16.856470 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846740    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:23:16.856470 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846744    2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:23:16.856470 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846748    2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:23:16.856470 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846752    2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:23:16.856470 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846756    2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:23:16.856470 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846760    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:23:16.856470 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846765    2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:23:16.856470 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846769    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:23:16.856470 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846773    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:23:16.856470 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846777    2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:23:16.857014 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846781    2543 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:23:16.857014 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846786    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:23:16.857014 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846791    2543 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:23:16.857014 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846795    2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:23:16.857014 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846799    2543 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:23:16.857014 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846803    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:23:16.857014 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:16.846808    2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:23:16.857014 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.846816    2543 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:23:16.857014 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.846965    2543 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 17:23:16.857014 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.849542    2543 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 17:23:16.857014 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.850875    2543 server.go:1019] "Starting client certificate rotation"
Jul 28 17:23:16.857014 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.850978    2543 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:23:16.857014 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.851637    2543 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:23:16.857014 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.853877    2543 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:23:16.857729 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.857712    2543 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:23:16.865105 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.865086    2543 log.go:25] "Validated CRI v1 runtime API"
Jul 28 17:23:16.868459 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.868440    2543 log.go:25] "Validated CRI v1 image API"
Jul 28 17:23:16.869313 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.869291    2543 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 17:23:16.872910 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.872883    2543 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8dc6d47d-6225-4779-92c9-70eac0661896:/dev/nvme0n1p3 cda54114-e115-4cb3-86bb-2ed3e30545c1:/dev/nvme0n1p4]
Jul 28 17:23:16.872992 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.872909    2543 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 17:23:16.877878 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.877756    2543 manager.go:216] Machine: {Timestamp:2026-07-28 17:23:16.876650026 +0000 UTC m=+0.282923757 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105608 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b1c2b27435748e96f5195301ca784 SystemUUID:ec2b1c2b-2743-5748-e96f-5195301ca784 BootID:b8937ff1-55e2-4a4e-a3cf-d2206fac3e33 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:6f:5a:d1:72:fb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:6f:5a:d1:72:fb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:4b:40:04:11:4a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 17:23:16.877878 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.877866    2543 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 17:23:16.878055 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.877983    2543 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 17:23:16.880612 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.880580    2543 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 17:23:16.880763 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.880614    2543 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-177.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 17:23:16.881192 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.881182    2543 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 17:23:16.881232 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.881196    2543 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 17:23:16.881232 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.881203    2543 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:23:16.881232 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.881216    2543 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:23:16.882218 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.882206    2543 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 17:23:16.884455 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.884443    2543 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:23:16.884560 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.884550    2543 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 17:23:16.885107 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.885098    2543 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 17:23:16.885160 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.885112    2543 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 17:23:16.885160 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.885127    2543 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 17:23:16.885160 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.885150    2543 kubelet.go:398] "Adding apiserver pod source"
Jul 28 17:23:16.885160 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.885159    2543 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 17:23:16.885509 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.885494    2543 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:23:16.885562 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.885514    2543 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:23:16.888726 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.888704    2543 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 17:23:16.889217 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.889192    2543 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:23:16.890381 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.890366    2543 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 17:23:16.890426 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.890398    2543 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 17:23:16.891800 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.891787    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 17:23:16.891841 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.891805    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 17:23:16.891841 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.891811    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 17:23:16.891841 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.891817    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 17:23:16.891841 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.891826    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 17:23:16.891841 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.891833    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 17:23:16.891972 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.891845    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 17:23:16.891972 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.891852    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 17:23:16.891972 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.891859    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 17:23:16.891972 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.891865    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 17:23:16.891972 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.891872    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 17:23:16.891972 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.891877    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 17:23:16.892979 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.892969    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 17:23:16.893014 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.892981    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 17:23:16.896261 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.896243    2543 server.go:1268] "Started kubelet"
Jul 28 17:23:16.896380 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.896331    2543 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 17:23:16.896754 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.896707    2543 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 17:23:16.896859 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.896767    2543 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 17:23:16.897171 ip-10-0-135-177 systemd[1]: Started Kubernetes Kubelet.
Jul 28 17:23:16.897701 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.897672    2543 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 17:23:16.898438 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.898427    2543 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 17:23:16.898984 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.898956    2543 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-177.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 17:23:16.899066 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:16.898967    2543 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 17:23:16.899147 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:16.899087    2543 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-177.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 17:23:16.901455 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:16.901439    2543 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 17:23:16.903061 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.903046    2543 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 17:23:16.903061 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.903049    2543 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 17:23:16.903290 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:16.903270    2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-177.ec2.internal\" not found"
Jul 28 17:23:16.903750 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.903733    2543 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 17:23:16.903833 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.903755    2543 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 17:23:16.903833 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.903770    2543 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 17:23:16.903833 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.903778    2543 factory.go:55] Registering systemd factory
Jul 28 17:23:16.903833 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.903814    2543 factory.go:223] Registration of the systemd container factory successfully
Jul 28 17:23:16.903983 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.903880    2543 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 17:23:16.903983 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.903888    2543 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 17:23:16.904063 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.904044    2543 factory.go:153] Registering CRI-O factory
Jul 28 17:23:16.904063 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.904057    2543 factory.go:223] Registration of the crio container factory successfully
Jul 28 17:23:16.904223 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.904120    2543 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 17:23:16.904223 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.904159    2543 factory.go:103] Registering Raw factory
Jul 28 17:23:16.904223 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.904175    2543 manager.go:1196] Started watching for new ooms in manager
Jul 28 17:23:16.904706 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.904687    2543 manager.go:349] Starting recovery of all containers
Jul 28 17:23:16.906457 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:16.906425    2543 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-177.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 17:23:16.906909 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:16.906881    2543 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 17:23:16.907436 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:16.906481    2543 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-177.ec2.internal.18c683f677ed7137  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-177.ec2.internal,UID:ip-10-0-135-177.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-177.ec2.internal,},FirstTimestamp:2026-07-28 17:23:16.896223543 +0000 UTC m=+0.302497276,LastTimestamp:2026-07-28 17:23:16.896223543 +0000 UTC m=+0.302497276,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-177.ec2.internal,}"
Jul 28 17:23:16.917685 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.917668    2543 manager.go:354] Recovery completed
Jul 28 17:23:16.921715 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.921702    2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:23:16.923336 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.923313    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-177.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:23:16.923411 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.923353    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-177.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:23:16.923411 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.923368    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-177.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:23:16.923845 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.923824    2543 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 17:23:16.923845 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.923835    2543 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 17:23:16.923955 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.923853    2543 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:23:16.923955 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.923894    2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:23:16.924125 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.924107    2543 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6kxhj"
Jul 28 17:23:16.924620 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.924607    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-177.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:23:16.924667 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.924632    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-177.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:23:16.924667 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.924642    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-177.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:23:16.925376 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.925364    2543 policy_none.go:49] "None policy: Start"
Jul 28 17:23:16.925429 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.925381    2543 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 17:23:16.925429 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.925395    2543 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 17:23:16.925988 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:16.925909    2543 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-177.ec2.internal.18c683f6798b25d8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-177.ec2.internal,UID:ip-10-0-135-177.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-177.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-177.ec2.internal,},FirstTimestamp:2026-07-28 17:23:16.923336152 +0000 UTC m=+0.329609886,LastTimestamp:2026-07-28 17:23:16.923336152 +0000 UTC m=+0.329609886,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-177.ec2.internal,}"
Jul 28 17:23:16.926215 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.926204    2543 policy_none.go:47] "Start"
Jul 28 17:23:16.935896 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:16.935726    2543 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-177.ec2.internal.18c683f6798b82b2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-177.ec2.internal,UID:ip-10-0-135-177.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-177.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-177.ec2.internal,},FirstTimestamp:2026-07-28 17:23:16.923359922 +0000 UTC m=+0.329633655,LastTimestamp:2026-07-28 17:23:16.923359922 +0000 UTC m=+0.329633655,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-177.ec2.internal,}"
Jul 28 17:23:16.936422 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.936405    2543 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6kxhj"
Jul 28 17:23:16.938320 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:16.938258    2543 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-177.ec2.internal.18c683f6798bb7de  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-177.ec2.internal,UID:ip-10-0-135-177.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-177.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-177.ec2.internal,},FirstTimestamp:2026-07-28 17:23:16.923373534 +0000 UTC m=+0.329647266,LastTimestamp:2026-07-28 17:23:16.923373534 +0000 UTC m=+0.329647266,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-177.ec2.internal,}"
Jul 28 17:23:16.959676 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.959660    2543 manager.go:337] "Starting Device Plugin manager"
Jul 28 17:23:16.959752 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:16.959699    2543 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 17:23:16.959752 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.959709    2543 server.go:87] "Starting device plugin registration server"
Jul 28 17:23:16.959952 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.959941    2543 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 17:23:16.960007 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.959953    2543 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 17:23:16.960111 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.960095    2543 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 17:23:16.960339 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.960274    2543 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 17:23:16.960339 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:16.960288    2543 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 17:23:16.960701 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:16.960682    2543 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 17:23:16.960766 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:16.960736    2543 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-177.ec2.internal\" not found"
Jul 28 17:23:17.060945 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.060912    2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:23:17.061892 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.061872    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-177.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:23:17.062015 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.061904    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-177.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:23:17.062015 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.061914    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-177.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:23:17.062015 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.061947    2543 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-177.ec2.internal"
Jul 28 17:23:17.070702 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.070682    2543 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-177.ec2.internal"
Jul 28 17:23:17.079968 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:17.070708    2543 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-177.ec2.internal\": node \"ip-10-0-135-177.ec2.internal\" not found"
Jul 28 17:23:17.088969 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:17.088950    2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-177.ec2.internal\" not found"
Jul 28 17:23:17.144948 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.144917    2543 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 17:23:17.146118 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.146088    2543 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 17:23:17.146118 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.146122    2543 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 17:23:17.146290 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.146162    2543 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 17:23:17.146290 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:17.146198    2543 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 17:23:17.148779 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.148759    2543 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:23:17.189927 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:17.189893    2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-177.ec2.internal\" not found"
Jul 28 17:23:17.247480 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.247452    2543 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-177.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-177.ec2.internal"]
Jul 28 17:23:17.247547 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.247541    2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:23:17.248793 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.248768    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-177.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:23:17.248878 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.248805    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-177.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:23:17.248878 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.248816    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-177.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:23:17.250055 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.250042    2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:23:17.250193 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.250179    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-177.ec2.internal"
Jul 28 17:23:17.250262 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.250215    2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:23:17.250868 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.250852    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-177.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:23:17.250929 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.250891    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-177.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:23:17.250929 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.250902    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-177.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:23:17.251659 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.251629    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-177.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:23:17.251756 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.251664    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-177.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:23:17.251756 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.251683    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-177.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:23:17.252869 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.252847    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-177.ec2.internal"
Jul 28 17:23:17.252961 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.252876    2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:23:17.253637 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.253620    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-177.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:23:17.253727 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.253643    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-177.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:23:17.253727 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.253652    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-177.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:23:17.273916 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:17.273899    2543 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-177.ec2.internal\" not found" node="ip-10-0-135-177.ec2.internal"
Jul 28 17:23:17.278542 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:17.278523    2543 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-177.ec2.internal\" not found" node="ip-10-0-135-177.ec2.internal"
Jul 28 17:23:17.290082 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:17.290064    2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-177.ec2.internal\" not found"
Jul 28 17:23:17.305320 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.305294    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a0b77ac91bc5dbb86ffeab81485172d7-config\") pod \"kube-apiserver-proxy-ip-10-0-135-177.ec2.internal\" (UID: \"a0b77ac91bc5dbb86ffeab81485172d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-177.ec2.internal"
Jul 28 17:23:17.305401 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.305324    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1bb5c2c1c39b3c131016259171f34c67-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-177.ec2.internal\" (UID: \"1bb5c2c1c39b3c131016259171f34c67\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-177.ec2.internal"
Jul 28 17:23:17.305401 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.305342    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bb5c2c1c39b3c131016259171f34c67-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-177.ec2.internal\" (UID: \"1bb5c2c1c39b3c131016259171f34c67\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-177.ec2.internal"
Jul 28 17:23:17.390546 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:17.390465    2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-177.ec2.internal\" not found"
Jul 28 17:23:17.405954 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.405931    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a0b77ac91bc5dbb86ffeab81485172d7-config\") pod \"kube-apiserver-proxy-ip-10-0-135-177.ec2.internal\" (UID: \"a0b77ac91bc5dbb86ffeab81485172d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-177.ec2.internal"
Jul 28 17:23:17.406060 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.406007    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1bb5c2c1c39b3c131016259171f34c67-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-177.ec2.internal\" (UID: \"1bb5c2c1c39b3c131016259171f34c67\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-177.ec2.internal"
Jul 28 17:23:17.406060 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.406025    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a0b77ac91bc5dbb86ffeab81485172d7-config\") pod \"kube-apiserver-proxy-ip-10-0-135-177.ec2.internal\" (UID: \"a0b77ac91bc5dbb86ffeab81485172d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-177.ec2.internal"
Jul 28 17:23:17.406060 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.406029    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1bb5c2c1c39b3c131016259171f34c67-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-177.ec2.internal\" (UID: \"1bb5c2c1c39b3c131016259171f34c67\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-177.ec2.internal"
Jul 28 17:23:17.406249 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.406072    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bb5c2c1c39b3c131016259171f34c67-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-177.ec2.internal\" (UID: \"1bb5c2c1c39b3c131016259171f34c67\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-177.ec2.internal"
Jul 28 17:23:17.406249 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.406093    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bb5c2c1c39b3c131016259171f34c67-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-177.ec2.internal\" (UID: \"1bb5c2c1c39b3c131016259171f34c67\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-177.ec2.internal"
Jul 28 17:23:17.497085 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:17.497039    2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-177.ec2.internal\" not found"
Jul 28 17:23:17.580961 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.580928    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-177.ec2.internal"
Jul 28 17:23:17.582067 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.582049    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-177.ec2.internal"
Jul 28 17:23:17.598039 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:17.598017    2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-177.ec2.internal\" not found"
Jul 28 17:23:17.705278 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:17.705174    2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-177.ec2.internal\" not found"
Jul 28 17:23:17.721641 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.721608    2543 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:23:17.756962 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.756938    2543 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:23:17.806407 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.806369    2543 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-177.ec2.internal"
Jul 28 17:23:17.826915 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.826893    2543 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:23:17.828237 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.828223    2543 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-177.ec2.internal"
Jul 28 17:23:17.841881 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.841862    2543 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:23:17.850694 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.850673    2543 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 17:23:17.850790 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.850773    2543 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:23:17.850837 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.850817    2543 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:23:17.850879 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.850827    2543 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:23:17.887768 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.887747    2543 apiserver.go:52] "Watching apiserver"
Jul 28 17:23:17.903450 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.903423    2543 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 17:23:17.903611 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.903595    2543 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 17:23:17.904089 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.904068    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-ljfhd","openshift-multus/multus-st8s2","openshift-multus/network-metrics-daemon-sjtrc","openshift-network-diagnostics/network-check-target-9c8w8","kube-system/global-pull-secret-syncer-jtb46","kube-system/kube-apiserver-proxy-ip-10-0-135-177.ec2.internal","openshift-cluster-node-tuning-operator/tuned-9chjn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-177.ec2.internal","openshift-network-operator/iptables-alerter-7pp8b","openshift-ovn-kubernetes/ovnkube-node-64lk7","kube-system/konnectivity-agent-7qw2x","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jmn4","openshift-dns/node-resolver-kvtkx","openshift-image-registry/node-ca-4kw4j"]
Jul 28 17:23:17.905616 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.905598    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:17.906561 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.906544    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kvtkx"
Jul 28 17:23:17.907721 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.907702    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:23:17.907813 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:17.907774    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjtrc" podUID="a2c1c169-0fd2-4e73-a497-0b69fe85d11f"
Jul 28 17:23:17.908825 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.908809    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9c8w8"
Jul 28 17:23:17.908899 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:17.908863    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9c8w8" podUID="2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4"
Jul 28 17:23:17.909608 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.909594    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 17:23:17.909757 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.909741    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:17.909847 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:17.909831    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtb46" podUID="145627c2-9b95-4251-9f16-2920555de2fa"
Jul 28 17:23:17.910717 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.910699    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4kw4j"
Jul 28 17:23:17.911058 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.911044    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 17:23:17.911291 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.911275    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zs6l6\""
Jul 28 17:23:17.911363 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.911313    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:23:17.911587 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.911570    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 17:23:17.911800 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.911783    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7pp8b"
Jul 28 17:23:17.911849 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.911835    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-z9dgz\""
Jul 28 17:23:17.913083 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.913063    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ljfhd"
Jul 28 17:23:17.914290 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.914274    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-st8s2"
Jul 28 17:23:17.915344 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.915330    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:17.916997 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.916972    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7qw2x"
Jul 28 17:23:17.919183 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.918696    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jmn4"
Jul 28 17:23:17.921147 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.921122    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 17:23:17.921249 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.921175    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 17:23:17.921249 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.921190    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 17:23:17.921431 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.921415    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 17:23:17.921503 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.921448    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 17:23:17.921632 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.921595    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 17:23:17.921632 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.921624    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f424t\""
Jul 28 17:23:17.921826 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.921670    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 17:23:17.921826 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.921698    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bmc4t\""
Jul 28 17:23:17.921826 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.921712    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 17:23:17.922008 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.921984    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 17:23:17.922138 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.922121    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rk7lk\""
Jul 28 17:23:17.922230 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.922201    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:23:17.922230 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.922220    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 17:23:17.922335 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.922320    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 17:23:17.922373 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.922350    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gk5j7\""
Jul 28 17:23:17.923306 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.923288    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 17:23:17.923401 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.923367    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 17:23:17.923466 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.923447    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 17:23:17.923529 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.923484    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 17:23:17.923529 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.923507    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 17:23:17.924201 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.923589    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pmflc\""
Jul 28 17:23:17.924607 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.924581    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n7647\""
Jul 28 17:23:17.925148 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.925124    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 17:23:17.925316 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.925208    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 17:23:17.925316 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.925135    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 17:23:17.925471 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.925351    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jtx85\""
Jul 28 17:23:17.925471 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.925400    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 17:23:17.928029 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.928011    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 17:23:17.928752 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.928736    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 17:23:17.935499 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.935481    2543 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:23:17.940035 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.939985    2543 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 17:18:17 +0000 UTC" deadline="2028-02-22 03:29:30.095140231 +0000 UTC"
Jul 28 17:23:17.940116 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.940036    2543 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13762h6m12.155108847s"
Jul 28 17:23:17.973896 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.973870    2543 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ffwjj"
Jul 28 17:23:17.985852 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:17.985835    2543 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ffwjj"
Jul 28 17:23:18.004816 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.004803    2543 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 17:23:18.009694 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.009676    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/849d3b0f-9f02-4f4d-8811-13794309755a-system-cni-dir\") pod \"multus-additional-cni-plugins-ljfhd\" (UID: \"849d3b0f-9f02-4f4d-8811-13794309755a\") " pod="openshift-multus/multus-additional-cni-plugins-ljfhd"
Jul 28 17:23:18.009827 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.009700    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-host-cni-netd\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.009827 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.009718    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmrmg\" (UniqueName: \"kubernetes.io/projected/ea4785ea-08ca-45f8-a8bc-42524b8d81b4-kube-api-access-mmrmg\") pod \"node-ca-4kw4j\" (UID: \"ea4785ea-08ca-45f8-a8bc-42524b8d81b4\") " pod="openshift-image-registry/node-ca-4kw4j"
Jul 28 17:23:18.009827 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.009740    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-host-var-lib-cni-multus\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.009827 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.009778    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-multus-conf-dir\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.009827 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.009807    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-systemd-units\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.010092 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.009842    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-var-lib-openvswitch\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.010092 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.009865    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-etc-openvswitch\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.010092 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.009918    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w89v\" (UniqueName: \"kubernetes.io/projected/2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4-kube-api-access-7w89v\") pod \"network-check-target-9c8w8\" (UID: \"2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4\") " pod="openshift-network-diagnostics/network-check-target-9c8w8"
Jul 28 17:23:18.010092 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.009938    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/145627c2-9b95-4251-9f16-2920555de2fa-original-pull-secret\") pod \"global-pull-secret-syncer-jtb46\" (UID: \"145627c2-9b95-4251-9f16-2920555de2fa\") " pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:18.010092 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.009956    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/849d3b0f-9f02-4f4d-8811-13794309755a-cnibin\") pod \"multus-additional-cni-plugins-ljfhd\" (UID: \"849d3b0f-9f02-4f4d-8811-13794309755a\") " pod="openshift-multus/multus-additional-cni-plugins-ljfhd"
Jul 28 17:23:18.010092 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.009978    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/afa69cf5-92d6-4d5b-b15c-242b555e08aa-sys-fs\") pod \"aws-ebs-csi-driver-node-6jmn4\" (UID: \"afa69cf5-92d6-4d5b-b15c-242b555e08aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jmn4"
Jul 28 17:23:18.010092 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.010015    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/afa69cf5-92d6-4d5b-b15c-242b555e08aa-etc-selinux\") pod \"aws-ebs-csi-driver-node-6jmn4\" (UID: \"afa69cf5-92d6-4d5b-b15c-242b555e08aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jmn4"
Jul 28 17:23:18.010092 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.010052    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be7a3829-e798-4b35-8489-c872935568a8-cni-binary-copy\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.010526 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.010106    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-multus-socket-dir-parent\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.010526 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.010175    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k56l4\" (UniqueName: \"kubernetes.io/projected/be7a3829-e798-4b35-8489-c872935568a8-kube-api-access-k56l4\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.010526 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.010211    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/89471f5d-8eda-4a72-ab86-5f2290e75842-agent-certs\") pod \"konnectivity-agent-7qw2x\" (UID: \"89471f5d-8eda-4a72-ab86-5f2290e75842\") " pod="kube-system/konnectivity-agent-7qw2x"
Jul 28 17:23:18.010526 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.010237    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvch5\" (UniqueName: \"kubernetes.io/projected/afa69cf5-92d6-4d5b-b15c-242b555e08aa-kube-api-access-wvch5\") pod \"aws-ebs-csi-driver-node-6jmn4\" (UID: \"afa69cf5-92d6-4d5b-b15c-242b555e08aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jmn4"
Jul 28 17:23:18.010526 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.010262    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-etc-kubernetes\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.010526 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.010292    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/afa69cf5-92d6-4d5b-b15c-242b555e08aa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6jmn4\" (UID: \"afa69cf5-92d6-4d5b-b15c-242b555e08aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jmn4"
Jul 28 17:23:18.010526 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.010328    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-etc-sysctl-conf\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.010526 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.010350    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbp6g\" (UniqueName: \"kubernetes.io/projected/849d3b0f-9f02-4f4d-8811-13794309755a-kube-api-access-rbp6g\") pod \"multus-additional-cni-plugins-ljfhd\" (UID: \"849d3b0f-9f02-4f4d-8811-13794309755a\") " pod="openshift-multus/multus-additional-cni-plugins-ljfhd"
Jul 28 17:23:18.010526 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.010366    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-run\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.010526 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.010388    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/145627c2-9b95-4251-9f16-2920555de2fa-dbus\") pod \"global-pull-secret-syncer-jtb46\" (UID: \"145627c2-9b95-4251-9f16-2920555de2fa\") " pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:18.010526 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.010412    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d50b403c-3842-4b2f-a314-e65583a0b225-host-slash\") pod \"iptables-alerter-7pp8b\" (UID: \"d50b403c-3842-4b2f-a314-e65583a0b225\") " pod="openshift-network-operator/iptables-alerter-7pp8b"
Jul 28 17:23:18.010526 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.010433    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj9lr\" (UniqueName: \"kubernetes.io/projected/030014cc-fe9d-463d-a819-083f3396c15f-kube-api-access-jj9lr\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.010526 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.010460    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3acecb5c-18d4-4260-aa3e-8758184e6733-hosts-file\") pod \"node-resolver-kvtkx\" (UID: \"3acecb5c-18d4-4260-aa3e-8758184e6733\") " pod="openshift-dns/node-resolver-kvtkx"
Jul 28 17:23:18.010526 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.010478    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bb8012ba-acac-4f19-823c-2339cc6fc563-env-overrides\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.010526 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.010502    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/849d3b0f-9f02-4f4d-8811-13794309755a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ljfhd\" (UID: \"849d3b0f-9f02-4f4d-8811-13794309755a\") " pod="openshift-multus/multus-additional-cni-plugins-ljfhd"
Jul 28 17:23:18.011184 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.010524    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/afa69cf5-92d6-4d5b-b15c-242b555e08aa-socket-dir\") pod \"aws-ebs-csi-driver-node-6jmn4\" (UID: \"afa69cf5-92d6-4d5b-b15c-242b555e08aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jmn4"
Jul 28 17:23:18.011184 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.010563    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-var-lib-kubelet\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.011184 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.010599    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltckq\" (UniqueName: \"kubernetes.io/projected/d50b403c-3842-4b2f-a314-e65583a0b225-kube-api-access-ltckq\") pod \"iptables-alerter-7pp8b\" (UID: \"d50b403c-3842-4b2f-a314-e65583a0b225\") " pod="openshift-network-operator/iptables-alerter-7pp8b"
Jul 28 17:23:18.011184 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.010623    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3acecb5c-18d4-4260-aa3e-8758184e6733-tmp-dir\") pod \"node-resolver-kvtkx\" (UID: \"3acecb5c-18d4-4260-aa3e-8758184e6733\") " pod="openshift-dns/node-resolver-kvtkx"
Jul 28 17:23:18.011184 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.010650    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-host-var-lib-kubelet\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.011184 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.010698    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-hostroot\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.011184 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.010786    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-etc-sysctl-d\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.011184 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.010823    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-host-run-ovn-kubernetes\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.011184 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.010852    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-etc-modprobe-d\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.011184 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.010881    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-host-run-netns\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.011184 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.010908    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d50b403c-3842-4b2f-a314-e65583a0b225-iptables-alerter-script\") pod \"iptables-alerter-7pp8b\" (UID: \"d50b403c-3842-4b2f-a314-e65583a0b225\") " pod="openshift-network-operator/iptables-alerter-7pp8b"
Jul 28 17:23:18.011184 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.010934    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-system-cni-dir\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.011184 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.010977    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/be7a3829-e798-4b35-8489-c872935568a8-multus-daemon-config\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.011184 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.011018    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-etc-systemd\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.011184 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.011044    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/849d3b0f-9f02-4f4d-8811-13794309755a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ljfhd\" (UID: \"849d3b0f-9f02-4f4d-8811-13794309755a\") " pod="openshift-multus/multus-additional-cni-plugins-ljfhd"
Jul 28 17:23:18.011184 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.011078    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-run-systemd\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.011855 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.011101    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-run-ovn\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.011855 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.011126    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/afa69cf5-92d6-4d5b-b15c-242b555e08aa-device-dir\") pod \"aws-ebs-csi-driver-node-6jmn4\" (UID: \"afa69cf5-92d6-4d5b-b15c-242b555e08aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jmn4"
Jul 28 17:23:18.011855 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.011163    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/849d3b0f-9f02-4f4d-8811-13794309755a-os-release\") pod \"multus-additional-cni-plugins-ljfhd\" (UID: \"849d3b0f-9f02-4f4d-8811-13794309755a\") " pod="openshift-multus/multus-additional-cni-plugins-ljfhd"
Jul 28 17:23:18.011855 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.011185    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/849d3b0f-9f02-4f4d-8811-13794309755a-cni-binary-copy\") pod \"multus-additional-cni-plugins-ljfhd\" (UID: \"849d3b0f-9f02-4f4d-8811-13794309755a\") " pod="openshift-multus/multus-additional-cni-plugins-ljfhd"
Jul 28 17:23:18.011855 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.011229    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-node-log\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.011855 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.011249    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea4785ea-08ca-45f8-a8bc-42524b8d81b4-host\") pod \"node-ca-4kw4j\" (UID: \"ea4785ea-08ca-45f8-a8bc-42524b8d81b4\") " pod="openshift-image-registry/node-ca-4kw4j"
Jul 28 17:23:18.011855 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.011283    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/030014cc-fe9d-463d-a819-083f3396c15f-etc-tuned\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.011855 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.011307    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c1c169-0fd2-4e73-a497-0b69fe85d11f-metrics-certs\") pod \"network-metrics-daemon-sjtrc\" (UID: \"a2c1c169-0fd2-4e73-a497-0b69fe85d11f\") " pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:23:18.011855 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.011656    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm2v8\" (UniqueName: \"kubernetes.io/projected/a2c1c169-0fd2-4e73-a497-0b69fe85d11f-kube-api-access-fm2v8\") pod \"network-metrics-daemon-sjtrc\" (UID: \"a2c1c169-0fd2-4e73-a497-0b69fe85d11f\") " pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:23:18.011855 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.011701    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-multus-cni-dir\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.011855 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.011747    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-host-run-k8s-cni-cncf-io\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.011855 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.011809    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-host-run-netns\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.012435 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.011878    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-host-run-multus-certs\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.014585 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.011928    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/afa69cf5-92d6-4d5b-b15c-242b555e08aa-registration-dir\") pod \"aws-ebs-csi-driver-node-6jmn4\" (UID: \"afa69cf5-92d6-4d5b-b15c-242b555e08aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jmn4"
Jul 28 17:23:18.014585 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.014019    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-lib-modules\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.014585 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.014047    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/849d3b0f-9f02-4f4d-8811-13794309755a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ljfhd\" (UID: \"849d3b0f-9f02-4f4d-8811-13794309755a\") " pod="openshift-multus/multus-additional-cni-plugins-ljfhd"
Jul 28 17:23:18.014585 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.014079    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-host-slash\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.014585 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.014109    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-etc-kubernetes\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.014585 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.014138    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-host-kubelet\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.014585 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.014182    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-run-openvswitch\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.014585 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.014208    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-log-socket\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.014585 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.014240    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.014585 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.014308    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bb8012ba-acac-4f19-823c-2339cc6fc563-ovnkube-config\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.014585 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.014352    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bb8012ba-acac-4f19-823c-2339cc6fc563-ovnkube-script-lib\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.014585 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.014389    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/145627c2-9b95-4251-9f16-2920555de2fa-kubelet-config\") pod \"global-pull-secret-syncer-jtb46\" (UID: \"145627c2-9b95-4251-9f16-2920555de2fa\") " pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:18.014585 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.014417    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-cnibin\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.014585 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.014442    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-host-var-lib-cni-bin\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.014585 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.014488    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-host-cni-bin\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.014585 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.014555    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lmtd\" (UniqueName: \"kubernetes.io/projected/bb8012ba-acac-4f19-823c-2339cc6fc563-kube-api-access-7lmtd\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.015704 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.014619    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgsrr\" (UniqueName: \"kubernetes.io/projected/3acecb5c-18d4-4260-aa3e-8758184e6733-kube-api-access-qgsrr\") pod \"node-resolver-kvtkx\" (UID: \"3acecb5c-18d4-4260-aa3e-8758184e6733\") " pod="openshift-dns/node-resolver-kvtkx"
Jul 28 17:23:18.015704 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.014657    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/89471f5d-8eda-4a72-ab86-5f2290e75842-konnectivity-ca\") pod \"konnectivity-agent-7qw2x\" (UID: \"89471f5d-8eda-4a72-ab86-5f2290e75842\") " pod="kube-system/konnectivity-agent-7qw2x"
Jul 28 17:23:18.015704 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.014683    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-os-release\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.015704 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.014743    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-etc-sysconfig\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.015704 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.014808    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-host\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.015704 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.014841    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/030014cc-fe9d-463d-a819-083f3396c15f-tmp\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.015704 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.014861    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb8012ba-acac-4f19-823c-2339cc6fc563-ovn-node-metrics-cert\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.015704 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.014930    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ea4785ea-08ca-45f8-a8bc-42524b8d81b4-serviceca\") pod \"node-ca-4kw4j\" (UID: \"ea4785ea-08ca-45f8-a8bc-42524b8d81b4\") " pod="openshift-image-registry/node-ca-4kw4j"
Jul 28 17:23:18.015704 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.014962    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-sys\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.044465 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.044440    2543 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:23:18.116237 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116208    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be7a3829-e798-4b35-8489-c872935568a8-cni-binary-copy\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.116237 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116242    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-multus-socket-dir-parent\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.116445 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116258    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k56l4\" (UniqueName: \"kubernetes.io/projected/be7a3829-e798-4b35-8489-c872935568a8-kube-api-access-k56l4\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.116445 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116275    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/89471f5d-8eda-4a72-ab86-5f2290e75842-agent-certs\") pod \"konnectivity-agent-7qw2x\" (UID: \"89471f5d-8eda-4a72-ab86-5f2290e75842\") " pod="kube-system/konnectivity-agent-7qw2x"
Jul 28 17:23:18.116445 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116291    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvch5\" (UniqueName: \"kubernetes.io/projected/afa69cf5-92d6-4d5b-b15c-242b555e08aa-kube-api-access-wvch5\") pod \"aws-ebs-csi-driver-node-6jmn4\" (UID: \"afa69cf5-92d6-4d5b-b15c-242b555e08aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jmn4"
Jul 28 17:23:18.116445 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116316    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-etc-kubernetes\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.116445 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116334    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/afa69cf5-92d6-4d5b-b15c-242b555e08aa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6jmn4\" (UID: \"afa69cf5-92d6-4d5b-b15c-242b555e08aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jmn4"
Jul 28 17:23:18.116445 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116360    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-etc-sysctl-conf\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.116445 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116364    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-multus-socket-dir-parent\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.116445 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116396    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-etc-kubernetes\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.116445 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116388    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbp6g\" (UniqueName: \"kubernetes.io/projected/849d3b0f-9f02-4f4d-8811-13794309755a-kube-api-access-rbp6g\") pod \"multus-additional-cni-plugins-ljfhd\" (UID: \"849d3b0f-9f02-4f4d-8811-13794309755a\") " pod="openshift-multus/multus-additional-cni-plugins-ljfhd"
Jul 28 17:23:18.116445 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116445    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-run\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.116965 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116472    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/145627c2-9b95-4251-9f16-2920555de2fa-dbus\") pod \"global-pull-secret-syncer-jtb46\" (UID: \"145627c2-9b95-4251-9f16-2920555de2fa\") " pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:18.116965 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116494    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/afa69cf5-92d6-4d5b-b15c-242b555e08aa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6jmn4\" (UID: \"afa69cf5-92d6-4d5b-b15c-242b555e08aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jmn4"
Jul 28 17:23:18.116965 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116539    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-run\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.116965 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116587    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d50b403c-3842-4b2f-a314-e65583a0b225-host-slash\") pod \"iptables-alerter-7pp8b\" (UID: \"d50b403c-3842-4b2f-a314-e65583a0b225\") " pod="openshift-network-operator/iptables-alerter-7pp8b"
Jul 28 17:23:18.116965 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116622    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj9lr\" (UniqueName: \"kubernetes.io/projected/030014cc-fe9d-463d-a819-083f3396c15f-kube-api-access-jj9lr\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.116965 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116646    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-etc-sysctl-conf\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.116965 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116648    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3acecb5c-18d4-4260-aa3e-8758184e6733-hosts-file\") pod \"node-resolver-kvtkx\" (UID: \"3acecb5c-18d4-4260-aa3e-8758184e6733\") " pod="openshift-dns/node-resolver-kvtkx"
Jul 28 17:23:18.116965 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116668    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d50b403c-3842-4b2f-a314-e65583a0b225-host-slash\") pod \"iptables-alerter-7pp8b\" (UID: \"d50b403c-3842-4b2f-a314-e65583a0b225\") " pod="openshift-network-operator/iptables-alerter-7pp8b"
Jul 28 17:23:18.116965 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116688    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3acecb5c-18d4-4260-aa3e-8758184e6733-hosts-file\") pod \"node-resolver-kvtkx\" (UID: \"3acecb5c-18d4-4260-aa3e-8758184e6733\") " pod="openshift-dns/node-resolver-kvtkx"
Jul 28 17:23:18.116965 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116690    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bb8012ba-acac-4f19-823c-2339cc6fc563-env-overrides\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.116965 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116712    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/145627c2-9b95-4251-9f16-2920555de2fa-dbus\") pod \"global-pull-secret-syncer-jtb46\" (UID: \"145627c2-9b95-4251-9f16-2920555de2fa\") " pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:18.116965 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116713    2543 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 17:23:18.116965 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116730    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/849d3b0f-9f02-4f4d-8811-13794309755a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ljfhd\" (UID: \"849d3b0f-9f02-4f4d-8811-13794309755a\") " pod="openshift-multus/multus-additional-cni-plugins-ljfhd"
Jul 28 17:23:18.116965 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116751    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/afa69cf5-92d6-4d5b-b15c-242b555e08aa-socket-dir\") pod \"aws-ebs-csi-driver-node-6jmn4\" (UID: \"afa69cf5-92d6-4d5b-b15c-242b555e08aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jmn4"
Jul 28 17:23:18.116965 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116780    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-var-lib-kubelet\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.116965 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116805    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltckq\" (UniqueName: \"kubernetes.io/projected/d50b403c-3842-4b2f-a314-e65583a0b225-kube-api-access-ltckq\") pod \"iptables-alerter-7pp8b\" (UID: \"d50b403c-3842-4b2f-a314-e65583a0b225\") " pod="openshift-network-operator/iptables-alerter-7pp8b"
Jul 28 17:23:18.116965 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116830    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3acecb5c-18d4-4260-aa3e-8758184e6733-tmp-dir\") pod \"node-resolver-kvtkx\" (UID: \"3acecb5c-18d4-4260-aa3e-8758184e6733\") " pod="openshift-dns/node-resolver-kvtkx"
Jul 28 17:23:18.116965 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116855    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-host-var-lib-kubelet\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.117912 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116873    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/afa69cf5-92d6-4d5b-b15c-242b555e08aa-socket-dir\") pod \"aws-ebs-csi-driver-node-6jmn4\" (UID: \"afa69cf5-92d6-4d5b-b15c-242b555e08aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jmn4"
Jul 28 17:23:18.117912 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116896    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-host-var-lib-kubelet\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.117912 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.116928    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-var-lib-kubelet\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.117912 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117036    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/be7a3829-e798-4b35-8489-c872935568a8-cni-binary-copy\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.117912 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117040    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-hostroot\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.117912 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117094    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-etc-sysctl-d\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.117912 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117114    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-host-run-ovn-kubernetes\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.117912 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117131    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-etc-modprobe-d\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.117912 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117154    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-host-run-netns\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.117912 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117177    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3acecb5c-18d4-4260-aa3e-8758184e6733-tmp-dir\") pod \"node-resolver-kvtkx\" (UID: \"3acecb5c-18d4-4260-aa3e-8758184e6733\") " pod="openshift-dns/node-resolver-kvtkx"
Jul 28 17:23:18.117912 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117188    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-host-run-ovn-kubernetes\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.117912 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117070    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-hostroot\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.117912 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117179    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d50b403c-3842-4b2f-a314-e65583a0b225-iptables-alerter-script\") pod \"iptables-alerter-7pp8b\" (UID: \"d50b403c-3842-4b2f-a314-e65583a0b225\") " pod="openshift-network-operator/iptables-alerter-7pp8b"
Jul 28 17:23:18.117912 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117237    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bb8012ba-acac-4f19-823c-2339cc6fc563-env-overrides\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.117912 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117247    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-etc-sysctl-d\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.117912 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117247    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-host-run-netns\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.117912 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117281    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-system-cni-dir\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.117912 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117312    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/be7a3829-e798-4b35-8489-c872935568a8-multus-daemon-config\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.120447 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117338    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-etc-systemd\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.120447 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117366    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/849d3b0f-9f02-4f4d-8811-13794309755a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ljfhd\" (UID: \"849d3b0f-9f02-4f4d-8811-13794309755a\") " pod="openshift-multus/multus-additional-cni-plugins-ljfhd"
Jul 28 17:23:18.120447 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117384    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-system-cni-dir\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.120447 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117395    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-run-systemd\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.120447 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117402    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/849d3b0f-9f02-4f4d-8811-13794309755a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ljfhd\" (UID: \"849d3b0f-9f02-4f4d-8811-13794309755a\") " pod="openshift-multus/multus-additional-cni-plugins-ljfhd"
Jul 28 17:23:18.120447 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117420    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-run-ovn\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.120447 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117430    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-etc-systemd\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.120447 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117434    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-etc-modprobe-d\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.120447 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117456    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/afa69cf5-92d6-4d5b-b15c-242b555e08aa-device-dir\") pod \"aws-ebs-csi-driver-node-6jmn4\" (UID: \"afa69cf5-92d6-4d5b-b15c-242b555e08aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jmn4"
Jul 28 17:23:18.120447 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117465    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-run-systemd\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.120447 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117482    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/849d3b0f-9f02-4f4d-8811-13794309755a-os-release\") pod \"multus-additional-cni-plugins-ljfhd\" (UID: \"849d3b0f-9f02-4f4d-8811-13794309755a\") " pod="openshift-multus/multus-additional-cni-plugins-ljfhd"
Jul 28 17:23:18.120447 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117511    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/afa69cf5-92d6-4d5b-b15c-242b555e08aa-device-dir\") pod \"aws-ebs-csi-driver-node-6jmn4\" (UID: \"afa69cf5-92d6-4d5b-b15c-242b555e08aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jmn4"
Jul 28 17:23:18.120447 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117484    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-run-ovn\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.120447 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117544    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/849d3b0f-9f02-4f4d-8811-13794309755a-cni-binary-copy\") pod \"multus-additional-cni-plugins-ljfhd\" (UID: \"849d3b0f-9f02-4f4d-8811-13794309755a\") " pod="openshift-multus/multus-additional-cni-plugins-ljfhd"
Jul 28 17:23:18.120447 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117561    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/849d3b0f-9f02-4f4d-8811-13794309755a-os-release\") pod \"multus-additional-cni-plugins-ljfhd\" (UID: \"849d3b0f-9f02-4f4d-8811-13794309755a\") " pod="openshift-multus/multus-additional-cni-plugins-ljfhd"
Jul 28 17:23:18.120447 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117619    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-node-log\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.120447 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117649    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea4785ea-08ca-45f8-a8bc-42524b8d81b4-host\") pod \"node-ca-4kw4j\" (UID: \"ea4785ea-08ca-45f8-a8bc-42524b8d81b4\") " pod="openshift-image-registry/node-ca-4kw4j"
Jul 28 17:23:18.121320 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117692    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/030014cc-fe9d-463d-a819-083f3396c15f-etc-tuned\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.121320 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117717    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c1c169-0fd2-4e73-a497-0b69fe85d11f-metrics-certs\") pod \"network-metrics-daemon-sjtrc\" (UID: \"a2c1c169-0fd2-4e73-a497-0b69fe85d11f\") " pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:23:18.121320 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117748    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm2v8\" (UniqueName: \"kubernetes.io/projected/a2c1c169-0fd2-4e73-a497-0b69fe85d11f-kube-api-access-fm2v8\") pod \"network-metrics-daemon-sjtrc\" (UID: \"a2c1c169-0fd2-4e73-a497-0b69fe85d11f\") " pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:23:18.121320 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117778    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-multus-cni-dir\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.121320 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117804    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-host-run-k8s-cni-cncf-io\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.121320 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117819    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d50b403c-3842-4b2f-a314-e65583a0b225-iptables-alerter-script\") pod \"iptables-alerter-7pp8b\" (UID: \"d50b403c-3842-4b2f-a314-e65583a0b225\") " pod="openshift-network-operator/iptables-alerter-7pp8b"
Jul 28 17:23:18.121320 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117851    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-host-run-netns\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.121320 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:18.117916    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:23:18.121320 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117926    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/be7a3829-e798-4b35-8489-c872935568a8-multus-daemon-config\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.121320 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117934    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/849d3b0f-9f02-4f4d-8811-13794309755a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ljfhd\" (UID: \"849d3b0f-9f02-4f4d-8811-13794309755a\") " pod="openshift-multus/multus-additional-cni-plugins-ljfhd"
Jul 28 17:23:18.121320 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117959    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-host-run-multus-certs\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.121320 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:18.117998    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2c1c169-0fd2-4e73-a497-0b69fe85d11f-metrics-certs podName:a2c1c169-0fd2-4e73-a497-0b69fe85d11f nodeName:}" failed. No retries permitted until 2026-07-28 17:23:18.617972096 +0000 UTC m=+2.024245840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2c1c169-0fd2-4e73-a497-0b69fe85d11f-metrics-certs") pod "network-metrics-daemon-sjtrc" (UID: "a2c1c169-0fd2-4e73-a497-0b69fe85d11f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:23:18.121320 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118005    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-host-run-k8s-cni-cncf-io\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.121320 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118017    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-multus-cni-dir\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.121320 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.117916    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-host-run-multus-certs\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.121320 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118048    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea4785ea-08ca-45f8-a8bc-42524b8d81b4-host\") pod \"node-ca-4kw4j\" (UID: \"ea4785ea-08ca-45f8-a8bc-42524b8d81b4\") " pod="openshift-image-registry/node-ca-4kw4j"
Jul 28 17:23:18.121320 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118060    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/afa69cf5-92d6-4d5b-b15c-242b555e08aa-registration-dir\") pod \"aws-ebs-csi-driver-node-6jmn4\" (UID: \"afa69cf5-92d6-4d5b-b15c-242b555e08aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jmn4"
Jul 28 17:23:18.122277 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118063    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-host-run-netns\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.122277 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118085    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-node-log\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.122277 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118092    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/849d3b0f-9f02-4f4d-8811-13794309755a-cni-binary-copy\") pod \"multus-additional-cni-plugins-ljfhd\" (UID: \"849d3b0f-9f02-4f4d-8811-13794309755a\") " pod="openshift-multus/multus-additional-cni-plugins-ljfhd"
Jul 28 17:23:18.122277 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118087    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-lib-modules\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.122277 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118143    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/afa69cf5-92d6-4d5b-b15c-242b555e08aa-registration-dir\") pod \"aws-ebs-csi-driver-node-6jmn4\" (UID: \"afa69cf5-92d6-4d5b-b15c-242b555e08aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jmn4"
Jul 28 17:23:18.122277 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118154    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/849d3b0f-9f02-4f4d-8811-13794309755a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ljfhd\" (UID: \"849d3b0f-9f02-4f4d-8811-13794309755a\") " pod="openshift-multus/multus-additional-cni-plugins-ljfhd"
Jul 28 17:23:18.122277 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118180    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-host-slash\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.122277 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118190    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-lib-modules\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.122277 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118231    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-etc-kubernetes\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.122277 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118268    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-host-kubelet\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.122277 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118290    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-etc-kubernetes\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.122277 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118298    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-run-openvswitch\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.122277 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118316    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/849d3b0f-9f02-4f4d-8811-13794309755a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ljfhd\" (UID: \"849d3b0f-9f02-4f4d-8811-13794309755a\") " pod="openshift-multus/multus-additional-cni-plugins-ljfhd"
Jul 28 17:23:18.122277 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118329    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-host-slash\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.122277 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118334    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-log-socket\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.122277 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118372    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-host-kubelet\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.122277 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118387    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.123126 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118393    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-run-openvswitch\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.123126 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118417    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bb8012ba-acac-4f19-823c-2339cc6fc563-ovnkube-config\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.123126 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118421    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-log-socket\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.123126 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118450    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bb8012ba-acac-4f19-823c-2339cc6fc563-ovnkube-script-lib\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.123126 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118453    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.123126 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118497    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/145627c2-9b95-4251-9f16-2920555de2fa-kubelet-config\") pod \"global-pull-secret-syncer-jtb46\" (UID: \"145627c2-9b95-4251-9f16-2920555de2fa\") " pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:18.123126 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118527    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-cnibin\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.123126 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118553    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-host-var-lib-cni-bin\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.123126 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118582    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/145627c2-9b95-4251-9f16-2920555de2fa-kubelet-config\") pod \"global-pull-secret-syncer-jtb46\" (UID: \"145627c2-9b95-4251-9f16-2920555de2fa\") " pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:18.123126 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118598    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-host-cni-bin\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.123126 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118626    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lmtd\" (UniqueName: \"kubernetes.io/projected/bb8012ba-acac-4f19-823c-2339cc6fc563-kube-api-access-7lmtd\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.123126 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118644    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-cnibin\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.123126 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118651    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgsrr\" (UniqueName: \"kubernetes.io/projected/3acecb5c-18d4-4260-aa3e-8758184e6733-kube-api-access-qgsrr\") pod \"node-resolver-kvtkx\" (UID: \"3acecb5c-18d4-4260-aa3e-8758184e6733\") " pod="openshift-dns/node-resolver-kvtkx"
Jul 28 17:23:18.123126 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118680    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/89471f5d-8eda-4a72-ab86-5f2290e75842-konnectivity-ca\") pod \"konnectivity-agent-7qw2x\" (UID: \"89471f5d-8eda-4a72-ab86-5f2290e75842\") " pod="kube-system/konnectivity-agent-7qw2x"
Jul 28 17:23:18.123126 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118694    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-host-var-lib-cni-bin\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.123126 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118705    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-os-release\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.123126 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118731    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-etc-sysconfig\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.123126 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118754    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-host\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.124054 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118775    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/030014cc-fe9d-463d-a819-083f3396c15f-tmp\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.124054 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118802    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb8012ba-acac-4f19-823c-2339cc6fc563-ovn-node-metrics-cert\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.124054 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118825    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ea4785ea-08ca-45f8-a8bc-42524b8d81b4-serviceca\") pod \"node-ca-4kw4j\" (UID: \"ea4785ea-08ca-45f8-a8bc-42524b8d81b4\") " pod="openshift-image-registry/node-ca-4kw4j"
Jul 28 17:23:18.124054 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118846    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-sys\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.124054 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118873    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/849d3b0f-9f02-4f4d-8811-13794309755a-system-cni-dir\") pod \"multus-additional-cni-plugins-ljfhd\" (UID: \"849d3b0f-9f02-4f4d-8811-13794309755a\") " pod="openshift-multus/multus-additional-cni-plugins-ljfhd"
Jul 28 17:23:18.124054 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118899    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-host-cni-netd\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.124054 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118924    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmrmg\" (UniqueName: \"kubernetes.io/projected/ea4785ea-08ca-45f8-a8bc-42524b8d81b4-kube-api-access-mmrmg\") pod \"node-ca-4kw4j\" (UID: \"ea4785ea-08ca-45f8-a8bc-42524b8d81b4\") " pod="openshift-image-registry/node-ca-4kw4j"
Jul 28 17:23:18.124054 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118952    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-host-var-lib-cni-multus\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.124054 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118977    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-multus-conf-dir\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.124054 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.118982    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bb8012ba-acac-4f19-823c-2339cc6fc563-ovnkube-config\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.124054 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.119008    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bb8012ba-acac-4f19-823c-2339cc6fc563-ovnkube-script-lib\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.124054 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.119144    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-multus-conf-dir\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.124054 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.119401    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/849d3b0f-9f02-4f4d-8811-13794309755a-system-cni-dir\") pod \"multus-additional-cni-plugins-ljfhd\" (UID: \"849d3b0f-9f02-4f4d-8811-13794309755a\") " pod="openshift-multus/multus-additional-cni-plugins-ljfhd"
Jul 28 17:23:18.124054 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.119602    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-host-var-lib-cni-multus\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.124054 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.119625    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/89471f5d-8eda-4a72-ab86-5f2290e75842-konnectivity-ca\") pod \"konnectivity-agent-7qw2x\" (UID: \"89471f5d-8eda-4a72-ab86-5f2290e75842\") " pod="kube-system/konnectivity-agent-7qw2x"
Jul 28 17:23:18.124054 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.119653    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-host-cni-bin\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.124054 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.119684    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-sys\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.124054 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.119714    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-etc-sysconfig\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.124977 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.119728    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-systemd-units\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.124977 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.119746    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/be7a3829-e798-4b35-8489-c872935568a8-os-release\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.124977 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.119759    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-var-lib-openvswitch\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.124977 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.119772    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-host-cni-netd\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.124977 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.119789    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-etc-openvswitch\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.124977 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.119793    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-systemd-units\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.124977 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.119794    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-var-lib-openvswitch\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.124977 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.119820    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w89v\" (UniqueName: \"kubernetes.io/projected/2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4-kube-api-access-7w89v\") pod \"network-check-target-9c8w8\" (UID: \"2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4\") " pod="openshift-network-diagnostics/network-check-target-9c8w8"
Jul 28 17:23:18.124977 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.119847    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/145627c2-9b95-4251-9f16-2920555de2fa-original-pull-secret\") pod \"global-pull-secret-syncer-jtb46\" (UID: \"145627c2-9b95-4251-9f16-2920555de2fa\") " pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:18.124977 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.119853    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb8012ba-acac-4f19-823c-2339cc6fc563-etc-openvswitch\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.124977 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.119872    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/849d3b0f-9f02-4f4d-8811-13794309755a-cnibin\") pod \"multus-additional-cni-plugins-ljfhd\" (UID: \"849d3b0f-9f02-4f4d-8811-13794309755a\") " pod="openshift-multus/multus-additional-cni-plugins-ljfhd"
Jul 28 17:23:18.124977 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.119896    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/afa69cf5-92d6-4d5b-b15c-242b555e08aa-sys-fs\") pod \"aws-ebs-csi-driver-node-6jmn4\" (UID: \"afa69cf5-92d6-4d5b-b15c-242b555e08aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jmn4"
Jul 28 17:23:18.124977 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.119940    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/849d3b0f-9f02-4f4d-8811-13794309755a-cnibin\") pod \"multus-additional-cni-plugins-ljfhd\" (UID: \"849d3b0f-9f02-4f4d-8811-13794309755a\") " pod="openshift-multus/multus-additional-cni-plugins-ljfhd"
Jul 28 17:23:18.124977 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.119956    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/afa69cf5-92d6-4d5b-b15c-242b555e08aa-sys-fs\") pod \"aws-ebs-csi-driver-node-6jmn4\" (UID: \"afa69cf5-92d6-4d5b-b15c-242b555e08aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jmn4"
Jul 28 17:23:18.124977 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:18.119952    2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:23:18.124977 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.119976    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/afa69cf5-92d6-4d5b-b15c-242b555e08aa-etc-selinux\") pod \"aws-ebs-csi-driver-node-6jmn4\" (UID: \"afa69cf5-92d6-4d5b-b15c-242b555e08aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jmn4"
Jul 28 17:23:18.124977 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.119999    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/030014cc-fe9d-463d-a819-083f3396c15f-host\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.125541 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:18.120028    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/145627c2-9b95-4251-9f16-2920555de2fa-original-pull-secret podName:145627c2-9b95-4251-9f16-2920555de2fa nodeName:}" failed. No retries permitted until 2026-07-28 17:23:18.620003953 +0000 UTC m=+2.026277688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/145627c2-9b95-4251-9f16-2920555de2fa-original-pull-secret") pod "global-pull-secret-syncer-jtb46" (UID: "145627c2-9b95-4251-9f16-2920555de2fa") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:23:18.125541 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.120138    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/afa69cf5-92d6-4d5b-b15c-242b555e08aa-etc-selinux\") pod \"aws-ebs-csi-driver-node-6jmn4\" (UID: \"afa69cf5-92d6-4d5b-b15c-242b555e08aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jmn4"
Jul 28 17:23:18.125541 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.121263    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/030014cc-fe9d-463d-a819-083f3396c15f-etc-tuned\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.125541 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.121655    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/89471f5d-8eda-4a72-ab86-5f2290e75842-agent-certs\") pod \"konnectivity-agent-7qw2x\" (UID: \"89471f5d-8eda-4a72-ab86-5f2290e75842\") " pod="kube-system/konnectivity-agent-7qw2x"
Jul 28 17:23:18.125541 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.121703    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/030014cc-fe9d-463d-a819-083f3396c15f-tmp\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.125541 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.122451    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb8012ba-acac-4f19-823c-2339cc6fc563-ovn-node-metrics-cert\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.125541 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.122817    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ea4785ea-08ca-45f8-a8bc-42524b8d81b4-serviceca\") pod \"node-ca-4kw4j\" (UID: \"ea4785ea-08ca-45f8-a8bc-42524b8d81b4\") " pod="openshift-image-registry/node-ca-4kw4j"
Jul 28 17:23:18.125541 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:18.123161    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb5c2c1c39b3c131016259171f34c67.slice/crio-3821e26cfb96abacaf70548cca9c76545c6686ce337ee8a9867b69d4f57afb19 WatchSource:0}: Error finding container 3821e26cfb96abacaf70548cca9c76545c6686ce337ee8a9867b69d4f57afb19: Status 404 returned error can't find the container with id 3821e26cfb96abacaf70548cca9c76545c6686ce337ee8a9867b69d4f57afb19
Jul 28 17:23:18.129937 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.129924    2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:23:18.137850 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.137830    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltckq\" (UniqueName: \"kubernetes.io/projected/d50b403c-3842-4b2f-a314-e65583a0b225-kube-api-access-ltckq\") pod \"iptables-alerter-7pp8b\" (UID: \"d50b403c-3842-4b2f-a314-e65583a0b225\") " pod="openshift-network-operator/iptables-alerter-7pp8b"
Jul 28 17:23:18.143703 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.143684    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvch5\" (UniqueName: \"kubernetes.io/projected/afa69cf5-92d6-4d5b-b15c-242b555e08aa-kube-api-access-wvch5\") pod \"aws-ebs-csi-driver-node-6jmn4\" (UID: \"afa69cf5-92d6-4d5b-b15c-242b555e08aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jmn4"
Jul 28 17:23:18.143843 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:18.143827    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:23:18.143888 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:18.143847    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:23:18.143888 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:18.143858    2543 projected.go:196] Error preparing data for projected volume kube-api-access-7w89v for pod openshift-network-diagnostics/network-check-target-9c8w8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:23:18.143965 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:18.143913    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4-kube-api-access-7w89v podName:2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:18.643896261 +0000 UTC m=+2.050170001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7w89v" (UniqueName: "kubernetes.io/projected/2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4-kube-api-access-7w89v") pod "network-check-target-9c8w8" (UID: "2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:23:18.143965 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.143949    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgsrr\" (UniqueName: \"kubernetes.io/projected/3acecb5c-18d4-4260-aa3e-8758184e6733-kube-api-access-qgsrr\") pod \"node-resolver-kvtkx\" (UID: \"3acecb5c-18d4-4260-aa3e-8758184e6733\") " pod="openshift-dns/node-resolver-kvtkx"
Jul 28 17:23:18.146101 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.146077    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj9lr\" (UniqueName: \"kubernetes.io/projected/030014cc-fe9d-463d-a819-083f3396c15f-kube-api-access-jj9lr\") pod \"tuned-9chjn\" (UID: \"030014cc-fe9d-463d-a819-083f3396c15f\") " pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.146685 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.146660    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lmtd\" (UniqueName: \"kubernetes.io/projected/bb8012ba-acac-4f19-823c-2339cc6fc563-kube-api-access-7lmtd\") pod \"ovnkube-node-64lk7\" (UID: \"bb8012ba-acac-4f19-823c-2339cc6fc563\") " pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.146793 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.146778    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmrmg\" (UniqueName: \"kubernetes.io/projected/ea4785ea-08ca-45f8-a8bc-42524b8d81b4-kube-api-access-mmrmg\") pod \"node-ca-4kw4j\" (UID: \"ea4785ea-08ca-45f8-a8bc-42524b8d81b4\") " pod="openshift-image-registry/node-ca-4kw4j"
Jul 28 17:23:18.147756 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.147730    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm2v8\" (UniqueName: \"kubernetes.io/projected/a2c1c169-0fd2-4e73-a497-0b69fe85d11f-kube-api-access-fm2v8\") pod \"network-metrics-daemon-sjtrc\" (UID: \"a2c1c169-0fd2-4e73-a497-0b69fe85d11f\") " pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:23:18.147862 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.147834    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k56l4\" (UniqueName: \"kubernetes.io/projected/be7a3829-e798-4b35-8489-c872935568a8-kube-api-access-k56l4\") pod \"multus-st8s2\" (UID: \"be7a3829-e798-4b35-8489-c872935568a8\") " pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.149146 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.149126    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbp6g\" (UniqueName: \"kubernetes.io/projected/849d3b0f-9f02-4f4d-8811-13794309755a-kube-api-access-rbp6g\") pod \"multus-additional-cni-plugins-ljfhd\" (UID: \"849d3b0f-9f02-4f4d-8811-13794309755a\") " pod="openshift-multus/multus-additional-cni-plugins-ljfhd"
Jul 28 17:23:18.149878 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.149843    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-177.ec2.internal" event={"ID":"1bb5c2c1c39b3c131016259171f34c67","Type":"ContainerStarted","Data":"3821e26cfb96abacaf70548cca9c76545c6686ce337ee8a9867b69d4f57afb19"}
Jul 28 17:23:18.228354 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.228265    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9chjn"
Jul 28 17:23:18.237046 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:18.237024    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod030014cc_fe9d_463d_a819_083f3396c15f.slice/crio-a73a6072cc9dd9f93b0fc932b51bad09fc849c5524fac9cf17242a7af8ae050f WatchSource:0}: Error finding container a73a6072cc9dd9f93b0fc932b51bad09fc849c5524fac9cf17242a7af8ae050f: Status 404 returned error can't find the container with id a73a6072cc9dd9f93b0fc932b51bad09fc849c5524fac9cf17242a7af8ae050f
Jul 28 17:23:18.245005 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.244977    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kvtkx"
Jul 28 17:23:18.253017 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:18.252992    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3acecb5c_18d4_4260_aa3e_8758184e6733.slice/crio-c7f0ded13ba9ac7a05ab5ea9c63db6c0cea9c66958d8122a02f5f4dd148b278f WatchSource:0}: Error finding container c7f0ded13ba9ac7a05ab5ea9c63db6c0cea9c66958d8122a02f5f4dd148b278f: Status 404 returned error can't find the container with id c7f0ded13ba9ac7a05ab5ea9c63db6c0cea9c66958d8122a02f5f4dd148b278f
Jul 28 17:23:18.262415 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.262394    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4kw4j"
Jul 28 17:23:18.271107 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:18.271078    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea4785ea_08ca_45f8_a8bc_42524b8d81b4.slice/crio-366e89df5eedb1815b312262487e09db68ab49a3abfd472130d88c0195c30291 WatchSource:0}: Error finding container 366e89df5eedb1815b312262487e09db68ab49a3abfd472130d88c0195c30291: Status 404 returned error can't find the container with id 366e89df5eedb1815b312262487e09db68ab49a3abfd472130d88c0195c30291
Jul 28 17:23:18.272906 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.272878    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7pp8b"
Jul 28 17:23:18.280344 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:18.280319    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50b403c_3842_4b2f_a314_e65583a0b225.slice/crio-c1c29b4e8de00fe0df1442d1b83022ca98955eba36bc2e12e7244c308390bf5d WatchSource:0}: Error finding container c1c29b4e8de00fe0df1442d1b83022ca98955eba36bc2e12e7244c308390bf5d: Status 404 returned error can't find the container with id c1c29b4e8de00fe0df1442d1b83022ca98955eba36bc2e12e7244c308390bf5d
Jul 28 17:23:18.290161 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.290129    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ljfhd"
Jul 28 17:23:18.297794 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:18.297772    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod849d3b0f_9f02_4f4d_8811_13794309755a.slice/crio-7008a962ed18bcbba6f38c2b400ed9fd402ba1964dc01c8766d361795b75eff5 WatchSource:0}: Error finding container 7008a962ed18bcbba6f38c2b400ed9fd402ba1964dc01c8766d361795b75eff5: Status 404 returned error can't find the container with id 7008a962ed18bcbba6f38c2b400ed9fd402ba1964dc01c8766d361795b75eff5
Jul 28 17:23:18.305629 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.305608    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-st8s2"
Jul 28 17:23:18.314205 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:18.314179    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe7a3829_e798_4b35_8489_c872935568a8.slice/crio-efe72f50c3b8ce1459275e73b68dcc741b9d5d4fb361224dbb1a8a73fb633106 WatchSource:0}: Error finding container efe72f50c3b8ce1459275e73b68dcc741b9d5d4fb361224dbb1a8a73fb633106: Status 404 returned error can't find the container with id efe72f50c3b8ce1459275e73b68dcc741b9d5d4fb361224dbb1a8a73fb633106
Jul 28 17:23:18.315929 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.315885    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:18.323319 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:18.323294    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb8012ba_acac_4f19_823c_2339cc6fc563.slice/crio-ef298e447e71d111b6899a17a1f20cc604164ee5b9e13b707d3cb38be4f5fcc4 WatchSource:0}: Error finding container ef298e447e71d111b6899a17a1f20cc604164ee5b9e13b707d3cb38be4f5fcc4: Status 404 returned error can't find the container with id ef298e447e71d111b6899a17a1f20cc604164ee5b9e13b707d3cb38be4f5fcc4
Jul 28 17:23:18.327390 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.327373    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7qw2x"
Jul 28 17:23:18.330784 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.330765    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jmn4"
Jul 28 17:23:18.335392 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:18.335371    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89471f5d_8eda_4a72_ab86_5f2290e75842.slice/crio-65a87caf623c0c8799080b85b1a2b9ac9699dbfb41ffbbdb10d3d5d9843126f9 WatchSource:0}: Error finding container 65a87caf623c0c8799080b85b1a2b9ac9699dbfb41ffbbdb10d3d5d9843126f9: Status 404 returned error can't find the container with id 65a87caf623c0c8799080b85b1a2b9ac9699dbfb41ffbbdb10d3d5d9843126f9
Jul 28 17:23:18.339594 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:18.339570    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafa69cf5_92d6_4d5b_b15c_242b555e08aa.slice/crio-0cc9fceea0893f89aa98d99c6b209962ec7d5c2b85fef1a7653bbbabd02d9d8a WatchSource:0}: Error finding container 0cc9fceea0893f89aa98d99c6b209962ec7d5c2b85fef1a7653bbbabd02d9d8a: Status 404 returned error can't find the container with id 0cc9fceea0893f89aa98d99c6b209962ec7d5c2b85fef1a7653bbbabd02d9d8a
Jul 28 17:23:18.423464 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:18.423433    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b77ac91bc5dbb86ffeab81485172d7.slice/crio-ce54fbbcc55ed54bd83633cc7071d79c5774646b5e3b47dcef3c523cc70c56ef WatchSource:0}: Error finding container ce54fbbcc55ed54bd83633cc7071d79c5774646b5e3b47dcef3c523cc70c56ef: Status 404 returned error can't find the container with id ce54fbbcc55ed54bd83633cc7071d79c5774646b5e3b47dcef3c523cc70c56ef
Jul 28 17:23:18.624520 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.624440    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c1c169-0fd2-4e73-a497-0b69fe85d11f-metrics-certs\") pod \"network-metrics-daemon-sjtrc\" (UID: \"a2c1c169-0fd2-4e73-a497-0b69fe85d11f\") " pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:23:18.624520 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.624509    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/145627c2-9b95-4251-9f16-2920555de2fa-original-pull-secret\") pod \"global-pull-secret-syncer-jtb46\" (UID: \"145627c2-9b95-4251-9f16-2920555de2fa\") " pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:18.624763 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:18.624598    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:23:18.624763 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:18.624656    2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:23:18.624763 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:18.624672    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2c1c169-0fd2-4e73-a497-0b69fe85d11f-metrics-certs podName:a2c1c169-0fd2-4e73-a497-0b69fe85d11f nodeName:}" failed. No retries permitted until 2026-07-28 17:23:19.62465431 +0000 UTC m=+3.030928032 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2c1c169-0fd2-4e73-a497-0b69fe85d11f-metrics-certs") pod "network-metrics-daemon-sjtrc" (UID: "a2c1c169-0fd2-4e73-a497-0b69fe85d11f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:23:18.624763 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:18.624711    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/145627c2-9b95-4251-9f16-2920555de2fa-original-pull-secret podName:145627c2-9b95-4251-9f16-2920555de2fa nodeName:}" failed. No retries permitted until 2026-07-28 17:23:19.62469455 +0000 UTC m=+3.030968286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/145627c2-9b95-4251-9f16-2920555de2fa-original-pull-secret") pod "global-pull-secret-syncer-jtb46" (UID: "145627c2-9b95-4251-9f16-2920555de2fa") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:23:18.725739 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.725076    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w89v\" (UniqueName: \"kubernetes.io/projected/2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4-kube-api-access-7w89v\") pod \"network-check-target-9c8w8\" (UID: \"2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4\") " pod="openshift-network-diagnostics/network-check-target-9c8w8"
Jul 28 17:23:18.725739 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:18.725277    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:23:18.725739 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:18.725298    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:23:18.725739 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:18.725312    2543 projected.go:196] Error preparing data for projected volume kube-api-access-7w89v for pod openshift-network-diagnostics/network-check-target-9c8w8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:23:18.725739 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:18.725368    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4-kube-api-access-7w89v podName:2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:19.725349138 +0000 UTC m=+3.131622866 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7w89v" (UniqueName: "kubernetes.io/projected/2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4-kube-api-access-7w89v") pod "network-check-target-9c8w8" (UID: "2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:23:18.779209 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.779174    2543 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:23:18.988051 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.987930    2543 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:18:18 +0000 UTC" deadline="2028-01-05 05:18:22.440029153 +0000 UTC"
Jul 28 17:23:18.988051 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:18.988000    2543 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12611h55m3.452034171s"
Jul 28 17:23:19.154258 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:19.154229    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:19.154432 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:19.154360    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtb46" podUID="145627c2-9b95-4251-9f16-2920555de2fa"
Jul 28 17:23:19.154829 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:19.154796    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:23:19.154956 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:19.154933    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjtrc" podUID="a2c1c169-0fd2-4e73-a497-0b69fe85d11f"
Jul 28 17:23:19.160745 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:19.160714    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-st8s2" event={"ID":"be7a3829-e798-4b35-8489-c872935568a8","Type":"ContainerStarted","Data":"efe72f50c3b8ce1459275e73b68dcc741b9d5d4fb361224dbb1a8a73fb633106"}
Jul 28 17:23:19.170401 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:19.170334    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7pp8b" event={"ID":"d50b403c-3842-4b2f-a314-e65583a0b225","Type":"ContainerStarted","Data":"c1c29b4e8de00fe0df1442d1b83022ca98955eba36bc2e12e7244c308390bf5d"}
Jul 28 17:23:19.191042 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:19.191010    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kvtkx" event={"ID":"3acecb5c-18d4-4260-aa3e-8758184e6733","Type":"ContainerStarted","Data":"c7f0ded13ba9ac7a05ab5ea9c63db6c0cea9c66958d8122a02f5f4dd148b278f"}
Jul 28 17:23:19.203866 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:19.203808    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9chjn" event={"ID":"030014cc-fe9d-463d-a819-083f3396c15f","Type":"ContainerStarted","Data":"a73a6072cc9dd9f93b0fc932b51bad09fc849c5524fac9cf17242a7af8ae050f"}
Jul 28 17:23:19.215446 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:19.215416    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-177.ec2.internal" event={"ID":"a0b77ac91bc5dbb86ffeab81485172d7","Type":"ContainerStarted","Data":"ce54fbbcc55ed54bd83633cc7071d79c5774646b5e3b47dcef3c523cc70c56ef"}
Jul 28 17:23:19.221244 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:19.221195    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7qw2x" event={"ID":"89471f5d-8eda-4a72-ab86-5f2290e75842","Type":"ContainerStarted","Data":"65a87caf623c0c8799080b85b1a2b9ac9699dbfb41ffbbdb10d3d5d9843126f9"}
Jul 28 17:23:19.227234 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:19.227195    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ljfhd" event={"ID":"849d3b0f-9f02-4f4d-8811-13794309755a","Type":"ContainerStarted","Data":"7008a962ed18bcbba6f38c2b400ed9fd402ba1964dc01c8766d361795b75eff5"}
Jul 28 17:23:19.235815 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:19.235786    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4kw4j" event={"ID":"ea4785ea-08ca-45f8-a8bc-42524b8d81b4","Type":"ContainerStarted","Data":"366e89df5eedb1815b312262487e09db68ab49a3abfd472130d88c0195c30291"}
Jul 28 17:23:19.245497 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:19.245154    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jmn4" event={"ID":"afa69cf5-92d6-4d5b-b15c-242b555e08aa","Type":"ContainerStarted","Data":"0cc9fceea0893f89aa98d99c6b209962ec7d5c2b85fef1a7653bbbabd02d9d8a"}
Jul 28 17:23:19.253460 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:19.253432    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-64lk7" event={"ID":"bb8012ba-acac-4f19-823c-2339cc6fc563","Type":"ContainerStarted","Data":"ef298e447e71d111b6899a17a1f20cc604164ee5b9e13b707d3cb38be4f5fcc4"}
Jul 28 17:23:19.635717 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:19.633549    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c1c169-0fd2-4e73-a497-0b69fe85d11f-metrics-certs\") pod \"network-metrics-daemon-sjtrc\" (UID: \"a2c1c169-0fd2-4e73-a497-0b69fe85d11f\") " pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:23:19.635717 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:19.633649    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/145627c2-9b95-4251-9f16-2920555de2fa-original-pull-secret\") pod \"global-pull-secret-syncer-jtb46\" (UID: \"145627c2-9b95-4251-9f16-2920555de2fa\") " pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:19.635717 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:19.633805    2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:23:19.635717 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:19.633871    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/145627c2-9b95-4251-9f16-2920555de2fa-original-pull-secret podName:145627c2-9b95-4251-9f16-2920555de2fa nodeName:}" failed. No retries permitted until 2026-07-28 17:23:21.633851942 +0000 UTC m=+5.040125664 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/145627c2-9b95-4251-9f16-2920555de2fa-original-pull-secret") pod "global-pull-secret-syncer-jtb46" (UID: "145627c2-9b95-4251-9f16-2920555de2fa") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:23:19.635717 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:19.634320    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:23:19.635717 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:19.634372    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2c1c169-0fd2-4e73-a497-0b69fe85d11f-metrics-certs podName:a2c1c169-0fd2-4e73-a497-0b69fe85d11f nodeName:}" failed. No retries permitted until 2026-07-28 17:23:21.634355538 +0000 UTC m=+5.040629262 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2c1c169-0fd2-4e73-a497-0b69fe85d11f-metrics-certs") pod "network-metrics-daemon-sjtrc" (UID: "a2c1c169-0fd2-4e73-a497-0b69fe85d11f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:23:19.734645 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:19.734606    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w89v\" (UniqueName: \"kubernetes.io/projected/2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4-kube-api-access-7w89v\") pod \"network-check-target-9c8w8\" (UID: \"2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4\") " pod="openshift-network-diagnostics/network-check-target-9c8w8"
Jul 28 17:23:19.734851 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:19.734795    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:23:19.734851 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:19.734820    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:23:19.734851 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:19.734833    2543 projected.go:196] Error preparing data for projected volume kube-api-access-7w89v for pod openshift-network-diagnostics/network-check-target-9c8w8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:23:19.735038 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:19.734904    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4-kube-api-access-7w89v podName:2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:21.734884467 +0000 UTC m=+5.141158194 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7w89v" (UniqueName: "kubernetes.io/projected/2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4-kube-api-access-7w89v") pod "network-check-target-9c8w8" (UID: "2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:23:19.989099 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:19.989001    2543 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:18:18 +0000 UTC" deadline="2028-02-23 15:41:11.494807803 +0000 UTC"
Jul 28 17:23:19.989099 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:19.989048    2543 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13798h17m51.505765104s"
Jul 28 17:23:20.073863 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:20.073828    2543 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:23:20.147610 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:20.147069    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9c8w8"
Jul 28 17:23:20.147610 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:20.147218    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9c8w8" podUID="2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4"
Jul 28 17:23:20.815322 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:20.815274    2543 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:23:21.147330 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:21.147249    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:21.147767 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:21.147393    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtb46" podUID="145627c2-9b95-4251-9f16-2920555de2fa"
Jul 28 17:23:21.147767 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:21.147443    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:23:21.147767 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:21.147565    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjtrc" podUID="a2c1c169-0fd2-4e73-a497-0b69fe85d11f"
Jul 28 17:23:21.654093 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:21.654060    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c1c169-0fd2-4e73-a497-0b69fe85d11f-metrics-certs\") pod \"network-metrics-daemon-sjtrc\" (UID: \"a2c1c169-0fd2-4e73-a497-0b69fe85d11f\") " pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:23:21.654293 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:21.654169    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/145627c2-9b95-4251-9f16-2920555de2fa-original-pull-secret\") pod \"global-pull-secret-syncer-jtb46\" (UID: \"145627c2-9b95-4251-9f16-2920555de2fa\") " pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:21.654293 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:21.654246    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:23:21.654293 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:21.654284    2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:23:21.654461 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:21.654322    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2c1c169-0fd2-4e73-a497-0b69fe85d11f-metrics-certs podName:a2c1c169-0fd2-4e73-a497-0b69fe85d11f nodeName:}" failed. No retries permitted until 2026-07-28 17:23:25.654299179 +0000 UTC m=+9.060573013 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2c1c169-0fd2-4e73-a497-0b69fe85d11f-metrics-certs") pod "network-metrics-daemon-sjtrc" (UID: "a2c1c169-0fd2-4e73-a497-0b69fe85d11f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:23:21.654461 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:21.654345    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/145627c2-9b95-4251-9f16-2920555de2fa-original-pull-secret podName:145627c2-9b95-4251-9f16-2920555de2fa nodeName:}" failed. No retries permitted until 2026-07-28 17:23:25.654334683 +0000 UTC m=+9.060608407 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/145627c2-9b95-4251-9f16-2920555de2fa-original-pull-secret") pod "global-pull-secret-syncer-jtb46" (UID: "145627c2-9b95-4251-9f16-2920555de2fa") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:23:21.755300 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:21.754993    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w89v\" (UniqueName: \"kubernetes.io/projected/2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4-kube-api-access-7w89v\") pod \"network-check-target-9c8w8\" (UID: \"2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4\") " pod="openshift-network-diagnostics/network-check-target-9c8w8"
Jul 28 17:23:21.756327 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:21.755633    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:23:21.756327 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:21.755703    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:23:21.756327 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:21.755723    2543 projected.go:196] Error preparing data for projected volume kube-api-access-7w89v for pod openshift-network-diagnostics/network-check-target-9c8w8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:23:21.756327 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:21.755793    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4-kube-api-access-7w89v podName:2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:25.75577401 +0000 UTC m=+9.162047735 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7w89v" (UniqueName: "kubernetes.io/projected/2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4-kube-api-access-7w89v") pod "network-check-target-9c8w8" (UID: "2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:23:22.146817 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:22.146695    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9c8w8"
Jul 28 17:23:22.146997 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:22.146922    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9c8w8" podUID="2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4"
Jul 28 17:23:23.148151 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:23.147363    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:23.148151 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:23.147502    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtb46" podUID="145627c2-9b95-4251-9f16-2920555de2fa"
Jul 28 17:23:23.148151 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:23.147964    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:23:23.148151 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:23.148078    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjtrc" podUID="a2c1c169-0fd2-4e73-a497-0b69fe85d11f"
Jul 28 17:23:24.147297 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:24.147260    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9c8w8"
Jul 28 17:23:24.147482 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:24.147398    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9c8w8" podUID="2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4"
Jul 28 17:23:25.155467 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:25.155435    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:25.155939 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:25.155704    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:23:25.155939 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:25.155828    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjtrc" podUID="a2c1c169-0fd2-4e73-a497-0b69fe85d11f"
Jul 28 17:23:25.155939 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:25.155624    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtb46" podUID="145627c2-9b95-4251-9f16-2920555de2fa"
Jul 28 17:23:25.692114 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:25.692010    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/145627c2-9b95-4251-9f16-2920555de2fa-original-pull-secret\") pod \"global-pull-secret-syncer-jtb46\" (UID: \"145627c2-9b95-4251-9f16-2920555de2fa\") " pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:25.692114 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:25.692093    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c1c169-0fd2-4e73-a497-0b69fe85d11f-metrics-certs\") pod \"network-metrics-daemon-sjtrc\" (UID: \"a2c1c169-0fd2-4e73-a497-0b69fe85d11f\") " pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:23:25.692370 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:25.692209    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:23:25.692370 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:25.692276    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2c1c169-0fd2-4e73-a497-0b69fe85d11f-metrics-certs podName:a2c1c169-0fd2-4e73-a497-0b69fe85d11f nodeName:}" failed. No retries permitted until 2026-07-28 17:23:33.692255286 +0000 UTC m=+17.098529006 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2c1c169-0fd2-4e73-a497-0b69fe85d11f-metrics-certs") pod "network-metrics-daemon-sjtrc" (UID: "a2c1c169-0fd2-4e73-a497-0b69fe85d11f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:23:25.692370 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:25.692209    2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:23:25.692370 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:25.692330    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/145627c2-9b95-4251-9f16-2920555de2fa-original-pull-secret podName:145627c2-9b95-4251-9f16-2920555de2fa nodeName:}" failed. No retries permitted until 2026-07-28 17:23:33.692318022 +0000 UTC m=+17.098591745 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/145627c2-9b95-4251-9f16-2920555de2fa-original-pull-secret") pod "global-pull-secret-syncer-jtb46" (UID: "145627c2-9b95-4251-9f16-2920555de2fa") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:23:25.793566 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:25.793405    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w89v\" (UniqueName: \"kubernetes.io/projected/2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4-kube-api-access-7w89v\") pod \"network-check-target-9c8w8\" (UID: \"2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4\") " pod="openshift-network-diagnostics/network-check-target-9c8w8"
Jul 28 17:23:25.793732 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:25.793598    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:23:25.793732 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:25.793620    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:23:25.793732 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:25.793632    2543 projected.go:196] Error preparing data for projected volume kube-api-access-7w89v for pod openshift-network-diagnostics/network-check-target-9c8w8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:23:25.793732 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:25.793692    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4-kube-api-access-7w89v podName:2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:33.79367434 +0000 UTC m=+17.199948059 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7w89v" (UniqueName: "kubernetes.io/projected/2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4-kube-api-access-7w89v") pod "network-check-target-9c8w8" (UID: "2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:23:26.146492 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:26.146417    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9c8w8"
Jul 28 17:23:26.146655 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:26.146548    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9c8w8" podUID="2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4"
Jul 28 17:23:27.147403 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:27.147368    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:27.147879 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:27.147484    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtb46" podUID="145627c2-9b95-4251-9f16-2920555de2fa"
Jul 28 17:23:27.147879 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:27.147512    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:23:27.147879 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:27.147640    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjtrc" podUID="a2c1c169-0fd2-4e73-a497-0b69fe85d11f"
Jul 28 17:23:28.147316 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:28.147280    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9c8w8"
Jul 28 17:23:28.147495 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:28.147397    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9c8w8" podUID="2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4"
Jul 28 17:23:29.146942 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:29.146895    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:23:29.146942 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:29.146930    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:29.147210 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:29.147031    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjtrc" podUID="a2c1c169-0fd2-4e73-a497-0b69fe85d11f"
Jul 28 17:23:29.147210 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:29.147155    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtb46" podUID="145627c2-9b95-4251-9f16-2920555de2fa"
Jul 28 17:23:30.147280 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:30.147249    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9c8w8"
Jul 28 17:23:30.147648 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:30.147343    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9c8w8" podUID="2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4"
Jul 28 17:23:31.146778 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:31.146737    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:31.146945 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:31.146874    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtb46" podUID="145627c2-9b95-4251-9f16-2920555de2fa"
Jul 28 17:23:31.147028 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:31.146748    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:23:31.147177 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:31.147157    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjtrc" podUID="a2c1c169-0fd2-4e73-a497-0b69fe85d11f"
Jul 28 17:23:32.146507 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:32.146292    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9c8w8"
Jul 28 17:23:32.146972 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:32.146610    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9c8w8" podUID="2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4"
Jul 28 17:23:33.147152 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:33.147099    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:23:33.147600 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:33.147268    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjtrc" podUID="a2c1c169-0fd2-4e73-a497-0b69fe85d11f"
Jul 28 17:23:33.147600 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:33.147318    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:33.147600 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:33.147439    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtb46" podUID="145627c2-9b95-4251-9f16-2920555de2fa"
Jul 28 17:23:33.758912 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:33.758877    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/145627c2-9b95-4251-9f16-2920555de2fa-original-pull-secret\") pod \"global-pull-secret-syncer-jtb46\" (UID: \"145627c2-9b95-4251-9f16-2920555de2fa\") " pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:33.759101 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:33.758941    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c1c169-0fd2-4e73-a497-0b69fe85d11f-metrics-certs\") pod \"network-metrics-daemon-sjtrc\" (UID: \"a2c1c169-0fd2-4e73-a497-0b69fe85d11f\") " pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:23:33.759101 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:33.759049    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:23:33.759101 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:33.759050    2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:23:33.759282 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:33.759113    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2c1c169-0fd2-4e73-a497-0b69fe85d11f-metrics-certs podName:a2c1c169-0fd2-4e73-a497-0b69fe85d11f nodeName:}" failed. No retries permitted until 2026-07-28 17:23:49.759094102 +0000 UTC m=+33.165367828 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2c1c169-0fd2-4e73-a497-0b69fe85d11f-metrics-certs") pod "network-metrics-daemon-sjtrc" (UID: "a2c1c169-0fd2-4e73-a497-0b69fe85d11f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:23:33.759282 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:33.759187    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/145627c2-9b95-4251-9f16-2920555de2fa-original-pull-secret podName:145627c2-9b95-4251-9f16-2920555de2fa nodeName:}" failed. No retries permitted until 2026-07-28 17:23:49.759163743 +0000 UTC m=+33.165437477 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/145627c2-9b95-4251-9f16-2920555de2fa-original-pull-secret") pod "global-pull-secret-syncer-jtb46" (UID: "145627c2-9b95-4251-9f16-2920555de2fa") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:23:33.860295 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:33.860256    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w89v\" (UniqueName: \"kubernetes.io/projected/2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4-kube-api-access-7w89v\") pod \"network-check-target-9c8w8\" (UID: \"2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4\") " pod="openshift-network-diagnostics/network-check-target-9c8w8"
Jul 28 17:23:33.860474 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:33.860450    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:23:33.860522 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:33.860481    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:23:33.860522 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:33.860496    2543 projected.go:196] Error preparing data for projected volume kube-api-access-7w89v for pod openshift-network-diagnostics/network-check-target-9c8w8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:23:33.860593 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:33.860561    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4-kube-api-access-7w89v podName:2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:49.860540975 +0000 UTC m=+33.266814698 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7w89v" (UniqueName: "kubernetes.io/projected/2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4-kube-api-access-7w89v") pod "network-check-target-9c8w8" (UID: "2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:23:34.146628 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:34.146604    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9c8w8"
Jul 28 17:23:34.146735 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:34.146700    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9c8w8" podUID="2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4"
Jul 28 17:23:35.146563 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:35.146529    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:23:35.146563 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:35.146563    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:35.147125 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:35.146651    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjtrc" podUID="a2c1c169-0fd2-4e73-a497-0b69fe85d11f"
Jul 28 17:23:35.147125 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:35.146748    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtb46" podUID="145627c2-9b95-4251-9f16-2920555de2fa"
Jul 28 17:23:36.147296 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:36.147254    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9c8w8"
Jul 28 17:23:36.147780 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:36.147383    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9c8w8" podUID="2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4"
Jul 28 17:23:37.147811 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:37.147781    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:37.148270 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:37.147879    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtb46" podUID="145627c2-9b95-4251-9f16-2920555de2fa"
Jul 28 17:23:37.148270 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:37.147947    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:23:37.148270 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:37.148045    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjtrc" podUID="a2c1c169-0fd2-4e73-a497-0b69fe85d11f"
Jul 28 17:23:38.147080 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:38.147051    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9c8w8"
Jul 28 17:23:38.147274 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:38.147182    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9c8w8" podUID="2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4"
Jul 28 17:23:39.146861 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:39.146394    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:39.146861 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:39.146720    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtb46" podUID="145627c2-9b95-4251-9f16-2920555de2fa"
Jul 28 17:23:39.147552 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:39.147281    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:23:39.147552 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:39.147417    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjtrc" podUID="a2c1c169-0fd2-4e73-a497-0b69fe85d11f"
Jul 28 17:23:39.314203 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:39.314170    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kvtkx" event={"ID":"3acecb5c-18d4-4260-aa3e-8758184e6733","Type":"ContainerStarted","Data":"11c078a65acde3c41567492dc606c7d7f6cdb412ea7c45f106058df701b7c18d"}
Jul 28 17:23:39.315621 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:39.315598    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9chjn" event={"ID":"030014cc-fe9d-463d-a819-083f3396c15f","Type":"ContainerStarted","Data":"4d29c61cef8d6e66562491a1a0d770e4630e4ccc2a0523eb8af66110fd3bfee6"}
Jul 28 17:23:39.316985 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:39.316966    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-177.ec2.internal" event={"ID":"a0b77ac91bc5dbb86ffeab81485172d7","Type":"ContainerStarted","Data":"da8c297774145c72394ec2affa6048edc04ce8d73dfcc46c0f836d230aad283a"}
Jul 28 17:23:39.318310 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:39.318290    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7qw2x" event={"ID":"89471f5d-8eda-4a72-ab86-5f2290e75842","Type":"ContainerStarted","Data":"8ca1e4bf2da11c9f5a7b5a7f184a41393106f9f66dc437e77f08c7154eec619b"}
Jul 28 17:23:39.319754 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:39.319727    2543 generic.go:356] "Generic (PLEG): container finished" podID="849d3b0f-9f02-4f4d-8811-13794309755a" containerID="38a0be162605b02559895abe39e5d26bb200745267a78f9d245cde76ffb105a4" exitCode=0
Jul 28 17:23:39.319835 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:39.319811    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ljfhd" event={"ID":"849d3b0f-9f02-4f4d-8811-13794309755a","Type":"ContainerDied","Data":"38a0be162605b02559895abe39e5d26bb200745267a78f9d245cde76ffb105a4"}
Jul 28 17:23:39.321316 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:39.321252    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4kw4j" event={"ID":"ea4785ea-08ca-45f8-a8bc-42524b8d81b4","Type":"ContainerStarted","Data":"557b5354bff781461cff4410ea3fbaa00f4604a3c372856f1e77ef42685a7bfe"}
Jul 28 17:23:39.322729 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:39.322704    2543 generic.go:356] "Generic (PLEG): container finished" podID="1bb5c2c1c39b3c131016259171f34c67" containerID="8df5308d25005359c03f049c34ec32c1c6cddd91743e2c5926366afc45613369" exitCode=0
Jul 28 17:23:39.322806 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:39.322794    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-177.ec2.internal" event={"ID":"1bb5c2c1c39b3c131016259171f34c67","Type":"ContainerDied","Data":"8df5308d25005359c03f049c34ec32c1c6cddd91743e2c5926366afc45613369"}
Jul 28 17:23:39.324244 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:39.324222    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jmn4" event={"ID":"afa69cf5-92d6-4d5b-b15c-242b555e08aa","Type":"ContainerStarted","Data":"c2b373f0b236f90ca86b7da53e8e2ac07e4a9cf8e489f438c7fa505942dc0f55"}
Jul 28 17:23:39.326729 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:39.326709    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-64lk7" event={"ID":"bb8012ba-acac-4f19-823c-2339cc6fc563","Type":"ContainerStarted","Data":"55c4c6e0785768b9e8927334becc64b628eda8fc0d28a74712457a8694760764"}
Jul 28 17:23:39.326828 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:39.326782    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-64lk7" event={"ID":"bb8012ba-acac-4f19-823c-2339cc6fc563","Type":"ContainerStarted","Data":"d24336752c438a533be4d25ad83ef3653a768be7dbd011990c40540b1c1e9820"}
Jul 28 17:23:39.326828 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:39.326818    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-64lk7" event={"ID":"bb8012ba-acac-4f19-823c-2339cc6fc563","Type":"ContainerStarted","Data":"4159c3b409352287e159d55ceee89784fd135eef39841e55568383b6ab46e588"}
Jul 28 17:23:39.327910 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:39.327888    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-st8s2" event={"ID":"be7a3829-e798-4b35-8489-c872935568a8","Type":"ContainerStarted","Data":"af6a86807a06906c17a3670d83a3a999ebb62dfea48b6a6c7b0dafff8f6e41df"}
Jul 28 17:23:39.331889 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:39.331847    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-kvtkx" podStartSLOduration=2.058041462 podStartE2EDuration="22.331836496s" totalImagesPullingTime="20.273795034s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:39.331667142 +0000 UTC m=+22.737940883" watchObservedRunningTime="2026-07-28 17:23:39.331836496 +0000 UTC m=+22.738110249"
Jul 28 17:23:39.346677 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:39.346640    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-177.ec2.internal" podStartSLOduration=21.346621245 podStartE2EDuration="21.346621245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:39.34651993 +0000 UTC m=+22.752793671" watchObservedRunningTime="2026-07-28 17:23:39.346621245 +0000 UTC m=+22.752894986"
Jul 28 17:23:39.385118 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:39.385078    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-st8s2" podStartSLOduration=2.168129154 podStartE2EDuration="22.38506374s" totalImagesPullingTime="20.216934586s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:39.384515334 +0000 UTC m=+22.790789075" watchObservedRunningTime="2026-07-28 17:23:39.38506374 +0000 UTC m=+22.791337485"
Jul 28 17:23:39.385342 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:39.385310    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9chjn" podStartSLOduration=2.096892335 podStartE2EDuration="22.385299155s" totalImagesPullingTime="20.28840682s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:39.367275625 +0000 UTC m=+22.773549362" watchObservedRunningTime="2026-07-28 17:23:39.385299155 +0000 UTC m=+22.791572899"
Jul 28 17:23:39.416019 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:39.415698    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4kw4j" podStartSLOduration=2.19298986 podStartE2EDuration="22.415678555s" totalImagesPullingTime="20.222688695s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:39.415623698 +0000 UTC m=+22.821897439" watchObservedRunningTime="2026-07-28 17:23:39.415678555 +0000 UTC m=+22.821952298"
Jul 28 17:23:39.460005 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:39.459955    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7qw2x" podStartSLOduration=6.661797022 podStartE2EDuration="22.459939143s" totalImagesPullingTime="15.798142121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:39.459553403 +0000 UTC m=+22.865827146" watchObservedRunningTime="2026-07-28 17:23:39.459939143 +0000 UTC m=+22.866212883"
Jul 28 17:23:40.147217 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:40.147187    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9c8w8"
Jul 28 17:23:40.147818 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:40.147338    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9c8w8" podUID="2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4"
Jul 28 17:23:40.332900 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:40.332823    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7pp8b" event={"ID":"d50b403c-3842-4b2f-a314-e65583a0b225","Type":"ContainerStarted","Data":"3c848384b5f3a5b46d5b9c897784ca59db13ecee01fef753a9e36877e21b7008"}
Jul 28 17:23:40.335047 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:40.335023    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-177.ec2.internal" event={"ID":"1bb5c2c1c39b3c131016259171f34c67","Type":"ContainerStarted","Data":"5111cf54ff43399e44b37e8e2af1b82f6e18f350c989c4874c62d4eef823f332"}
Jul 28 17:23:40.338620 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:40.338595    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-64lk7" event={"ID":"bb8012ba-acac-4f19-823c-2339cc6fc563","Type":"ContainerStarted","Data":"b3db4d089db23ed91ea97fd2519691b42faf2ef153e1eff01bcadd22e16a7f28"}
Jul 28 17:23:40.338747 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:40.338674    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-64lk7" event={"ID":"bb8012ba-acac-4f19-823c-2339cc6fc563","Type":"ContainerStarted","Data":"4340b4af767eb80942ddab3c4b428ddb81cfe113b5ab1a3a07955f29dde8ba98"}
Jul 28 17:23:40.338747 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:40.338727    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-64lk7" event={"ID":"bb8012ba-acac-4f19-823c-2339cc6fc563","Type":"ContainerStarted","Data":"90e552802c43664edccdc806e2be3414f2f67dd21ba4a448d935285e414e5122"}
Jul 28 17:23:40.348893 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:40.348856    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7pp8b" podStartSLOduration=3.135936081 podStartE2EDuration="23.348837861s" totalImagesPullingTime="20.21290178s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:40.348563899 +0000 UTC m=+23.754837644" watchObservedRunningTime="2026-07-28 17:23:40.348837861 +0000 UTC m=+23.755111604"
Jul 28 17:23:40.365044 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:40.364998    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-177.ec2.internal" podStartSLOduration=22.364984318 podStartE2EDuration="22.364984318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:40.364778309 +0000 UTC m=+23.771052053" watchObservedRunningTime="2026-07-28 17:23:40.364984318 +0000 UTC m=+23.771258059"
Jul 28 17:23:40.467367 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:40.467345    2543 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 17:23:40.980893 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:40.980781    2543 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T17:23:40.467361778Z","UUID":"685cb514-35df-4238-94f9-80c766ac294a","Handler":null,"Name":"","Endpoint":""}
Jul 28 17:23:40.982580 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:40.982553    2543 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 17:23:40.982728 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:40.982616    2543 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 17:23:41.146932 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:41.146897    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:23:41.147106 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:41.146951    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:41.147192 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:41.147160    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjtrc" podUID="a2c1c169-0fd2-4e73-a497-0b69fe85d11f"
Jul 28 17:23:41.147352 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:41.147260    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtb46" podUID="145627c2-9b95-4251-9f16-2920555de2fa"
Jul 28 17:23:41.343220 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:41.343182    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jmn4" event={"ID":"afa69cf5-92d6-4d5b-b15c-242b555e08aa","Type":"ContainerStarted","Data":"e94d2ba8d236a7e703e091576879b175a96b38fb8e38af9ae656645436a5798b"}
Jul 28 17:23:42.146804 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:42.146772    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9c8w8"
Jul 28 17:23:42.146993 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:42.146901    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9c8w8" podUID="2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4"
Jul 28 17:23:42.348151 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:42.348107    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jmn4" event={"ID":"afa69cf5-92d6-4d5b-b15c-242b555e08aa","Type":"ContainerStarted","Data":"28ef23c6c5ee341e7f14f7fc4bf7ac56deca770ee5f111f3b95f5d2f87f7ebd5"}
Jul 28 17:23:42.351820 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:42.351792    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-64lk7" event={"ID":"bb8012ba-acac-4f19-823c-2339cc6fc563","Type":"ContainerStarted","Data":"28bb94c5eccb24a791255a413958d8316c7765b59c7117848b4ddec2b94b0ea5"}
Jul 28 17:23:42.365145 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:42.365088    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6jmn4" podStartSLOduration=3.00121086 podStartE2EDuration="25.36507617s" totalImagesPullingTime="22.36386531s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:23:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:42.364802594 +0000 UTC m=+25.771076337" watchObservedRunningTime="2026-07-28 17:23:42.36507617 +0000 UTC m=+25.771349912"
Jul 28 17:23:43.146567 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:43.146536    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:23:43.146745 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:43.146536    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:43.146745 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:43.146690    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjtrc" podUID="a2c1c169-0fd2-4e73-a497-0b69fe85d11f"
Jul 28 17:23:43.146845 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:43.146746    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtb46" podUID="145627c2-9b95-4251-9f16-2920555de2fa"
Jul 28 17:23:43.575951 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:43.575920    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7qw2x"
Jul 28 17:23:43.576610 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:43.576593    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7qw2x"
Jul 28 17:23:44.146914 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:44.146708    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9c8w8"
Jul 28 17:23:44.147045 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:44.146942    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9c8w8" podUID="2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4"
Jul 28 17:23:44.357219 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:44.357187    2543 generic.go:356] "Generic (PLEG): container finished" podID="849d3b0f-9f02-4f4d-8811-13794309755a" containerID="2aa2112b31553350e808a796b2aad793ae0fd98670f6319d26aac7893458408c" exitCode=0
Jul 28 17:23:44.357371 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:44.357279    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ljfhd" event={"ID":"849d3b0f-9f02-4f4d-8811-13794309755a","Type":"ContainerDied","Data":"2aa2112b31553350e808a796b2aad793ae0fd98670f6319d26aac7893458408c"}
Jul 28 17:23:44.361107 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:44.361086    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-64lk7" event={"ID":"bb8012ba-acac-4f19-823c-2339cc6fc563","Type":"ContainerStarted","Data":"160c4d14c03e22692cb65e98bfd63f4bd9c7464ecb9d8ba3babb0bb95ef3079c"}
Jul 28 17:23:44.361440 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:44.361424    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7qw2x"
Jul 28 17:23:44.361788 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:44.361770    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7qw2x"
Jul 28 17:23:44.402911 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:44.402838    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-64lk7" podStartSLOduration=6.922869156 podStartE2EDuration="27.402825784s" totalImagesPullingTime="20.479956628s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:44.402100209 +0000 UTC m=+27.808373950" watchObservedRunningTime="2026-07-28 17:23:44.402825784 +0000 UTC m=+27.809099525"
Jul 28 17:23:45.159505 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:45.158879    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:23:45.159505 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:45.159071    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjtrc" podUID="a2c1c169-0fd2-4e73-a497-0b69fe85d11f"
Jul 28 17:23:45.159505 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:45.159322    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:45.159505 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:45.159458    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtb46" podUID="145627c2-9b95-4251-9f16-2920555de2fa"
Jul 28 17:23:45.363383 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:45.363287    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:45.363564 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:45.363549    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:45.363659 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:45.363644    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:45.378588 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:45.378563    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:45.378733 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:45.378684    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:23:45.845295 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:45.845029    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jtb46"]
Jul 28 17:23:45.845470 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:45.845413    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:45.845538 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:45.845505    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtb46" podUID="145627c2-9b95-4251-9f16-2920555de2fa"
Jul 28 17:23:45.845674 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:45.845654    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9c8w8"]
Jul 28 17:23:45.845770 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:45.845756    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9c8w8"
Jul 28 17:23:45.845865 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:45.845845    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9c8w8" podUID="2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4"
Jul 28 17:23:45.857942 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:45.857918    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sjtrc"]
Jul 28 17:23:45.858082 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:45.857990    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:23:45.858082 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:45.858064    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjtrc" podUID="a2c1c169-0fd2-4e73-a497-0b69fe85d11f"
Jul 28 17:23:46.366810 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:46.366775    2543 generic.go:356] "Generic (PLEG): container finished" podID="849d3b0f-9f02-4f4d-8811-13794309755a" containerID="f36aa4a043ff5e18244f255821ba52f273dd33dee9b8a676ad619930257f5688" exitCode=0
Jul 28 17:23:46.367335 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:46.366866    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ljfhd" event={"ID":"849d3b0f-9f02-4f4d-8811-13794309755a","Type":"ContainerDied","Data":"f36aa4a043ff5e18244f255821ba52f273dd33dee9b8a676ad619930257f5688"}
Jul 28 17:23:47.152441 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:47.151761    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9c8w8"
Jul 28 17:23:47.152441 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:47.151921    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9c8w8" podUID="2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4"
Jul 28 17:23:48.147372 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:48.147292    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:23:48.147776 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:48.147292    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:48.147776 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:48.147420    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjtrc" podUID="a2c1c169-0fd2-4e73-a497-0b69fe85d11f"
Jul 28 17:23:48.147776 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:48.147461    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtb46" podUID="145627c2-9b95-4251-9f16-2920555de2fa"
Jul 28 17:23:48.373394 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:48.373363    2543 generic.go:356] "Generic (PLEG): container finished" podID="849d3b0f-9f02-4f4d-8811-13794309755a" containerID="fe0e45cb156ad764d75720631b87ca1b00d905a7c88b648c5d67ab628a2d8fdd" exitCode=0
Jul 28 17:23:48.373533 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:48.373403    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ljfhd" event={"ID":"849d3b0f-9f02-4f4d-8811-13794309755a","Type":"ContainerDied","Data":"fe0e45cb156ad764d75720631b87ca1b00d905a7c88b648c5d67ab628a2d8fdd"}
Jul 28 17:23:49.147443 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:49.147200    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9c8w8"
Jul 28 17:23:49.147925 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:49.147550    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9c8w8" podUID="2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4"
Jul 28 17:23:49.783423 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:49.783387    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c1c169-0fd2-4e73-a497-0b69fe85d11f-metrics-certs\") pod \"network-metrics-daemon-sjtrc\" (UID: \"a2c1c169-0fd2-4e73-a497-0b69fe85d11f\") " pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:23:49.783627 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:49.783452    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/145627c2-9b95-4251-9f16-2920555de2fa-original-pull-secret\") pod \"global-pull-secret-syncer-jtb46\" (UID: \"145627c2-9b95-4251-9f16-2920555de2fa\") " pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:49.783627 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:49.783560    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:23:49.783627 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:49.783574    2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:23:49.783765 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:49.783638    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/145627c2-9b95-4251-9f16-2920555de2fa-original-pull-secret podName:145627c2-9b95-4251-9f16-2920555de2fa nodeName:}" failed. No retries permitted until 2026-07-28 17:24:21.783618017 +0000 UTC m=+65.189891755 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/145627c2-9b95-4251-9f16-2920555de2fa-original-pull-secret") pod "global-pull-secret-syncer-jtb46" (UID: "145627c2-9b95-4251-9f16-2920555de2fa") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:23:49.783765 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:49.783662    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2c1c169-0fd2-4e73-a497-0b69fe85d11f-metrics-certs podName:a2c1c169-0fd2-4e73-a497-0b69fe85d11f nodeName:}" failed. No retries permitted until 2026-07-28 17:24:21.783651898 +0000 UTC m=+65.189925621 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2c1c169-0fd2-4e73-a497-0b69fe85d11f-metrics-certs") pod "network-metrics-daemon-sjtrc" (UID: "a2c1c169-0fd2-4e73-a497-0b69fe85d11f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:23:49.883962 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:49.883931    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w89v\" (UniqueName: \"kubernetes.io/projected/2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4-kube-api-access-7w89v\") pod \"network-check-target-9c8w8\" (UID: \"2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4\") " pod="openshift-network-diagnostics/network-check-target-9c8w8"
Jul 28 17:23:49.884156 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:49.884068    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:23:49.884156 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:49.884088    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:23:49.884156 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:49.884099    2543 projected.go:196] Error preparing data for projected volume kube-api-access-7w89v for pod openshift-network-diagnostics/network-check-target-9c8w8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:23:49.884323 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:49.884177    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4-kube-api-access-7w89v podName:2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:21.884157867 +0000 UTC m=+65.290431610 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7w89v" (UniqueName: "kubernetes.io/projected/2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4-kube-api-access-7w89v") pod "network-check-target-9c8w8" (UID: "2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:23:50.147070 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:50.146984    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:23:50.147070 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:50.146981    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:50.147312 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:50.147127    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjtrc" podUID="a2c1c169-0fd2-4e73-a497-0b69fe85d11f"
Jul 28 17:23:50.147312 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:50.147235    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtb46" podUID="145627c2-9b95-4251-9f16-2920555de2fa"
Jul 28 17:23:51.147115 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:51.147080    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9c8w8"
Jul 28 17:23:51.147516 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:51.147229    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9c8w8" podUID="2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4"
Jul 28 17:23:52.146389 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.146357    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:23:52.146561 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.146357    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:52.146561 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:52.146480    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjtrc" podUID="a2c1c169-0fd2-4e73-a497-0b69fe85d11f"
Jul 28 17:23:52.146700 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:52.146580    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtb46" podUID="145627c2-9b95-4251-9f16-2920555de2fa"
Jul 28 17:23:52.456893 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.456830    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-177.ec2.internal" event="NodeReady"
Jul 28 17:23:52.457254 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.456974    2543 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 17:23:52.506070 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.506035    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hg8ms"]
Jul 28 17:23:52.518726 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.518701    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9r84r"]
Jul 28 17:23:52.518859 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.518837    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hg8ms"
Jul 28 17:23:52.521114 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.521078    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 17:23:52.521255 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.521116    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gw5z2\""
Jul 28 17:23:52.521407 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.521390    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 17:23:52.532501 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.532481    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hg8ms"]
Jul 28 17:23:52.532601 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.532513    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9r84r"]
Jul 28 17:23:52.532601 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.532591    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9r84r"
Jul 28 17:23:52.534864 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.534845    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 17:23:52.534973 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.534886    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-m7mlr\""
Jul 28 17:23:52.534973 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.534912    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 17:23:52.535110 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.534975    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 17:23:52.702329 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.702298    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3af720f1-3267-48f6-94e2-6f6527c0fcf6-tmp-dir\") pod \"dns-default-hg8ms\" (UID: \"3af720f1-3267-48f6-94e2-6f6527c0fcf6\") " pod="openshift-dns/dns-default-hg8ms"
Jul 28 17:23:52.702329 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.702326    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a43fd4b-3d38-44b5-9b92-de7f6228df26-cert\") pod \"ingress-canary-9r84r\" (UID: \"6a43fd4b-3d38-44b5-9b92-de7f6228df26\") " pod="openshift-ingress-canary/ingress-canary-9r84r"
Jul 28 17:23:52.702510 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.702400    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh2pn\" (UniqueName: \"kubernetes.io/projected/3af720f1-3267-48f6-94e2-6f6527c0fcf6-kube-api-access-rh2pn\") pod \"dns-default-hg8ms\" (UID: \"3af720f1-3267-48f6-94e2-6f6527c0fcf6\") " pod="openshift-dns/dns-default-hg8ms"
Jul 28 17:23:52.702510 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.702426    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3af720f1-3267-48f6-94e2-6f6527c0fcf6-config-volume\") pod \"dns-default-hg8ms\" (UID: \"3af720f1-3267-48f6-94e2-6f6527c0fcf6\") " pod="openshift-dns/dns-default-hg8ms"
Jul 28 17:23:52.702510 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.702445    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3af720f1-3267-48f6-94e2-6f6527c0fcf6-metrics-tls\") pod \"dns-default-hg8ms\" (UID: \"3af720f1-3267-48f6-94e2-6f6527c0fcf6\") " pod="openshift-dns/dns-default-hg8ms"
Jul 28 17:23:52.702510 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.702478    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hpq4\" (UniqueName: \"kubernetes.io/projected/6a43fd4b-3d38-44b5-9b92-de7f6228df26-kube-api-access-5hpq4\") pod \"ingress-canary-9r84r\" (UID: \"6a43fd4b-3d38-44b5-9b92-de7f6228df26\") " pod="openshift-ingress-canary/ingress-canary-9r84r"
Jul 28 17:23:52.803415 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.803383    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3af720f1-3267-48f6-94e2-6f6527c0fcf6-metrics-tls\") pod \"dns-default-hg8ms\" (UID: \"3af720f1-3267-48f6-94e2-6f6527c0fcf6\") " pod="openshift-dns/dns-default-hg8ms"
Jul 28 17:23:52.803492 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.803433    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hpq4\" (UniqueName: \"kubernetes.io/projected/6a43fd4b-3d38-44b5-9b92-de7f6228df26-kube-api-access-5hpq4\") pod \"ingress-canary-9r84r\" (UID: \"6a43fd4b-3d38-44b5-9b92-de7f6228df26\") " pod="openshift-ingress-canary/ingress-canary-9r84r"
Jul 28 17:23:52.803492 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.803461    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3af720f1-3267-48f6-94e2-6f6527c0fcf6-tmp-dir\") pod \"dns-default-hg8ms\" (UID: \"3af720f1-3267-48f6-94e2-6f6527c0fcf6\") " pod="openshift-dns/dns-default-hg8ms"
Jul 28 17:23:52.803492 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.803478    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a43fd4b-3d38-44b5-9b92-de7f6228df26-cert\") pod \"ingress-canary-9r84r\" (UID: \"6a43fd4b-3d38-44b5-9b92-de7f6228df26\") " pod="openshift-ingress-canary/ingress-canary-9r84r"
Jul 28 17:23:52.803631 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.803533    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh2pn\" (UniqueName: \"kubernetes.io/projected/3af720f1-3267-48f6-94e2-6f6527c0fcf6-kube-api-access-rh2pn\") pod \"dns-default-hg8ms\" (UID: \"3af720f1-3267-48f6-94e2-6f6527c0fcf6\") " pod="openshift-dns/dns-default-hg8ms"
Jul 28 17:23:52.803631 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:52.803546    2543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:23:52.803631 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.803565    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3af720f1-3267-48f6-94e2-6f6527c0fcf6-config-volume\") pod \"dns-default-hg8ms\" (UID: \"3af720f1-3267-48f6-94e2-6f6527c0fcf6\") " pod="openshift-dns/dns-default-hg8ms"
Jul 28 17:23:52.803631 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:52.803612    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af720f1-3267-48f6-94e2-6f6527c0fcf6-metrics-tls podName:3af720f1-3267-48f6-94e2-6f6527c0fcf6 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:53.303589818 +0000 UTC m=+36.709863558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3af720f1-3267-48f6-94e2-6f6527c0fcf6-metrics-tls") pod "dns-default-hg8ms" (UID: "3af720f1-3267-48f6-94e2-6f6527c0fcf6") : secret "dns-default-metrics-tls" not found
Jul 28 17:23:52.803839 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.803776    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3af720f1-3267-48f6-94e2-6f6527c0fcf6-tmp-dir\") pod \"dns-default-hg8ms\" (UID: \"3af720f1-3267-48f6-94e2-6f6527c0fcf6\") " pod="openshift-dns/dns-default-hg8ms"
Jul 28 17:23:52.804046 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.804030    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3af720f1-3267-48f6-94e2-6f6527c0fcf6-config-volume\") pod \"dns-default-hg8ms\" (UID: \"3af720f1-3267-48f6-94e2-6f6527c0fcf6\") " pod="openshift-dns/dns-default-hg8ms"
Jul 28 17:23:52.808881 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.808859    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a43fd4b-3d38-44b5-9b92-de7f6228df26-cert\") pod \"ingress-canary-9r84r\" (UID: \"6a43fd4b-3d38-44b5-9b92-de7f6228df26\") " pod="openshift-ingress-canary/ingress-canary-9r84r"
Jul 28 17:23:52.815174 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.815153    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh2pn\" (UniqueName: \"kubernetes.io/projected/3af720f1-3267-48f6-94e2-6f6527c0fcf6-kube-api-access-rh2pn\") pod \"dns-default-hg8ms\" (UID: \"3af720f1-3267-48f6-94e2-6f6527c0fcf6\") " pod="openshift-dns/dns-default-hg8ms"
Jul 28 17:23:52.815384 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.815368    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hpq4\" (UniqueName: \"kubernetes.io/projected/6a43fd4b-3d38-44b5-9b92-de7f6228df26-kube-api-access-5hpq4\") pod \"ingress-canary-9r84r\" (UID: \"6a43fd4b-3d38-44b5-9b92-de7f6228df26\") " pod="openshift-ingress-canary/ingress-canary-9r84r"
Jul 28 17:23:52.840687 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:52.840668    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9r84r"
Jul 28 17:23:53.030550 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:53.030472    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9r84r"]
Jul 28 17:23:53.035485 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:23:53.035459    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a43fd4b_3d38_44b5_9b92_de7f6228df26.slice/crio-0b9b8be324cd68692a74eb479c507b871e40453c071eb18d0d6c59788f2387c5 WatchSource:0}: Error finding container 0b9b8be324cd68692a74eb479c507b871e40453c071eb18d0d6c59788f2387c5: Status 404 returned error can't find the container with id 0b9b8be324cd68692a74eb479c507b871e40453c071eb18d0d6c59788f2387c5
Jul 28 17:23:53.146734 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:53.146704    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9c8w8"
Jul 28 17:23:53.149698 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:53.149677    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:23:53.149903 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:53.149884    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rqpml\""
Jul 28 17:23:53.150158 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:53.150127    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:23:53.310047 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:53.309947    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3af720f1-3267-48f6-94e2-6f6527c0fcf6-metrics-tls\") pod \"dns-default-hg8ms\" (UID: \"3af720f1-3267-48f6-94e2-6f6527c0fcf6\") " pod="openshift-dns/dns-default-hg8ms"
Jul 28 17:23:53.310223 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:53.310107    2543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:23:53.310223 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:53.310185    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af720f1-3267-48f6-94e2-6f6527c0fcf6-metrics-tls podName:3af720f1-3267-48f6-94e2-6f6527c0fcf6 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:54.310169799 +0000 UTC m=+37.716443522 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3af720f1-3267-48f6-94e2-6f6527c0fcf6-metrics-tls") pod "dns-default-hg8ms" (UID: "3af720f1-3267-48f6-94e2-6f6527c0fcf6") : secret "dns-default-metrics-tls" not found
Jul 28 17:23:53.387657 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:53.387628    2543 generic.go:356] "Generic (PLEG): container finished" podID="849d3b0f-9f02-4f4d-8811-13794309755a" containerID="e38610705351369ec9aec838fef7ae6db8a564e9891133a8db8cccc300e684e3" exitCode=0
Jul 28 17:23:53.387815 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:53.387693    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ljfhd" event={"ID":"849d3b0f-9f02-4f4d-8811-13794309755a","Type":"ContainerDied","Data":"e38610705351369ec9aec838fef7ae6db8a564e9891133a8db8cccc300e684e3"}
Jul 28 17:23:53.388903 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:53.388885    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9r84r" event={"ID":"6a43fd4b-3d38-44b5-9b92-de7f6228df26","Type":"ContainerStarted","Data":"0b9b8be324cd68692a74eb479c507b871e40453c071eb18d0d6c59788f2387c5"}
Jul 28 17:23:54.146540 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:54.146468    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:23:54.146540 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:54.146493    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:23:54.149441 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:54.149415    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:23:54.149673 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:54.149657    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dpv4x\""
Jul 28 17:23:54.149771 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:54.149701    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:23:54.316775 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:54.316742    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3af720f1-3267-48f6-94e2-6f6527c0fcf6-metrics-tls\") pod \"dns-default-hg8ms\" (UID: \"3af720f1-3267-48f6-94e2-6f6527c0fcf6\") " pod="openshift-dns/dns-default-hg8ms"
Jul 28 17:23:54.316956 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:54.316916    2543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:23:54.317022 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:54.316997    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af720f1-3267-48f6-94e2-6f6527c0fcf6-metrics-tls podName:3af720f1-3267-48f6-94e2-6f6527c0fcf6 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:56.316973359 +0000 UTC m=+39.723247078 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3af720f1-3267-48f6-94e2-6f6527c0fcf6-metrics-tls") pod "dns-default-hg8ms" (UID: "3af720f1-3267-48f6-94e2-6f6527c0fcf6") : secret "dns-default-metrics-tls" not found
Jul 28 17:23:54.395596 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:54.395559    2543 generic.go:356] "Generic (PLEG): container finished" podID="849d3b0f-9f02-4f4d-8811-13794309755a" containerID="23fce6c36058f637e3145145f325aa86e7d1d33d658c2d9f0de192ab2f55a254" exitCode=0
Jul 28 17:23:54.395753 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:54.395613    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ljfhd" event={"ID":"849d3b0f-9f02-4f4d-8811-13794309755a","Type":"ContainerDied","Data":"23fce6c36058f637e3145145f325aa86e7d1d33d658c2d9f0de192ab2f55a254"}
Jul 28 17:23:55.402208 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:55.402177    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ljfhd" event={"ID":"849d3b0f-9f02-4f4d-8811-13794309755a","Type":"ContainerStarted","Data":"b2e6737cf748d6df8324f6565eed53e7a40cd2a7262cfc25a28acb20a9bedb38"}
Jul 28 17:23:55.403711 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:55.403682    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9r84r" event={"ID":"6a43fd4b-3d38-44b5-9b92-de7f6228df26","Type":"ContainerStarted","Data":"0a117173f6f58ffb2d560b52e8307f85f6a8f258e2cac3bb51dffb24327211c6"}
Jul 28 17:23:55.435352 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:55.435307    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ljfhd" podStartSLOduration=7.072419755 podStartE2EDuration="38.435291266s" totalImagesPullingTime="31.362871511s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:23:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:55.43492432 +0000 UTC m=+38.841198061" watchObservedRunningTime="2026-07-28 17:23:55.435291266 +0000 UTC m=+38.841565006"
Jul 28 17:23:55.467104 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:55.467059    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9r84r" podStartSLOduration=1.245085794 podStartE2EDuration="3.467042889s" totalImagesPullingTime="2.221957095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:55.465494055 +0000 UTC m=+38.871767806" watchObservedRunningTime="2026-07-28 17:23:55.467042889 +0000 UTC m=+38.873316660"
Jul 28 17:23:56.334766 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:56.334582    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3af720f1-3267-48f6-94e2-6f6527c0fcf6-metrics-tls\") pod \"dns-default-hg8ms\" (UID: \"3af720f1-3267-48f6-94e2-6f6527c0fcf6\") " pod="openshift-dns/dns-default-hg8ms"
Jul 28 17:23:56.334919 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:56.334731    2543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:23:56.334919 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:56.334868    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af720f1-3267-48f6-94e2-6f6527c0fcf6-metrics-tls podName:3af720f1-3267-48f6-94e2-6f6527c0fcf6 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:00.334853238 +0000 UTC m=+43.741126957 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3af720f1-3267-48f6-94e2-6f6527c0fcf6-metrics-tls") pod "dns-default-hg8ms" (UID: "3af720f1-3267-48f6-94e2-6f6527c0fcf6") : secret "dns-default-metrics-tls" not found
Jul 28 17:23:57.164753 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:57.164721    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5657cc6765-srj79"]
Jul 28 17:23:57.166525 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:57.166510    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:23:57.170892 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:57.170875    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 28 17:23:57.170980 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:57.170874    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 17:23:57.171416 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:57.171368    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 28 17:23:57.171648 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:57.171632    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 28 17:23:57.175590 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:57.175574    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 28 17:23:57.175694 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:57.175680    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-5fcgw\""
Jul 28 17:23:57.175823 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:57.175808    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 17:23:57.192002 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:57.191982    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5657cc6765-srj79"]
Jul 28 17:23:57.240807 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:57.240787    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/908c16e7-90b9-4f02-a950-e62eafb70d81-stats-auth\") pod \"router-default-5657cc6765-srj79\" (UID: \"908c16e7-90b9-4f02-a950-e62eafb70d81\") " pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:23:57.240940 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:57.240829    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/908c16e7-90b9-4f02-a950-e62eafb70d81-metrics-certs\") pod \"router-default-5657cc6765-srj79\" (UID: \"908c16e7-90b9-4f02-a950-e62eafb70d81\") " pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:23:57.240940 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:57.240848    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhcbl\" (UniqueName: \"kubernetes.io/projected/908c16e7-90b9-4f02-a950-e62eafb70d81-kube-api-access-fhcbl\") pod \"router-default-5657cc6765-srj79\" (UID: \"908c16e7-90b9-4f02-a950-e62eafb70d81\") " pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:23:57.241055 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:57.240970    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/908c16e7-90b9-4f02-a950-e62eafb70d81-service-ca-bundle\") pod \"router-default-5657cc6765-srj79\" (UID: \"908c16e7-90b9-4f02-a950-e62eafb70d81\") " pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:23:57.241107 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:57.241050    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/908c16e7-90b9-4f02-a950-e62eafb70d81-default-certificate\") pod \"router-default-5657cc6765-srj79\" (UID: \"908c16e7-90b9-4f02-a950-e62eafb70d81\") " pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:23:57.342087 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:57.342062    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/908c16e7-90b9-4f02-a950-e62eafb70d81-service-ca-bundle\") pod \"router-default-5657cc6765-srj79\" (UID: \"908c16e7-90b9-4f02-a950-e62eafb70d81\") " pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:23:57.342246 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:57.342112    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/908c16e7-90b9-4f02-a950-e62eafb70d81-default-certificate\") pod \"router-default-5657cc6765-srj79\" (UID: \"908c16e7-90b9-4f02-a950-e62eafb70d81\") " pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:23:57.342305 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:57.342249    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/908c16e7-90b9-4f02-a950-e62eafb70d81-service-ca-bundle podName:908c16e7-90b9-4f02-a950-e62eafb70d81 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:57.842231713 +0000 UTC m=+41.248505432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/908c16e7-90b9-4f02-a950-e62eafb70d81-service-ca-bundle") pod "router-default-5657cc6765-srj79" (UID: "908c16e7-90b9-4f02-a950-e62eafb70d81") : configmap references non-existent config key: service-ca.crt
Jul 28 17:23:57.342305 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:57.342244    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/908c16e7-90b9-4f02-a950-e62eafb70d81-stats-auth\") pod \"router-default-5657cc6765-srj79\" (UID: \"908c16e7-90b9-4f02-a950-e62eafb70d81\") " pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:23:57.342305 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:57.342302    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/908c16e7-90b9-4f02-a950-e62eafb70d81-metrics-certs\") pod \"router-default-5657cc6765-srj79\" (UID: \"908c16e7-90b9-4f02-a950-e62eafb70d81\") " pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:23:57.342457 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:57.342329    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhcbl\" (UniqueName: \"kubernetes.io/projected/908c16e7-90b9-4f02-a950-e62eafb70d81-kube-api-access-fhcbl\") pod \"router-default-5657cc6765-srj79\" (UID: \"908c16e7-90b9-4f02-a950-e62eafb70d81\") " pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:23:57.342457 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:57.342397    2543 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:23:57.342457 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:57.342447    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/908c16e7-90b9-4f02-a950-e62eafb70d81-metrics-certs podName:908c16e7-90b9-4f02-a950-e62eafb70d81 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:57.842433137 +0000 UTC m=+41.248706870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/908c16e7-90b9-4f02-a950-e62eafb70d81-metrics-certs") pod "router-default-5657cc6765-srj79" (UID: "908c16e7-90b9-4f02-a950-e62eafb70d81") : secret "router-metrics-certs-default" not found
Jul 28 17:23:57.346120 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:57.346098    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/908c16e7-90b9-4f02-a950-e62eafb70d81-stats-auth\") pod \"router-default-5657cc6765-srj79\" (UID: \"908c16e7-90b9-4f02-a950-e62eafb70d81\") " pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:23:57.346222 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:57.346184    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/908c16e7-90b9-4f02-a950-e62eafb70d81-default-certificate\") pod \"router-default-5657cc6765-srj79\" (UID: \"908c16e7-90b9-4f02-a950-e62eafb70d81\") " pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:23:57.350854 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:57.350832    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhcbl\" (UniqueName: \"kubernetes.io/projected/908c16e7-90b9-4f02-a950-e62eafb70d81-kube-api-access-fhcbl\") pod \"router-default-5657cc6765-srj79\" (UID: \"908c16e7-90b9-4f02-a950-e62eafb70d81\") " pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:23:57.845761 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:57.845715    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/908c16e7-90b9-4f02-a950-e62eafb70d81-metrics-certs\") pod \"router-default-5657cc6765-srj79\" (UID: \"908c16e7-90b9-4f02-a950-e62eafb70d81\") " pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:23:57.845947 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:57.845784    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/908c16e7-90b9-4f02-a950-e62eafb70d81-service-ca-bundle\") pod \"router-default-5657cc6765-srj79\" (UID: \"908c16e7-90b9-4f02-a950-e62eafb70d81\") " pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:23:57.845947 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:57.845864    2543 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:23:57.845947 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:57.845924    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/908c16e7-90b9-4f02-a950-e62eafb70d81-metrics-certs podName:908c16e7-90b9-4f02-a950-e62eafb70d81 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:58.845910078 +0000 UTC m=+42.252183796 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/908c16e7-90b9-4f02-a950-e62eafb70d81-metrics-certs") pod "router-default-5657cc6765-srj79" (UID: "908c16e7-90b9-4f02-a950-e62eafb70d81") : secret "router-metrics-certs-default" not found
Jul 28 17:23:57.845947 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:57.845937    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/908c16e7-90b9-4f02-a950-e62eafb70d81-service-ca-bundle podName:908c16e7-90b9-4f02-a950-e62eafb70d81 nodeName:}" failed. No retries permitted until 2026-07-28 17:23:58.845931534 +0000 UTC m=+42.252205253 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/908c16e7-90b9-4f02-a950-e62eafb70d81-service-ca-bundle") pod "router-default-5657cc6765-srj79" (UID: "908c16e7-90b9-4f02-a950-e62eafb70d81") : configmap references non-existent config key: service-ca.crt
Jul 28 17:23:58.852246 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:58.852208    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/908c16e7-90b9-4f02-a950-e62eafb70d81-service-ca-bundle\") pod \"router-default-5657cc6765-srj79\" (UID: \"908c16e7-90b9-4f02-a950-e62eafb70d81\") " pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:23:58.852636 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:23:58.852285    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/908c16e7-90b9-4f02-a950-e62eafb70d81-metrics-certs\") pod \"router-default-5657cc6765-srj79\" (UID: \"908c16e7-90b9-4f02-a950-e62eafb70d81\") " pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:23:58.852636 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:58.852375    2543 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:23:58.852636 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:58.852377    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/908c16e7-90b9-4f02-a950-e62eafb70d81-service-ca-bundle podName:908c16e7-90b9-4f02-a950-e62eafb70d81 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:00.852359822 +0000 UTC m=+44.258633541 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/908c16e7-90b9-4f02-a950-e62eafb70d81-service-ca-bundle") pod "router-default-5657cc6765-srj79" (UID: "908c16e7-90b9-4f02-a950-e62eafb70d81") : configmap references non-existent config key: service-ca.crt
Jul 28 17:23:58.852636 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:23:58.852422    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/908c16e7-90b9-4f02-a950-e62eafb70d81-metrics-certs podName:908c16e7-90b9-4f02-a950-e62eafb70d81 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:00.852410967 +0000 UTC m=+44.258684686 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/908c16e7-90b9-4f02-a950-e62eafb70d81-metrics-certs") pod "router-default-5657cc6765-srj79" (UID: "908c16e7-90b9-4f02-a950-e62eafb70d81") : secret "router-metrics-certs-default" not found
Jul 28 17:24:00.364329 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:00.364296    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3af720f1-3267-48f6-94e2-6f6527c0fcf6-metrics-tls\") pod \"dns-default-hg8ms\" (UID: \"3af720f1-3267-48f6-94e2-6f6527c0fcf6\") " pod="openshift-dns/dns-default-hg8ms"
Jul 28 17:24:00.364682 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:00.364453    2543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:24:00.364682 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:00.364524    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af720f1-3267-48f6-94e2-6f6527c0fcf6-metrics-tls podName:3af720f1-3267-48f6-94e2-6f6527c0fcf6 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:08.364508387 +0000 UTC m=+51.770782106 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3af720f1-3267-48f6-94e2-6f6527c0fcf6-metrics-tls") pod "dns-default-hg8ms" (UID: "3af720f1-3267-48f6-94e2-6f6527c0fcf6") : secret "dns-default-metrics-tls" not found
Jul 28 17:24:00.429805 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:00.429777    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-czx2d"]
Jul 28 17:24:00.460656 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:00.460633    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-czx2d"]
Jul 28 17:24:00.460796 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:00.460730    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-czx2d"
Jul 28 17:24:00.463356 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:00.463320    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b7f8h\""
Jul 28 17:24:00.463471 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:00.463405    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 17:24:00.464216 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:00.464201    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 17:24:00.464297 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:00.464225    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 17:24:00.464297 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:00.464238    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 17:24:00.565699 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:00.565662    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/66911cb6-549d-4f4f-8ac6-286401ad1025-data-volume\") pod \"insights-runtime-extractor-czx2d\" (UID: \"66911cb6-549d-4f4f-8ac6-286401ad1025\") " pod="openshift-insights/insights-runtime-extractor-czx2d"
Jul 28 17:24:00.565832 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:00.565781    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/66911cb6-549d-4f4f-8ac6-286401ad1025-crio-socket\") pod \"insights-runtime-extractor-czx2d\" (UID: \"66911cb6-549d-4f4f-8ac6-286401ad1025\") " pod="openshift-insights/insights-runtime-extractor-czx2d"
Jul 28 17:24:00.565880 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:00.565833    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/66911cb6-549d-4f4f-8ac6-286401ad1025-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-czx2d\" (UID: \"66911cb6-549d-4f4f-8ac6-286401ad1025\") " pod="openshift-insights/insights-runtime-extractor-czx2d"
Jul 28 17:24:00.565880 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:00.565856    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5vtv\" (UniqueName: \"kubernetes.io/projected/66911cb6-549d-4f4f-8ac6-286401ad1025-kube-api-access-z5vtv\") pod \"insights-runtime-extractor-czx2d\" (UID: \"66911cb6-549d-4f4f-8ac6-286401ad1025\") " pod="openshift-insights/insights-runtime-extractor-czx2d"
Jul 28 17:24:00.565957 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:00.565940    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/66911cb6-549d-4f4f-8ac6-286401ad1025-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-czx2d\" (UID: \"66911cb6-549d-4f4f-8ac6-286401ad1025\") " pod="openshift-insights/insights-runtime-extractor-czx2d"
Jul 28 17:24:00.667280 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:00.667212    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/66911cb6-549d-4f4f-8ac6-286401ad1025-crio-socket\") pod \"insights-runtime-extractor-czx2d\" (UID: \"66911cb6-549d-4f4f-8ac6-286401ad1025\") " pod="openshift-insights/insights-runtime-extractor-czx2d"
Jul 28 17:24:00.667280 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:00.667249    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/66911cb6-549d-4f4f-8ac6-286401ad1025-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-czx2d\" (UID: \"66911cb6-549d-4f4f-8ac6-286401ad1025\") " pod="openshift-insights/insights-runtime-extractor-czx2d"
Jul 28 17:24:00.667280 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:00.667269    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5vtv\" (UniqueName: \"kubernetes.io/projected/66911cb6-549d-4f4f-8ac6-286401ad1025-kube-api-access-z5vtv\") pod \"insights-runtime-extractor-czx2d\" (UID: \"66911cb6-549d-4f4f-8ac6-286401ad1025\") " pod="openshift-insights/insights-runtime-extractor-czx2d"
Jul 28 17:24:00.667488 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:00.667320    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/66911cb6-549d-4f4f-8ac6-286401ad1025-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-czx2d\" (UID: \"66911cb6-549d-4f4f-8ac6-286401ad1025\") " pod="openshift-insights/insights-runtime-extractor-czx2d"
Jul 28 17:24:00.667488 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:00.667347    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/66911cb6-549d-4f4f-8ac6-286401ad1025-data-volume\") pod \"insights-runtime-extractor-czx2d\" (UID: \"66911cb6-549d-4f4f-8ac6-286401ad1025\") " pod="openshift-insights/insights-runtime-extractor-czx2d"
Jul 28 17:24:00.667488 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:00.667357    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/66911cb6-549d-4f4f-8ac6-286401ad1025-crio-socket\") pod \"insights-runtime-extractor-czx2d\" (UID: \"66911cb6-549d-4f4f-8ac6-286401ad1025\") " pod="openshift-insights/insights-runtime-extractor-czx2d"
Jul 28 17:24:00.667488 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:00.667402    2543 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:24:00.667488 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:00.667434    2543 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:24:00.667488 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:00.667468    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66911cb6-549d-4f4f-8ac6-286401ad1025-insights-runtime-extractor-tls podName:66911cb6-549d-4f4f-8ac6-286401ad1025 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:01.167448558 +0000 UTC m=+44.573722279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/66911cb6-549d-4f4f-8ac6-286401ad1025-insights-runtime-extractor-tls") pod "insights-runtime-extractor-czx2d" (UID: "66911cb6-549d-4f4f-8ac6-286401ad1025") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:24:00.667676 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:00.667496    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66911cb6-549d-4f4f-8ac6-286401ad1025-kube-rbac-proxy-cm podName:66911cb6-549d-4f4f-8ac6-286401ad1025 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:01.167479208 +0000 UTC m=+44.573752930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/66911cb6-549d-4f4f-8ac6-286401ad1025-kube-rbac-proxy-cm") pod "insights-runtime-extractor-czx2d" (UID: "66911cb6-549d-4f4f-8ac6-286401ad1025") : configmap "kube-rbac-proxy" not found
Jul 28 17:24:00.667676 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:00.667617    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/66911cb6-549d-4f4f-8ac6-286401ad1025-data-volume\") pod \"insights-runtime-extractor-czx2d\" (UID: \"66911cb6-549d-4f4f-8ac6-286401ad1025\") " pod="openshift-insights/insights-runtime-extractor-czx2d"
Jul 28 17:24:00.675963 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:00.675944    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5vtv\" (UniqueName: \"kubernetes.io/projected/66911cb6-549d-4f4f-8ac6-286401ad1025-kube-api-access-z5vtv\") pod \"insights-runtime-extractor-czx2d\" (UID: \"66911cb6-549d-4f4f-8ac6-286401ad1025\") " pod="openshift-insights/insights-runtime-extractor-czx2d"
Jul 28 17:24:00.867966 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:00.867934    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/908c16e7-90b9-4f02-a950-e62eafb70d81-metrics-certs\") pod \"router-default-5657cc6765-srj79\" (UID: \"908c16e7-90b9-4f02-a950-e62eafb70d81\") " pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:24:00.868118 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:00.868050    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/908c16e7-90b9-4f02-a950-e62eafb70d81-service-ca-bundle\") pod \"router-default-5657cc6765-srj79\" (UID: \"908c16e7-90b9-4f02-a950-e62eafb70d81\") " pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:24:00.868118 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:00.868089    2543 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:24:00.868215 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:00.868186    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/908c16e7-90b9-4f02-a950-e62eafb70d81-metrics-certs podName:908c16e7-90b9-4f02-a950-e62eafb70d81 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:04.86816812 +0000 UTC m=+48.274441859 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/908c16e7-90b9-4f02-a950-e62eafb70d81-metrics-certs") pod "router-default-5657cc6765-srj79" (UID: "908c16e7-90b9-4f02-a950-e62eafb70d81") : secret "router-metrics-certs-default" not found
Jul 28 17:24:00.868256 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:00.868217    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/908c16e7-90b9-4f02-a950-e62eafb70d81-service-ca-bundle podName:908c16e7-90b9-4f02-a950-e62eafb70d81 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:04.868205943 +0000 UTC m=+48.274479677 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/908c16e7-90b9-4f02-a950-e62eafb70d81-service-ca-bundle") pod "router-default-5657cc6765-srj79" (UID: "908c16e7-90b9-4f02-a950-e62eafb70d81") : configmap references non-existent config key: service-ca.crt
Jul 28 17:24:01.169897 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:01.169864    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/66911cb6-549d-4f4f-8ac6-286401ad1025-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-czx2d\" (UID: \"66911cb6-549d-4f4f-8ac6-286401ad1025\") " pod="openshift-insights/insights-runtime-extractor-czx2d"
Jul 28 17:24:01.170069 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:01.169968    2543 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:24:01.170069 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:01.170024    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66911cb6-549d-4f4f-8ac6-286401ad1025-kube-rbac-proxy-cm podName:66911cb6-549d-4f4f-8ac6-286401ad1025 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:02.170008667 +0000 UTC m=+45.576282391 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/66911cb6-549d-4f4f-8ac6-286401ad1025-kube-rbac-proxy-cm") pod "insights-runtime-extractor-czx2d" (UID: "66911cb6-549d-4f4f-8ac6-286401ad1025") : configmap "kube-rbac-proxy" not found
Jul 28 17:24:01.170069 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:01.169971    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/66911cb6-549d-4f4f-8ac6-286401ad1025-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-czx2d\" (UID: \"66911cb6-549d-4f4f-8ac6-286401ad1025\") " pod="openshift-insights/insights-runtime-extractor-czx2d"
Jul 28 17:24:01.170069 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:01.170057    2543 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:24:01.170261 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:01.170117    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66911cb6-549d-4f4f-8ac6-286401ad1025-insights-runtime-extractor-tls podName:66911cb6-549d-4f4f-8ac6-286401ad1025 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:02.170100832 +0000 UTC m=+45.576374552 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/66911cb6-549d-4f4f-8ac6-286401ad1025-insights-runtime-extractor-tls") pod "insights-runtime-extractor-czx2d" (UID: "66911cb6-549d-4f4f-8ac6-286401ad1025") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:24:02.177036 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:02.177002    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/66911cb6-549d-4f4f-8ac6-286401ad1025-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-czx2d\" (UID: \"66911cb6-549d-4f4f-8ac6-286401ad1025\") " pod="openshift-insights/insights-runtime-extractor-czx2d"
Jul 28 17:24:02.177553 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:02.177074    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/66911cb6-549d-4f4f-8ac6-286401ad1025-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-czx2d\" (UID: \"66911cb6-549d-4f4f-8ac6-286401ad1025\") " pod="openshift-insights/insights-runtime-extractor-czx2d"
Jul 28 17:24:02.177553 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:02.177121    2543 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:24:02.177553 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:02.177190    2543 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:24:02.177553 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:02.177196    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66911cb6-549d-4f4f-8ac6-286401ad1025-kube-rbac-proxy-cm podName:66911cb6-549d-4f4f-8ac6-286401ad1025 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:04.177182202 +0000 UTC m=+47.583455921 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/66911cb6-549d-4f4f-8ac6-286401ad1025-kube-rbac-proxy-cm") pod "insights-runtime-extractor-czx2d" (UID: "66911cb6-549d-4f4f-8ac6-286401ad1025") : configmap "kube-rbac-proxy" not found
Jul 28 17:24:02.177553 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:02.177236    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66911cb6-549d-4f4f-8ac6-286401ad1025-insights-runtime-extractor-tls podName:66911cb6-549d-4f4f-8ac6-286401ad1025 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:04.177222625 +0000 UTC m=+47.583496345 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/66911cb6-549d-4f4f-8ac6-286401ad1025-insights-runtime-extractor-tls") pod "insights-runtime-extractor-czx2d" (UID: "66911cb6-549d-4f4f-8ac6-286401ad1025") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:24:02.378304 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:02.378281    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kvtkx_3acecb5c-18d4-4260-aa3e-8758184e6733/dns-node-resolver/0.log"
Jul 28 17:24:02.980853 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:02.980827    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4kw4j_ea4785ea-08ca-45f8-a8bc-42524b8d81b4/node-ca/0.log"
Jul 28 17:24:03.979148 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:03.979106    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9r84r_6a43fd4b-3d38-44b5-9b92-de7f6228df26/serve-healthcheck-canary/0.log"
Jul 28 17:24:04.191184 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:04.191154    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/66911cb6-549d-4f4f-8ac6-286401ad1025-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-czx2d\" (UID: \"66911cb6-549d-4f4f-8ac6-286401ad1025\") " pod="openshift-insights/insights-runtime-extractor-czx2d"
Jul 28 17:24:04.191324 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:04.191217    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/66911cb6-549d-4f4f-8ac6-286401ad1025-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-czx2d\" (UID: \"66911cb6-549d-4f4f-8ac6-286401ad1025\") " pod="openshift-insights/insights-runtime-extractor-czx2d"
Jul 28 17:24:04.191324 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:04.191296    2543 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:24:04.191428 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:04.191358    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66911cb6-549d-4f4f-8ac6-286401ad1025-insights-runtime-extractor-tls podName:66911cb6-549d-4f4f-8ac6-286401ad1025 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:08.19133814 +0000 UTC m=+51.597611859 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/66911cb6-549d-4f4f-8ac6-286401ad1025-insights-runtime-extractor-tls") pod "insights-runtime-extractor-czx2d" (UID: "66911cb6-549d-4f4f-8ac6-286401ad1025") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:24:04.191428 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:04.191296    2543 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:24:04.191515 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:04.191448    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66911cb6-549d-4f4f-8ac6-286401ad1025-kube-rbac-proxy-cm podName:66911cb6-549d-4f4f-8ac6-286401ad1025 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:08.191431071 +0000 UTC m=+51.597704801 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/66911cb6-549d-4f4f-8ac6-286401ad1025-kube-rbac-proxy-cm") pod "insights-runtime-extractor-czx2d" (UID: "66911cb6-549d-4f4f-8ac6-286401ad1025") : configmap "kube-rbac-proxy" not found
Jul 28 17:24:04.895542 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:04.895507    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/908c16e7-90b9-4f02-a950-e62eafb70d81-service-ca-bundle\") pod \"router-default-5657cc6765-srj79\" (UID: \"908c16e7-90b9-4f02-a950-e62eafb70d81\") " pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:24:04.895707 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:04.895686    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/908c16e7-90b9-4f02-a950-e62eafb70d81-metrics-certs\") pod \"router-default-5657cc6765-srj79\" (UID: \"908c16e7-90b9-4f02-a950-e62eafb70d81\") " pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:24:04.895748 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:04.895724    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/908c16e7-90b9-4f02-a950-e62eafb70d81-service-ca-bundle podName:908c16e7-90b9-4f02-a950-e62eafb70d81 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:12.895701251 +0000 UTC m=+56.301974973 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/908c16e7-90b9-4f02-a950-e62eafb70d81-service-ca-bundle") pod "router-default-5657cc6765-srj79" (UID: "908c16e7-90b9-4f02-a950-e62eafb70d81") : configmap references non-existent config key: service-ca.crt
Jul 28 17:24:04.895790 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:04.895780    2543 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:24:04.895841 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:04.895830    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/908c16e7-90b9-4f02-a950-e62eafb70d81-metrics-certs podName:908c16e7-90b9-4f02-a950-e62eafb70d81 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:12.895818803 +0000 UTC m=+56.302092536 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/908c16e7-90b9-4f02-a950-e62eafb70d81-metrics-certs") pod "router-default-5657cc6765-srj79" (UID: "908c16e7-90b9-4f02-a950-e62eafb70d81") : secret "router-metrics-certs-default" not found
Jul 28 17:24:06.865003 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:06.864971    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s"]
Jul 28 17:24:06.878047 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:06.878020    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s"]
Jul 28 17:24:06.878176 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:06.878146    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s"
Jul 28 17:24:06.880605 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:06.880582    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 28 17:24:06.880739 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:06.880635    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-s9k99\""
Jul 28 17:24:06.880739 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:06.880676    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 28 17:24:06.881338 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:06.881321    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:24:06.881424 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:06.881389    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 28 17:24:06.909239 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:06.909214    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6a0600d3-cc7b-4caf-a420-e669a2c09715-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s\" (UID: \"6a0600d3-cc7b-4caf-a420-e669a2c09715\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s"
Jul 28 17:24:06.909324 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:06.909244    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6a0600d3-cc7b-4caf-a420-e669a2c09715-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s\" (UID: \"6a0600d3-cc7b-4caf-a420-e669a2c09715\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s"
Jul 28 17:24:06.909324 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:06.909278    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwc45\" (UniqueName: \"kubernetes.io/projected/6a0600d3-cc7b-4caf-a420-e669a2c09715-kube-api-access-mwc45\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s\" (UID: \"6a0600d3-cc7b-4caf-a420-e669a2c09715\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s"
Jul 28 17:24:07.010588 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:07.010567    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6a0600d3-cc7b-4caf-a420-e669a2c09715-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s\" (UID: \"6a0600d3-cc7b-4caf-a420-e669a2c09715\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s"
Jul 28 17:24:07.010702 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:07.010595    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6a0600d3-cc7b-4caf-a420-e669a2c09715-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s\" (UID: \"6a0600d3-cc7b-4caf-a420-e669a2c09715\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s"
Jul 28 17:24:07.010702 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:07.010638    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwc45\" (UniqueName: \"kubernetes.io/projected/6a0600d3-cc7b-4caf-a420-e669a2c09715-kube-api-access-mwc45\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s\" (UID: \"6a0600d3-cc7b-4caf-a420-e669a2c09715\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s"
Jul 28 17:24:07.011090 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:07.011073    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6a0600d3-cc7b-4caf-a420-e669a2c09715-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s\" (UID: \"6a0600d3-cc7b-4caf-a420-e669a2c09715\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s"
Jul 28 17:24:07.012840 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:07.012819    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6a0600d3-cc7b-4caf-a420-e669a2c09715-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s\" (UID: \"6a0600d3-cc7b-4caf-a420-e669a2c09715\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s"
Jul 28 17:24:07.021416 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:07.021382    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwc45\" (UniqueName: \"kubernetes.io/projected/6a0600d3-cc7b-4caf-a420-e669a2c09715-kube-api-access-mwc45\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s\" (UID: \"6a0600d3-cc7b-4caf-a420-e669a2c09715\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s"
Jul 28 17:24:07.187372 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:07.187318    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s"
Jul 28 17:24:07.322906 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:07.322859    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s"]
Jul 28 17:24:07.326625 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:24:07.326595    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0600d3_cc7b_4caf_a420_e669a2c09715.slice/crio-673c1e8422b58a6df357af980b3b19ddafe6f5d419daf3e90d26b55517bde422 WatchSource:0}: Error finding container 673c1e8422b58a6df357af980b3b19ddafe6f5d419daf3e90d26b55517bde422: Status 404 returned error can't find the container with id 673c1e8422b58a6df357af980b3b19ddafe6f5d419daf3e90d26b55517bde422
Jul 28 17:24:07.430867 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:07.430837    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s" event={"ID":"6a0600d3-cc7b-4caf-a420-e669a2c09715","Type":"ContainerStarted","Data":"673c1e8422b58a6df357af980b3b19ddafe6f5d419daf3e90d26b55517bde422"}
Jul 28 17:24:08.219660 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:08.219604    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/66911cb6-549d-4f4f-8ac6-286401ad1025-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-czx2d\" (UID: \"66911cb6-549d-4f4f-8ac6-286401ad1025\") " pod="openshift-insights/insights-runtime-extractor-czx2d"
Jul 28 17:24:08.220126 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:08.219753    2543 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:24:08.220126 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:08.219809    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/66911cb6-549d-4f4f-8ac6-286401ad1025-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-czx2d\" (UID: \"66911cb6-549d-4f4f-8ac6-286401ad1025\") " pod="openshift-insights/insights-runtime-extractor-czx2d"
Jul 28 17:24:08.220126 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:08.219834    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66911cb6-549d-4f4f-8ac6-286401ad1025-kube-rbac-proxy-cm podName:66911cb6-549d-4f4f-8ac6-286401ad1025 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:16.219811615 +0000 UTC m=+59.626085334 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/66911cb6-549d-4f4f-8ac6-286401ad1025-kube-rbac-proxy-cm") pod "insights-runtime-extractor-czx2d" (UID: "66911cb6-549d-4f4f-8ac6-286401ad1025") : configmap "kube-rbac-proxy" not found
Jul 28 17:24:08.220126 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:08.219921    2543 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:24:08.220126 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:08.219985    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66911cb6-549d-4f4f-8ac6-286401ad1025-insights-runtime-extractor-tls podName:66911cb6-549d-4f4f-8ac6-286401ad1025 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:16.219969575 +0000 UTC m=+59.626243308 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/66911cb6-549d-4f4f-8ac6-286401ad1025-insights-runtime-extractor-tls") pod "insights-runtime-extractor-czx2d" (UID: "66911cb6-549d-4f4f-8ac6-286401ad1025") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:24:08.420838 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:08.420802    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3af720f1-3267-48f6-94e2-6f6527c0fcf6-metrics-tls\") pod \"dns-default-hg8ms\" (UID: \"3af720f1-3267-48f6-94e2-6f6527c0fcf6\") " pod="openshift-dns/dns-default-hg8ms"
Jul 28 17:24:08.421015 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:08.420976    2543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:24:08.421072 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:08.421062    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af720f1-3267-48f6-94e2-6f6527c0fcf6-metrics-tls podName:3af720f1-3267-48f6-94e2-6f6527c0fcf6 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:24.421037601 +0000 UTC m=+67.827311345 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3af720f1-3267-48f6-94e2-6f6527c0fcf6-metrics-tls") pod "dns-default-hg8ms" (UID: "3af720f1-3267-48f6-94e2-6f6527c0fcf6") : secret "dns-default-metrics-tls" not found
Jul 28 17:24:10.439311 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:10.439281    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s" event={"ID":"6a0600d3-cc7b-4caf-a420-e669a2c09715","Type":"ContainerStarted","Data":"d69676a8ccc2d6278cc9d7ff67da37c002053689f69106057312485c87932200"}
Jul 28 17:24:10.456123 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:10.456077    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s" podStartSLOduration=1.989563513 podStartE2EDuration="4.456062507s" totalImagesPullingTime="2.466498994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:10.455585343 +0000 UTC m=+53.861859086" watchObservedRunningTime="2026-07-28 17:24:10.456062507 +0000 UTC m=+53.862336249"
Jul 28 17:24:12.367292 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:12.367261    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-zqrks"]
Jul 28 17:24:12.389657 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:12.389632    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-zqrks"]
Jul 28 17:24:12.389791 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:12.389743    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zqrks"
Jul 28 17:24:12.393048 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:12.393026    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 28 17:24:12.393884 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:12.393868    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-dt977\""
Jul 28 17:24:12.393967 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:12.393924    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 28 17:24:12.453432 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:12.453406    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4knl\" (UniqueName: \"kubernetes.io/projected/ef7e5979-918f-49c1-9724-7c736713e0ae-kube-api-access-l4knl\") pod \"migrator-6865c9967-zqrks\" (UID: \"ef7e5979-918f-49c1-9724-7c736713e0ae\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zqrks"
Jul 28 17:24:12.554778 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:12.554746    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4knl\" (UniqueName: \"kubernetes.io/projected/ef7e5979-918f-49c1-9724-7c736713e0ae-kube-api-access-l4knl\") pod \"migrator-6865c9967-zqrks\" (UID: \"ef7e5979-918f-49c1-9724-7c736713e0ae\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zqrks"
Jul 28 17:24:12.562929 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:12.562903    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4knl\" (UniqueName: \"kubernetes.io/projected/ef7e5979-918f-49c1-9724-7c736713e0ae-kube-api-access-l4knl\") pod \"migrator-6865c9967-zqrks\" (UID: \"ef7e5979-918f-49c1-9724-7c736713e0ae\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zqrks"
Jul 28 17:24:12.699267 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:12.699201    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zqrks"
Jul 28 17:24:12.843531 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:12.843502    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-zqrks"]
Jul 28 17:24:12.846937 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:24:12.846915    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef7e5979_918f_49c1_9724_7c736713e0ae.slice/crio-13beb20a36802c010f3be03df33f7849a802b00992a1d836cede78f39441e292 WatchSource:0}: Error finding container 13beb20a36802c010f3be03df33f7849a802b00992a1d836cede78f39441e292: Status 404 returned error can't find the container with id 13beb20a36802c010f3be03df33f7849a802b00992a1d836cede78f39441e292
Jul 28 17:24:12.958235 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:12.958150    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/908c16e7-90b9-4f02-a950-e62eafb70d81-service-ca-bundle\") pod \"router-default-5657cc6765-srj79\" (UID: \"908c16e7-90b9-4f02-a950-e62eafb70d81\") " pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:24:12.958400 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:12.958289    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/908c16e7-90b9-4f02-a950-e62eafb70d81-metrics-certs\") pod \"router-default-5657cc6765-srj79\" (UID: \"908c16e7-90b9-4f02-a950-e62eafb70d81\") " pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:24:12.958400 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:12.958303    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/908c16e7-90b9-4f02-a950-e62eafb70d81-service-ca-bundle podName:908c16e7-90b9-4f02-a950-e62eafb70d81 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:28.958288064 +0000 UTC m=+72.364561782 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/908c16e7-90b9-4f02-a950-e62eafb70d81-service-ca-bundle") pod "router-default-5657cc6765-srj79" (UID: "908c16e7-90b9-4f02-a950-e62eafb70d81") : configmap references non-existent config key: service-ca.crt
Jul 28 17:24:12.958483 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:12.958430    2543 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:24:12.958483 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:12.958481    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/908c16e7-90b9-4f02-a950-e62eafb70d81-metrics-certs podName:908c16e7-90b9-4f02-a950-e62eafb70d81 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:28.95846923 +0000 UTC m=+72.364742949 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/908c16e7-90b9-4f02-a950-e62eafb70d81-metrics-certs") pod "router-default-5657cc6765-srj79" (UID: "908c16e7-90b9-4f02-a950-e62eafb70d81") : secret "router-metrics-certs-default" not found
Jul 28 17:24:13.414234 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:13.414200    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-7kqkc"]
Jul 28 17:24:13.426820 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:13.426774    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-7kqkc"]
Jul 28 17:24:13.427011 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:13.426894    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-7kqkc"
Jul 28 17:24:13.429476 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:13.429446    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 28 17:24:13.430491 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:13.430472    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 28 17:24:13.430725 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:13.430456    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 28 17:24:13.430801 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:13.430501    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 28 17:24:13.430801 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:13.430470    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-rvbbr\""
Jul 28 17:24:13.448841 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:13.448814    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zqrks" event={"ID":"ef7e5979-918f-49c1-9724-7c736713e0ae","Type":"ContainerStarted","Data":"13beb20a36802c010f3be03df33f7849a802b00992a1d836cede78f39441e292"}
Jul 28 17:24:13.462233 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:13.462209    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8ff4f104-4836-4c8a-8a29-cb3a76ea1ac6-signing-cabundle\") pod \"service-ca-658d7b7c78-7kqkc\" (UID: \"8ff4f104-4836-4c8a-8a29-cb3a76ea1ac6\") " pod="openshift-service-ca/service-ca-658d7b7c78-7kqkc"
Jul 28 17:24:13.462349 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:13.462255    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqzj8\" (UniqueName: \"kubernetes.io/projected/8ff4f104-4836-4c8a-8a29-cb3a76ea1ac6-kube-api-access-hqzj8\") pod \"service-ca-658d7b7c78-7kqkc\" (UID: \"8ff4f104-4836-4c8a-8a29-cb3a76ea1ac6\") " pod="openshift-service-ca/service-ca-658d7b7c78-7kqkc"
Jul 28 17:24:13.462349 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:13.462332    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8ff4f104-4836-4c8a-8a29-cb3a76ea1ac6-signing-key\") pod \"service-ca-658d7b7c78-7kqkc\" (UID: \"8ff4f104-4836-4c8a-8a29-cb3a76ea1ac6\") " pod="openshift-service-ca/service-ca-658d7b7c78-7kqkc"
Jul 28 17:24:13.563167 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:13.563122    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8ff4f104-4836-4c8a-8a29-cb3a76ea1ac6-signing-key\") pod \"service-ca-658d7b7c78-7kqkc\" (UID: \"8ff4f104-4836-4c8a-8a29-cb3a76ea1ac6\") " pod="openshift-service-ca/service-ca-658d7b7c78-7kqkc"
Jul 28 17:24:13.563337 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:13.563218    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8ff4f104-4836-4c8a-8a29-cb3a76ea1ac6-signing-cabundle\") pod \"service-ca-658d7b7c78-7kqkc\" (UID: \"8ff4f104-4836-4c8a-8a29-cb3a76ea1ac6\") " pod="openshift-service-ca/service-ca-658d7b7c78-7kqkc"
Jul 28 17:24:13.563337 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:13.563268    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqzj8\" (UniqueName: \"kubernetes.io/projected/8ff4f104-4836-4c8a-8a29-cb3a76ea1ac6-kube-api-access-hqzj8\") pod \"service-ca-658d7b7c78-7kqkc\" (UID: \"8ff4f104-4836-4c8a-8a29-cb3a76ea1ac6\") " pod="openshift-service-ca/service-ca-658d7b7c78-7kqkc"
Jul 28 17:24:13.563858 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:13.563833    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8ff4f104-4836-4c8a-8a29-cb3a76ea1ac6-signing-cabundle\") pod \"service-ca-658d7b7c78-7kqkc\" (UID: \"8ff4f104-4836-4c8a-8a29-cb3a76ea1ac6\") " pod="openshift-service-ca/service-ca-658d7b7c78-7kqkc"
Jul 28 17:24:13.565573 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:13.565541    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8ff4f104-4836-4c8a-8a29-cb3a76ea1ac6-signing-key\") pod \"service-ca-658d7b7c78-7kqkc\" (UID: \"8ff4f104-4836-4c8a-8a29-cb3a76ea1ac6\") " pod="openshift-service-ca/service-ca-658d7b7c78-7kqkc"
Jul 28 17:24:13.572958 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:13.572925    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqzj8\" (UniqueName: \"kubernetes.io/projected/8ff4f104-4836-4c8a-8a29-cb3a76ea1ac6-kube-api-access-hqzj8\") pod \"service-ca-658d7b7c78-7kqkc\" (UID: \"8ff4f104-4836-4c8a-8a29-cb3a76ea1ac6\") " pod="openshift-service-ca/service-ca-658d7b7c78-7kqkc"
Jul 28 17:24:13.737730 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:13.737704    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-7kqkc"
Jul 28 17:24:14.162461 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:14.162422    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-7kqkc"]
Jul 28 17:24:14.165061 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:24:14.165033    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff4f104_4836_4c8a_8a29_cb3a76ea1ac6.slice/crio-d975041488003da1bc7951c033e73936f397211f2d77e60bb95456d3970598b3 WatchSource:0}: Error finding container d975041488003da1bc7951c033e73936f397211f2d77e60bb95456d3970598b3: Status 404 returned error can't find the container with id d975041488003da1bc7951c033e73936f397211f2d77e60bb95456d3970598b3
Jul 28 17:24:14.453083 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:14.453055    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-7kqkc" event={"ID":"8ff4f104-4836-4c8a-8a29-cb3a76ea1ac6","Type":"ContainerStarted","Data":"d975041488003da1bc7951c033e73936f397211f2d77e60bb95456d3970598b3"}
Jul 28 17:24:15.457547 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:15.457515    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zqrks" event={"ID":"ef7e5979-918f-49c1-9724-7c736713e0ae","Type":"ContainerStarted","Data":"156eb940b2a12af8d442f66fb372c4e686a55d458414b59cd4bbead3c56e02eb"}
Jul 28 17:24:15.457927 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:15.457565    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zqrks" event={"ID":"ef7e5979-918f-49c1-9724-7c736713e0ae","Type":"ContainerStarted","Data":"4426bda8c49ee14d973d63853e7809dce77d1843e1aad56f4244346ba4619188"}
Jul 28 17:24:16.285969 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:16.285938    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/66911cb6-549d-4f4f-8ac6-286401ad1025-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-czx2d\" (UID: \"66911cb6-549d-4f4f-8ac6-286401ad1025\") " pod="openshift-insights/insights-runtime-extractor-czx2d"
Jul 28 17:24:16.286122 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:16.286007    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/66911cb6-549d-4f4f-8ac6-286401ad1025-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-czx2d\" (UID: \"66911cb6-549d-4f4f-8ac6-286401ad1025\") " pod="openshift-insights/insights-runtime-extractor-czx2d"
Jul 28 17:24:16.286122 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:16.286095    2543 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:24:16.286216 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:16.286182    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66911cb6-549d-4f4f-8ac6-286401ad1025-insights-runtime-extractor-tls podName:66911cb6-549d-4f4f-8ac6-286401ad1025 nodeName:}" failed. No retries permitted until 2026-07-28 17:24:32.286166624 +0000 UTC m=+75.692440343 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/66911cb6-549d-4f4f-8ac6-286401ad1025-insights-runtime-extractor-tls") pod "insights-runtime-extractor-czx2d" (UID: "66911cb6-549d-4f4f-8ac6-286401ad1025") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:24:16.286508 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:16.286490    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/66911cb6-549d-4f4f-8ac6-286401ad1025-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-czx2d\" (UID: \"66911cb6-549d-4f4f-8ac6-286401ad1025\") " pod="openshift-insights/insights-runtime-extractor-czx2d"
Jul 28 17:24:16.461823 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:16.461747    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-7kqkc" event={"ID":"8ff4f104-4836-4c8a-8a29-cb3a76ea1ac6","Type":"ContainerStarted","Data":"85a721905aa8ca2abf7fb85f72b1d90e9fbc0238e693aa796a06ee3887eb43c4"}
Jul 28 17:24:16.476490 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:16.476445    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zqrks" podStartSLOduration=2.934289849 podStartE2EDuration="4.476429139s" totalImagesPullingTime="1.54213929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:15.476458281 +0000 UTC m=+58.882732031" watchObservedRunningTime="2026-07-28 17:24:16.476429139 +0000 UTC m=+59.882702883"
Jul 28 17:24:16.477172 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:16.477116    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-7kqkc" podStartSLOduration=1.567407899 podStartE2EDuration="3.477107783s" totalImagesPullingTime="1.909699884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:16.476567856 +0000 UTC m=+59.882841588" watchObservedRunningTime="2026-07-28 17:24:16.477107783 +0000 UTC m=+59.883381523"
Jul 28 17:24:17.379524 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:17.379489    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-64lk7"
Jul 28 17:24:21.832539 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:21.832498    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c1c169-0fd2-4e73-a497-0b69fe85d11f-metrics-certs\") pod \"network-metrics-daemon-sjtrc\" (UID: \"a2c1c169-0fd2-4e73-a497-0b69fe85d11f\") " pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:24:21.833043 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:21.832564    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/145627c2-9b95-4251-9f16-2920555de2fa-original-pull-secret\") pod \"global-pull-secret-syncer-jtb46\" (UID: \"145627c2-9b95-4251-9f16-2920555de2fa\") " pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:24:21.834986 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:21.834963    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:24:21.835057 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:21.834973    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:24:21.843045 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:21.843024    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 17:24:21.843114 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:21.843092    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2c1c169-0fd2-4e73-a497-0b69fe85d11f-metrics-certs podName:a2c1c169-0fd2-4e73-a497-0b69fe85d11f nodeName:}" failed. No retries permitted until 2026-07-28 17:25:25.843070282 +0000 UTC m=+129.249344015 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2c1c169-0fd2-4e73-a497-0b69fe85d11f-metrics-certs") pod "network-metrics-daemon-sjtrc" (UID: "a2c1c169-0fd2-4e73-a497-0b69fe85d11f") : secret "metrics-daemon-secret" not found
Jul 28 17:24:21.845119 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:21.845097    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/145627c2-9b95-4251-9f16-2920555de2fa-original-pull-secret\") pod \"global-pull-secret-syncer-jtb46\" (UID: \"145627c2-9b95-4251-9f16-2920555de2fa\") " pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:24:21.933738 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:21.933702    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w89v\" (UniqueName: \"kubernetes.io/projected/2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4-kube-api-access-7w89v\") pod \"network-check-target-9c8w8\" (UID: \"2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4\") " pod="openshift-network-diagnostics/network-check-target-9c8w8"
Jul 28 17:24:21.936314 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:21.936295    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:24:21.946793 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:21.946777    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:24:21.957200 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:21.957179    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w89v\" (UniqueName: \"kubernetes.io/projected/2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4-kube-api-access-7w89v\") pod \"network-check-target-9c8w8\" (UID: \"2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4\") " pod="openshift-network-diagnostics/network-check-target-9c8w8"
Jul 28 17:24:22.057002 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:22.056975    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtb46"
Jul 28 17:24:22.186943 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:22.186911    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jtb46"]
Jul 28 17:24:22.189885 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:24:22.189862    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145627c2_9b95_4251_9f16_2920555de2fa.slice/crio-069a1791a2e84ae5972f297a469476909c7fce6e8f695fdbcb08406c29bf8800 WatchSource:0}: Error finding container 069a1791a2e84ae5972f297a469476909c7fce6e8f695fdbcb08406c29bf8800: Status 404 returned error can't find the container with id 069a1791a2e84ae5972f297a469476909c7fce6e8f695fdbcb08406c29bf8800
Jul 28 17:24:22.258515 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:22.258492    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rqpml\""
Jul 28 17:24:22.266530 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:22.266513    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9c8w8"
Jul 28 17:24:22.389315 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:22.389242    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9c8w8"]
Jul 28 17:24:22.392921 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:24:22.392889    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7978b1_3a5e_4d89_9f4e_ed72f6ae67d4.slice/crio-69a6f86adb83f924ad9136f673ef3883c201b9a9e2ff02e8098f411dab107066 WatchSource:0}: Error finding container 69a6f86adb83f924ad9136f673ef3883c201b9a9e2ff02e8098f411dab107066: Status 404 returned error can't find the container with id 69a6f86adb83f924ad9136f673ef3883c201b9a9e2ff02e8098f411dab107066
Jul 28 17:24:22.481660 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:22.481623    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9c8w8" event={"ID":"2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4","Type":"ContainerStarted","Data":"69a6f86adb83f924ad9136f673ef3883c201b9a9e2ff02e8098f411dab107066"}
Jul 28 17:24:22.482847 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:22.482824    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jtb46" event={"ID":"145627c2-9b95-4251-9f16-2920555de2fa","Type":"ContainerStarted","Data":"069a1791a2e84ae5972f297a469476909c7fce6e8f695fdbcb08406c29bf8800"}
Jul 28 17:24:24.456079 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:24.456045    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3af720f1-3267-48f6-94e2-6f6527c0fcf6-metrics-tls\") pod \"dns-default-hg8ms\" (UID: \"3af720f1-3267-48f6-94e2-6f6527c0fcf6\") " pod="openshift-dns/dns-default-hg8ms"
Jul 28 17:24:24.459101 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:24.459045    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3af720f1-3267-48f6-94e2-6f6527c0fcf6-metrics-tls\") pod \"dns-default-hg8ms\" (UID: \"3af720f1-3267-48f6-94e2-6f6527c0fcf6\") " pod="openshift-dns/dns-default-hg8ms"
Jul 28 17:24:24.631393 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:24.631361    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gw5z2\""
Jul 28 17:24:24.639762 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:24.639509    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hg8ms"
Jul 28 17:24:27.425729 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:27.425702    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hg8ms"]
Jul 28 17:24:27.429452 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:24:27.429415    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af720f1_3267_48f6_94e2_6f6527c0fcf6.slice/crio-c8e91bfd6dc05a666556c3a980320646578f44c0492e2bb5bcea724566d8ee8d WatchSource:0}: Error finding container c8e91bfd6dc05a666556c3a980320646578f44c0492e2bb5bcea724566d8ee8d: Status 404 returned error can't find the container with id c8e91bfd6dc05a666556c3a980320646578f44c0492e2bb5bcea724566d8ee8d
Jul 28 17:24:27.500765 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:27.500733    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9c8w8" event={"ID":"2d7978b1-3a5e-4d89-9f4e-ed72f6ae67d4","Type":"ContainerStarted","Data":"2fd4ef08132abc12122583d8c2a0b4d1f5f2033dbb07b289ff25a2f9f107fb32"}
Jul 28 17:24:27.500897 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:27.500852    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9c8w8"
Jul 28 17:24:27.502278 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:27.502256    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jtb46" event={"ID":"145627c2-9b95-4251-9f16-2920555de2fa","Type":"ContainerStarted","Data":"6468fd58eaf8f5b97a655ec5a02c86b85842b16371d560c754433c2d83550221"}
Jul 28 17:24:27.503410 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:27.503387    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hg8ms" event={"ID":"3af720f1-3267-48f6-94e2-6f6527c0fcf6","Type":"ContainerStarted","Data":"c8e91bfd6dc05a666556c3a980320646578f44c0492e2bb5bcea724566d8ee8d"}
Jul 28 17:24:27.530484 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:27.530439    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9c8w8" podStartSLOduration=65.623146186 podStartE2EDuration="1m10.530423824s" totalImagesPullingTime="4.907277638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:27.529565076 +0000 UTC m=+70.935838817" watchObservedRunningTime="2026-07-28 17:24:27.530423824 +0000 UTC m=+70.936697564"
Jul 28 17:24:27.547326 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:27.547243    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jtb46" podStartSLOduration=65.424742815 podStartE2EDuration="1m10.547228877s" totalImagesPullingTime="5.122486062s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:27.545652182 +0000 UTC m=+70.951925925" watchObservedRunningTime="2026-07-28 17:24:27.547228877 +0000 UTC m=+70.953502617"
Jul 28 17:24:28.994046 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:28.994019    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/908c16e7-90b9-4f02-a950-e62eafb70d81-metrics-certs\") pod \"router-default-5657cc6765-srj79\" (UID: \"908c16e7-90b9-4f02-a950-e62eafb70d81\") " pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:24:28.994498 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:28.994075    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/908c16e7-90b9-4f02-a950-e62eafb70d81-service-ca-bundle\") pod \"router-default-5657cc6765-srj79\" (UID: \"908c16e7-90b9-4f02-a950-e62eafb70d81\") " pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:24:28.994642 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:28.994622    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/908c16e7-90b9-4f02-a950-e62eafb70d81-service-ca-bundle\") pod \"router-default-5657cc6765-srj79\" (UID: \"908c16e7-90b9-4f02-a950-e62eafb70d81\") " pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:24:28.996476 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:28.996450    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/908c16e7-90b9-4f02-a950-e62eafb70d81-metrics-certs\") pod \"router-default-5657cc6765-srj79\" (UID: \"908c16e7-90b9-4f02-a950-e62eafb70d81\") " pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:24:29.278386 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:29.278355    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-5fcgw\""
Jul 28 17:24:29.286119 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:29.286096    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:24:29.429442 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:29.429414    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5657cc6765-srj79"]
Jul 28 17:24:29.430221 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:24:29.430187    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908c16e7_90b9_4f02_a950_e62eafb70d81.slice/crio-51520da2694b4b52379005c9627799e1f223f4368cdcf1b5f99d96276a5560df WatchSource:0}: Error finding container 51520da2694b4b52379005c9627799e1f223f4368cdcf1b5f99d96276a5560df: Status 404 returned error can't find the container with id 51520da2694b4b52379005c9627799e1f223f4368cdcf1b5f99d96276a5560df
Jul 28 17:24:29.511142 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:29.511098    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5657cc6765-srj79" event={"ID":"908c16e7-90b9-4f02-a950-e62eafb70d81","Type":"ContainerStarted","Data":"fd6727eec5892a3a24ded660502dadc1f50d1aafcf536587affb26e10f434d83"}
Jul 28 17:24:29.511272 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:29.511168    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5657cc6765-srj79" event={"ID":"908c16e7-90b9-4f02-a950-e62eafb70d81","Type":"ContainerStarted","Data":"51520da2694b4b52379005c9627799e1f223f4368cdcf1b5f99d96276a5560df"}
Jul 28 17:24:29.512804 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:29.512779    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hg8ms" event={"ID":"3af720f1-3267-48f6-94e2-6f6527c0fcf6","Type":"ContainerStarted","Data":"5af424be10c18c96db22e7dc98177eaca1af679a049e8cb6c69115c4d71e5323"}
Jul 28 17:24:29.512896 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:29.512828    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hg8ms" event={"ID":"3af720f1-3267-48f6-94e2-6f6527c0fcf6","Type":"ContainerStarted","Data":"7d32bd15a8cd58a7d78da934cd7c74fd1fe68ba3f2656b456658b9eb6980a4ae"}
Jul 28 17:24:29.512896 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:29.512873    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hg8ms"
Jul 28 17:24:29.533965 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:29.533868    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5657cc6765-srj79" podStartSLOduration=32.533850252 podStartE2EDuration="32.533850252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:29.53341105 +0000 UTC m=+72.939684792" watchObservedRunningTime="2026-07-28 17:24:29.533850252 +0000 UTC m=+72.940123996"
Jul 28 17:24:29.555615 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:29.555573    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hg8ms" podStartSLOduration=36.033527252 podStartE2EDuration="37.555560139s" totalImagesPullingTime="1.522032887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:29.555395064 +0000 UTC m=+72.961668805" watchObservedRunningTime="2026-07-28 17:24:29.555560139 +0000 UTC m=+72.961833880"
Jul 28 17:24:30.286777 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:30.286742    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:24:30.289411 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:30.289388    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:24:30.516415 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:30.516385    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:24:30.517441 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:30.517423    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5657cc6765-srj79"
Jul 28 17:24:32.323148 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:32.323102    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/66911cb6-549d-4f4f-8ac6-286401ad1025-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-czx2d\" (UID: \"66911cb6-549d-4f4f-8ac6-286401ad1025\") " pod="openshift-insights/insights-runtime-extractor-czx2d"
Jul 28 17:24:32.325328 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:32.325308    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/66911cb6-549d-4f4f-8ac6-286401ad1025-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-czx2d\" (UID: \"66911cb6-549d-4f4f-8ac6-286401ad1025\") " pod="openshift-insights/insights-runtime-extractor-czx2d"
Jul 28 17:24:32.573000 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:32.572970    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b7f8h\""
Jul 28 17:24:32.580825 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:32.580772    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-czx2d"
Jul 28 17:24:32.707890 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:32.707861    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-czx2d"]
Jul 28 17:24:32.711164 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:24:32.711128    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66911cb6_549d_4f4f_8ac6_286401ad1025.slice/crio-38ac6791d8852ef6e40c41b45c3e5ac158d3bf24acb419f5cb0e3497fe306a7a WatchSource:0}: Error finding container 38ac6791d8852ef6e40c41b45c3e5ac158d3bf24acb419f5cb0e3497fe306a7a: Status 404 returned error can't find the container with id 38ac6791d8852ef6e40c41b45c3e5ac158d3bf24acb419f5cb0e3497fe306a7a
Jul 28 17:24:33.529147 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:33.529093    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-czx2d" event={"ID":"66911cb6-549d-4f4f-8ac6-286401ad1025","Type":"ContainerStarted","Data":"38ac6791d8852ef6e40c41b45c3e5ac158d3bf24acb419f5cb0e3497fe306a7a"}
Jul 28 17:24:36.087701 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.087667    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-zkptp"]
Jul 28 17:24:36.101255 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.101226    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64cb7bf944-dh52l"]
Jul 28 17:24:36.101423 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.101337    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-zkptp"
Jul 28 17:24:36.103550 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.103463    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 28 17:24:36.103987 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.103964    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64cb7bf944-dh52l"
Jul 28 17:24:36.104917 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.104633    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-fkkk9\""
Jul 28 17:24:36.104917 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.104869    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-zkptp"]
Jul 28 17:24:36.105944 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.105915    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 17:24:36.108088 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.108069    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-frkgx\""
Jul 28 17:24:36.108463 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.108428    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 17:24:36.108992 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.108843    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 17:24:36.109105 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.109037    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 17:24:36.109105 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.109071    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 17:24:36.109257 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.109186    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 17:24:36.109332 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.109306    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 17:24:36.114609 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.114553    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64cb7bf944-dh52l"]
Jul 28 17:24:36.143528 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.143502    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b7ff5c8fb-j995z"]
Jul 28 17:24:36.146558 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.146538    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b7ff5c8fb-j995z"
Jul 28 17:24:36.152544 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.152523    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-d8jtm\""
Jul 28 17:24:36.152661 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.152588    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 17:24:36.154797 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.154776    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 17:24:36.164599 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.164579    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 17:24:36.192031 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.192008    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b7ff5c8fb-j995z"]
Jul 28 17:24:36.210146 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.210109    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 17:24:36.255194 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.255166    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1980fe9a-8463-4a8d-8462-d591761ffdf1-service-ca\") pod \"console-64cb7bf944-dh52l\" (UID: \"1980fe9a-8463-4a8d-8462-d591761ffdf1\") " pod="openshift-console/console-64cb7bf944-dh52l"
Jul 28 17:24:36.255371 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.255206    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/949ffdd5-088b-4f73-b4d7-41921e724eab-installation-pull-secrets\") pod \"image-registry-5b7ff5c8fb-j995z\" (UID: \"949ffdd5-088b-4f73-b4d7-41921e724eab\") " pod="openshift-image-registry/image-registry-5b7ff5c8fb-j995z"
Jul 28 17:24:36.255371 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.255295    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1980fe9a-8463-4a8d-8462-d591761ffdf1-console-oauth-config\") pod \"console-64cb7bf944-dh52l\" (UID: \"1980fe9a-8463-4a8d-8462-d591761ffdf1\") " pod="openshift-console/console-64cb7bf944-dh52l"
Jul 28 17:24:36.255371 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.255327    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1980fe9a-8463-4a8d-8462-d591761ffdf1-oauth-serving-cert\") pod \"console-64cb7bf944-dh52l\" (UID: \"1980fe9a-8463-4a8d-8462-d591761ffdf1\") " pod="openshift-console/console-64cb7bf944-dh52l"
Jul 28 17:24:36.255371 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.255349    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6fq6\" (UniqueName: \"kubernetes.io/projected/1980fe9a-8463-4a8d-8462-d591761ffdf1-kube-api-access-f6fq6\") pod \"console-64cb7bf944-dh52l\" (UID: \"1980fe9a-8463-4a8d-8462-d591761ffdf1\") " pod="openshift-console/console-64cb7bf944-dh52l"
Jul 28 17:24:36.255566 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.255383    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/949ffdd5-088b-4f73-b4d7-41921e724eab-ca-trust-extracted\") pod \"image-registry-5b7ff5c8fb-j995z\" (UID: \"949ffdd5-088b-4f73-b4d7-41921e724eab\") " pod="openshift-image-registry/image-registry-5b7ff5c8fb-j995z"
Jul 28 17:24:36.255566 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.255468    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/949ffdd5-088b-4f73-b4d7-41921e724eab-image-registry-private-configuration\") pod \"image-registry-5b7ff5c8fb-j995z\" (UID: \"949ffdd5-088b-4f73-b4d7-41921e724eab\") " pod="openshift-image-registry/image-registry-5b7ff5c8fb-j995z"
Jul 28 17:24:36.255566 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.255496    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1980fe9a-8463-4a8d-8462-d591761ffdf1-console-serving-cert\") pod \"console-64cb7bf944-dh52l\" (UID: \"1980fe9a-8463-4a8d-8462-d591761ffdf1\") " pod="openshift-console/console-64cb7bf944-dh52l"
Jul 28 17:24:36.255566 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.255515    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1980fe9a-8463-4a8d-8462-d591761ffdf1-console-config\") pod \"console-64cb7bf944-dh52l\" (UID: \"1980fe9a-8463-4a8d-8462-d591761ffdf1\") " pod="openshift-console/console-64cb7bf944-dh52l"
Jul 28 17:24:36.255566 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.255530    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0231f8b7-62f6-4677-9ff7-a35efb83743d-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-zkptp\" (UID: \"0231f8b7-62f6-4677-9ff7-a35efb83743d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-zkptp"
Jul 28 17:24:36.255785 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.255564    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/949ffdd5-088b-4f73-b4d7-41921e724eab-trusted-ca\") pod \"image-registry-5b7ff5c8fb-j995z\" (UID: \"949ffdd5-088b-4f73-b4d7-41921e724eab\") " pod="openshift-image-registry/image-registry-5b7ff5c8fb-j995z"
Jul 28 17:24:36.255785 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.255637    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/949ffdd5-088b-4f73-b4d7-41921e724eab-registry-certificates\") pod \"image-registry-5b7ff5c8fb-j995z\" (UID: \"949ffdd5-088b-4f73-b4d7-41921e724eab\") " pod="openshift-image-registry/image-registry-5b7ff5c8fb-j995z"
Jul 28 17:24:36.255785 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.255667    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/949ffdd5-088b-4f73-b4d7-41921e724eab-bound-sa-token\") pod \"image-registry-5b7ff5c8fb-j995z\" (UID: \"949ffdd5-088b-4f73-b4d7-41921e724eab\") " pod="openshift-image-registry/image-registry-5b7ff5c8fb-j995z"
Jul 28 17:24:36.255785 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.255706    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/949ffdd5-088b-4f73-b4d7-41921e724eab-registry-tls\") pod \"image-registry-5b7ff5c8fb-j995z\" (UID: \"949ffdd5-088b-4f73-b4d7-41921e724eab\") " pod="openshift-image-registry/image-registry-5b7ff5c8fb-j995z"
Jul 28 17:24:36.255785 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.255731    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzltq\" (UniqueName: \"kubernetes.io/projected/949ffdd5-088b-4f73-b4d7-41921e724eab-kube-api-access-rzltq\") pod \"image-registry-5b7ff5c8fb-j995z\" (UID: \"949ffdd5-088b-4f73-b4d7-41921e724eab\") " pod="openshift-image-registry/image-registry-5b7ff5c8fb-j995z"
Jul 28 17:24:36.356359 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.356279    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1980fe9a-8463-4a8d-8462-d591761ffdf1-console-serving-cert\") pod \"console-64cb7bf944-dh52l\" (UID: \"1980fe9a-8463-4a8d-8462-d591761ffdf1\") " pod="openshift-console/console-64cb7bf944-dh52l"
Jul 28 17:24:36.356359 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.356328    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1980fe9a-8463-4a8d-8462-d591761ffdf1-console-config\") pod \"console-64cb7bf944-dh52l\" (UID: \"1980fe9a-8463-4a8d-8462-d591761ffdf1\") " pod="openshift-console/console-64cb7bf944-dh52l"
Jul 28 17:24:36.356359 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.356348    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0231f8b7-62f6-4677-9ff7-a35efb83743d-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-zkptp\" (UID: \"0231f8b7-62f6-4677-9ff7-a35efb83743d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-zkptp"
Jul 28 17:24:36.356359 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.356368    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/949ffdd5-088b-4f73-b4d7-41921e724eab-trusted-ca\") pod \"image-registry-5b7ff5c8fb-j995z\" (UID: \"949ffdd5-088b-4f73-b4d7-41921e724eab\") " pod="openshift-image-registry/image-registry-5b7ff5c8fb-j995z"
Jul 28 17:24:36.356639 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.356387    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/949ffdd5-088b-4f73-b4d7-41921e724eab-registry-certificates\") pod \"image-registry-5b7ff5c8fb-j995z\" (UID: \"949ffdd5-088b-4f73-b4d7-41921e724eab\") " pod="openshift-image-registry/image-registry-5b7ff5c8fb-j995z"
Jul 28 17:24:36.356639 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.356419    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/949ffdd5-088b-4f73-b4d7-41921e724eab-bound-sa-token\") pod \"image-registry-5b7ff5c8fb-j995z\" (UID: \"949ffdd5-088b-4f73-b4d7-41921e724eab\") " pod="openshift-image-registry/image-registry-5b7ff5c8fb-j995z"
Jul 28 17:24:36.356639 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.356459    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/949ffdd5-088b-4f73-b4d7-41921e724eab-registry-tls\") pod \"image-registry-5b7ff5c8fb-j995z\" (UID: \"949ffdd5-088b-4f73-b4d7-41921e724eab\") " pod="openshift-image-registry/image-registry-5b7ff5c8fb-j995z"
Jul 28 17:24:36.356639 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.356498    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzltq\" (UniqueName: \"kubernetes.io/projected/949ffdd5-088b-4f73-b4d7-41921e724eab-kube-api-access-rzltq\") pod \"image-registry-5b7ff5c8fb-j995z\" (UID: \"949ffdd5-088b-4f73-b4d7-41921e724eab\") " pod="openshift-image-registry/image-registry-5b7ff5c8fb-j995z"
Jul 28 17:24:36.356639 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.356544    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1980fe9a-8463-4a8d-8462-d591761ffdf1-service-ca\") pod \"console-64cb7bf944-dh52l\" (UID: \"1980fe9a-8463-4a8d-8462-d591761ffdf1\") " pod="openshift-console/console-64cb7bf944-dh52l"
Jul 28 17:24:36.356639 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.356565    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/949ffdd5-088b-4f73-b4d7-41921e724eab-installation-pull-secrets\") pod \"image-registry-5b7ff5c8fb-j995z\" (UID: \"949ffdd5-088b-4f73-b4d7-41921e724eab\") " pod="openshift-image-registry/image-registry-5b7ff5c8fb-j995z"
Jul 28 17:24:36.356639 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.356595    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1980fe9a-8463-4a8d-8462-d591761ffdf1-console-oauth-config\") pod \"console-64cb7bf944-dh52l\" (UID: \"1980fe9a-8463-4a8d-8462-d591761ffdf1\") " pod="openshift-console/console-64cb7bf944-dh52l"
Jul 28 17:24:36.356639 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.356636    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1980fe9a-8463-4a8d-8462-d591761ffdf1-oauth-serving-cert\") pod \"console-64cb7bf944-dh52l\" (UID: \"1980fe9a-8463-4a8d-8462-d591761ffdf1\") " pod="openshift-console/console-64cb7bf944-dh52l"
Jul 28 17:24:36.356995 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.356671    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6fq6\" (UniqueName: \"kubernetes.io/projected/1980fe9a-8463-4a8d-8462-d591761ffdf1-kube-api-access-f6fq6\") pod \"console-64cb7bf944-dh52l\" (UID: \"1980fe9a-8463-4a8d-8462-d591761ffdf1\") " pod="openshift-console/console-64cb7bf944-dh52l"
Jul 28 17:24:36.356995 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.356700    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/949ffdd5-088b-4f73-b4d7-41921e724eab-ca-trust-extracted\") pod \"image-registry-5b7ff5c8fb-j995z\" (UID: \"949ffdd5-088b-4f73-b4d7-41921e724eab\") " pod="openshift-image-registry/image-registry-5b7ff5c8fb-j995z"
Jul 28 17:24:36.356995 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.356746    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/949ffdd5-088b-4f73-b4d7-41921e724eab-image-registry-private-configuration\") pod \"image-registry-5b7ff5c8fb-j995z\" (UID: \"949ffdd5-088b-4f73-b4d7-41921e724eab\") " pod="openshift-image-registry/image-registry-5b7ff5c8fb-j995z"
Jul 28 17:24:36.357694 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.357606    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1980fe9a-8463-4a8d-8462-d591761ffdf1-console-config\") pod \"console-64cb7bf944-dh52l\" (UID: \"1980fe9a-8463-4a8d-8462-d591761ffdf1\") " pod="openshift-console/console-64cb7bf944-dh52l"
Jul 28 17:24:36.359019 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.358714    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/949ffdd5-088b-4f73-b4d7-41921e724eab-ca-trust-extracted\") pod \"image-registry-5b7ff5c8fb-j995z\" (UID: \"949ffdd5-088b-4f73-b4d7-41921e724eab\") " pod="openshift-image-registry/image-registry-5b7ff5c8fb-j995z"
Jul 28 17:24:36.360987 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.359604    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/949ffdd5-088b-4f73-b4d7-41921e724eab-registry-certificates\") pod \"image-registry-5b7ff5c8fb-j995z\" (UID: \"949ffdd5-088b-4f73-b4d7-41921e724eab\") " pod="openshift-image-registry/image-registry-5b7ff5c8fb-j995z"
Jul 28 17:24:36.360987 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.359877    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/949ffdd5-088b-4f73-b4d7-41921e724eab-trusted-ca\") pod \"image-registry-5b7ff5c8fb-j995z\" (UID: \"949ffdd5-088b-4f73-b4d7-41921e724eab\") " pod="openshift-image-registry/image-registry-5b7ff5c8fb-j995z"
Jul 28 17:24:36.360987 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.360052    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/949ffdd5-088b-4f73-b4d7-41921e724eab-image-registry-private-configuration\") pod \"image-registry-5b7ff5c8fb-j995z\" (UID: \"949ffdd5-088b-4f73-b4d7-41921e724eab\") " pod="openshift-image-registry/image-registry-5b7ff5c8fb-j995z"
Jul 28 17:24:36.360987 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.360670    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1980fe9a-8463-4a8d-8462-d591761ffdf1-console-serving-cert\") pod \"console-64cb7bf944-dh52l\" (UID: \"1980fe9a-8463-4a8d-8462-d591761ffdf1\") " pod="openshift-console/console-64cb7bf944-dh52l"
Jul 28 17:24:36.360987 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.360536    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1980fe9a-8463-4a8d-8462-d591761ffdf1-console-oauth-config\") pod \"console-64cb7bf944-dh52l\" (UID: \"1980fe9a-8463-4a8d-8462-d591761ffdf1\") " pod="openshift-console/console-64cb7bf944-dh52l"
Jul 28 17:24:36.362186 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.362031    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1980fe9a-8463-4a8d-8462-d591761ffdf1-oauth-serving-cert\") pod \"console-64cb7bf944-dh52l\" (UID: \"1980fe9a-8463-4a8d-8462-d591761ffdf1\") " pod="openshift-console/console-64cb7bf944-dh52l"
Jul 28 17:24:36.362304 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.362269    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0231f8b7-62f6-4677-9ff7-a35efb83743d-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-zkptp\" (UID: \"0231f8b7-62f6-4677-9ff7-a35efb83743d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-zkptp"
Jul 28 17:24:36.362540 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.362420    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1980fe9a-8463-4a8d-8462-d591761ffdf1-service-ca\") pod \"console-64cb7bf944-dh52l\" (UID: \"1980fe9a-8463-4a8d-8462-d591761ffdf1\") " pod="openshift-console/console-64cb7bf944-dh52l"
Jul 28 17:24:36.365199 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.362833    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/949ffdd5-088b-4f73-b4d7-41921e724eab-installation-pull-secrets\") pod \"image-registry-5b7ff5c8fb-j995z\" (UID: \"949ffdd5-088b-4f73-b4d7-41921e724eab\") " pod="openshift-image-registry/image-registry-5b7ff5c8fb-j995z"
Jul 28 17:24:36.365599 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.365576    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/949ffdd5-088b-4f73-b4d7-41921e724eab-bound-sa-token\") pod \"image-registry-5b7ff5c8fb-j995z\" (UID: \"949ffdd5-088b-4f73-b4d7-41921e724eab\") " pod="openshift-image-registry/image-registry-5b7ff5c8fb-j995z"
Jul 28 17:24:36.365797 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.365775    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/949ffdd5-088b-4f73-b4d7-41921e724eab-registry-tls\") pod \"image-registry-5b7ff5c8fb-j995z\" (UID: \"949ffdd5-088b-4f73-b4d7-41921e724eab\") " pod="openshift-image-registry/image-registry-5b7ff5c8fb-j995z"
Jul 28 17:24:36.367110 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.367086    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzltq\" (UniqueName: \"kubernetes.io/projected/949ffdd5-088b-4f73-b4d7-41921e724eab-kube-api-access-rzltq\") pod \"image-registry-5b7ff5c8fb-j995z\" (UID: \"949ffdd5-088b-4f73-b4d7-41921e724eab\") " pod="openshift-image-registry/image-registry-5b7ff5c8fb-j995z"
Jul 28 17:24:36.380789 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.380765    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6fq6\" (UniqueName: \"kubernetes.io/projected/1980fe9a-8463-4a8d-8462-d591761ffdf1-kube-api-access-f6fq6\") pod \"console-64cb7bf944-dh52l\" (UID: \"1980fe9a-8463-4a8d-8462-d591761ffdf1\") " pod="openshift-console/console-64cb7bf944-dh52l"
Jul 28 17:24:36.415861 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.415827    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-zkptp"
Jul 28 17:24:36.423920 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.423884    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64cb7bf944-dh52l"
Jul 28 17:24:36.457173 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:36.457125    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b7ff5c8fb-j995z"
Jul 28 17:24:37.414432 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:37.414398    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64cb7bf944-dh52l"]
Jul 28 17:24:37.621726 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:24:37.621654    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1980fe9a_8463_4a8d_8462_d591761ffdf1.slice/crio-c7670277097614b4c3dc500d592d4010f3f034c733c4cb1eabe709add30dcc56 WatchSource:0}: Error finding container c7670277097614b4c3dc500d592d4010f3f034c733c4cb1eabe709add30dcc56: Status 404 returned error can't find the container with id c7670277097614b4c3dc500d592d4010f3f034c733c4cb1eabe709add30dcc56
Jul 28 17:24:37.759852 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:37.759809    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b7ff5c8fb-j995z"]
Jul 28 17:24:37.763404 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:24:37.763382    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod949ffdd5_088b_4f73_b4d7_41921e724eab.slice/crio-09a67c4adf5cc8ca418b6e09dcf198acba6c17e13fa4711c9de333b9d00c79a3 WatchSource:0}: Error finding container 09a67c4adf5cc8ca418b6e09dcf198acba6c17e13fa4711c9de333b9d00c79a3: Status 404 returned error can't find the container with id 09a67c4adf5cc8ca418b6e09dcf198acba6c17e13fa4711c9de333b9d00c79a3
Jul 28 17:24:37.773655 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:37.773631    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-zkptp"]
Jul 28 17:24:37.777408 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:24:37.777379    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0231f8b7_62f6_4677_9ff7_a35efb83743d.slice/crio-f16cd4681493fb0cb6586fae7a78c60a445723035c9af9acbf9eeab1c0bc8b7b WatchSource:0}: Error finding container f16cd4681493fb0cb6586fae7a78c60a445723035c9af9acbf9eeab1c0bc8b7b: Status 404 returned error can't find the container with id f16cd4681493fb0cb6586fae7a78c60a445723035c9af9acbf9eeab1c0bc8b7b
Jul 28 17:24:38.549288 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:38.549198    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64cb7bf944-dh52l" event={"ID":"1980fe9a-8463-4a8d-8462-d591761ffdf1","Type":"ContainerStarted","Data":"c7670277097614b4c3dc500d592d4010f3f034c733c4cb1eabe709add30dcc56"}
Jul 28 17:24:38.552062 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:38.551980    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-czx2d" event={"ID":"66911cb6-549d-4f4f-8ac6-286401ad1025","Type":"ContainerStarted","Data":"8974de839fef67f40fc3d59611d3d96f395e98d3b9f23ddf1d59fbdf717da3fd"}
Jul 28 17:24:38.554080 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:38.554009    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-zkptp" event={"ID":"0231f8b7-62f6-4677-9ff7-a35efb83743d","Type":"ContainerStarted","Data":"f16cd4681493fb0cb6586fae7a78c60a445723035c9af9acbf9eeab1c0bc8b7b"}
Jul 28 17:24:38.557447 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:38.556629    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b7ff5c8fb-j995z" event={"ID":"949ffdd5-088b-4f73-b4d7-41921e724eab","Type":"ContainerStarted","Data":"fdababf747b1354790f62266d67877161c5ea1c7cd286a90138219858f0ec4c0"}
Jul 28 17:24:38.557447 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:38.556668    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b7ff5c8fb-j995z" event={"ID":"949ffdd5-088b-4f73-b4d7-41921e724eab","Type":"ContainerStarted","Data":"09a67c4adf5cc8ca418b6e09dcf198acba6c17e13fa4711c9de333b9d00c79a3"}
Jul 28 17:24:38.557447 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:38.557407    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5b7ff5c8fb-j995z"
Jul 28 17:24:38.580571 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:38.579587    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5b7ff5c8fb-j995z" podStartSLOduration=2.57956679 podStartE2EDuration="2.57956679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:38.578533037 +0000 UTC m=+81.984806778" watchObservedRunningTime="2026-07-28 17:24:38.57956679 +0000 UTC m=+81.985840531"
Jul 28 17:24:39.518428 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:39.518403    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hg8ms"
Jul 28 17:24:39.563442 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:39.563007    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-zkptp" event={"ID":"0231f8b7-62f6-4677-9ff7-a35efb83743d","Type":"ContainerStarted","Data":"fc8343795803e221e27dd05e3702bea3f8a8db605d86730ac05817c43e8b47c9"}
Jul 28 17:24:39.590201 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:39.590148    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-zkptp" podStartSLOduration=1.942487817 podStartE2EDuration="3.590111979s" totalImagesPullingTime="1.647624162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:39.589032484 +0000 UTC m=+82.995306223" watchObservedRunningTime="2026-07-28 17:24:39.590111979 +0000 UTC m=+82.996385721"
Jul 28 17:24:40.566848 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:40.566810    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-zkptp"
Jul 28 17:24:40.571988 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:40.571961    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-zkptp"
Jul 28 17:24:43.578585 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:43.578547    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64cb7bf944-dh52l" event={"ID":"1980fe9a-8463-4a8d-8462-d591761ffdf1","Type":"ContainerStarted","Data":"3cc87b01ebfb259ddb1ecc08115b21860780fd3948e65055eedfe36269cb43c3"}
Jul 28 17:24:43.580534 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:43.580510    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-czx2d" event={"ID":"66911cb6-549d-4f4f-8ac6-286401ad1025","Type":"ContainerStarted","Data":"0f2e2c98fae0311078197779c142def07c990aca0c4e00793c8a299455382574"}
Jul 28 17:24:43.602255 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:43.602188    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64cb7bf944-dh52l" podStartSLOduration=2.556029343 podStartE2EDuration="7.602170128s" totalImagesPullingTime="5.046140785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:43.597153167 +0000 UTC m=+87.003426903" watchObservedRunningTime="2026-07-28 17:24:43.602170128 +0000 UTC m=+87.008443874"
Jul 28 17:24:46.424113 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.424028    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64cb7bf944-dh52l"
Jul 28 17:24:46.424113 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.424089    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64cb7bf944-dh52l"
Jul 28 17:24:46.429684 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.429662    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64cb7bf944-dh52l"
Jul 28 17:24:46.593244 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.593198    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-czx2d" event={"ID":"66911cb6-549d-4f4f-8ac6-286401ad1025","Type":"ContainerStarted","Data":"9f47b4a34dda767b102f2255e64ad8d30889a98e848927b1ccb1d2be1ba246aa"}
Jul 28 17:24:46.597161 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.597116    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64cb7bf944-dh52l"
Jul 28 17:24:46.619045 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.619002    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-czx2d" podStartSLOduration=33.387358445 podStartE2EDuration="46.618987239s" totalImagesPullingTime="13.231628794s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:24:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:46.61824798 +0000 UTC m=+90.024521730" watchObservedRunningTime="2026-07-28 17:24:46.618987239 +0000 UTC m=+90.025260980"
Jul 28 17:24:46.643730 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.643702    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-g2cmr"]
Jul 28 17:24:46.646740 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.646723    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-g2cmr"
Jul 28 17:24:46.649681 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.649394    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 17:24:46.649681 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.649494    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 17:24:46.649681 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.649512    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n84sh\""
Jul 28 17:24:46.649887 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.649829    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 17:24:46.649940 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.649875    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 17:24:46.650069 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.650054    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 17:24:46.652020 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.651998    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 17:24:46.742358 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.742336    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/341c4c28-6384-46c0-964b-cc2a892e8b5f-node-exporter-wtmp\") pod \"node-exporter-g2cmr\" (UID: \"341c4c28-6384-46c0-964b-cc2a892e8b5f\") " pod="openshift-monitoring/node-exporter-g2cmr"
Jul 28 17:24:46.742527 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.742367    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/341c4c28-6384-46c0-964b-cc2a892e8b5f-node-exporter-accelerators-collector-config\") pod \"node-exporter-g2cmr\" (UID: \"341c4c28-6384-46c0-964b-cc2a892e8b5f\") " pod="openshift-monitoring/node-exporter-g2cmr"
Jul 28 17:24:46.742527 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.742390    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/341c4c28-6384-46c0-964b-cc2a892e8b5f-root\") pod \"node-exporter-g2cmr\" (UID: \"341c4c28-6384-46c0-964b-cc2a892e8b5f\") " pod="openshift-monitoring/node-exporter-g2cmr"
Jul 28 17:24:46.742527 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.742503    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/341c4c28-6384-46c0-964b-cc2a892e8b5f-node-exporter-textfile\") pod \"node-exporter-g2cmr\" (UID: \"341c4c28-6384-46c0-964b-cc2a892e8b5f\") " pod="openshift-monitoring/node-exporter-g2cmr"
Jul 28 17:24:46.742687 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.742619    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/341c4c28-6384-46c0-964b-cc2a892e8b5f-sys\") pod \"node-exporter-g2cmr\" (UID: \"341c4c28-6384-46c0-964b-cc2a892e8b5f\") " pod="openshift-monitoring/node-exporter-g2cmr"
Jul 28 17:24:46.742747 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.742721    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/341c4c28-6384-46c0-964b-cc2a892e8b5f-metrics-client-ca\") pod \"node-exporter-g2cmr\" (UID: \"341c4c28-6384-46c0-964b-cc2a892e8b5f\") " pod="openshift-monitoring/node-exporter-g2cmr"
Jul 28 17:24:46.742803 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.742791    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4ldc\" (UniqueName: \"kubernetes.io/projected/341c4c28-6384-46c0-964b-cc2a892e8b5f-kube-api-access-z4ldc\") pod \"node-exporter-g2cmr\" (UID: \"341c4c28-6384-46c0-964b-cc2a892e8b5f\") " pod="openshift-monitoring/node-exporter-g2cmr"
Jul 28 17:24:46.742912 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.742893    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/341c4c28-6384-46c0-964b-cc2a892e8b5f-node-exporter-tls\") pod \"node-exporter-g2cmr\" (UID: \"341c4c28-6384-46c0-964b-cc2a892e8b5f\") " pod="openshift-monitoring/node-exporter-g2cmr"
Jul 28 17:24:46.742999 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.742982    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/341c4c28-6384-46c0-964b-cc2a892e8b5f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g2cmr\" (UID: \"341c4c28-6384-46c0-964b-cc2a892e8b5f\") " pod="openshift-monitoring/node-exporter-g2cmr"
Jul 28 17:24:46.845027 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.844025    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/341c4c28-6384-46c0-964b-cc2a892e8b5f-node-exporter-textfile\") pod \"node-exporter-g2cmr\" (UID: \"341c4c28-6384-46c0-964b-cc2a892e8b5f\") " pod="openshift-monitoring/node-exporter-g2cmr"
Jul 28 17:24:46.845027 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.844095    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/341c4c28-6384-46c0-964b-cc2a892e8b5f-sys\") pod \"node-exporter-g2cmr\" (UID: \"341c4c28-6384-46c0-964b-cc2a892e8b5f\") " pod="openshift-monitoring/node-exporter-g2cmr"
Jul 28 17:24:46.845027 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.844129    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/341c4c28-6384-46c0-964b-cc2a892e8b5f-metrics-client-ca\") pod \"node-exporter-g2cmr\" (UID: \"341c4c28-6384-46c0-964b-cc2a892e8b5f\") " pod="openshift-monitoring/node-exporter-g2cmr"
Jul 28 17:24:46.845027 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.844179    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4ldc\" (UniqueName: \"kubernetes.io/projected/341c4c28-6384-46c0-964b-cc2a892e8b5f-kube-api-access-z4ldc\") pod \"node-exporter-g2cmr\" (UID: \"341c4c28-6384-46c0-964b-cc2a892e8b5f\") " pod="openshift-monitoring/node-exporter-g2cmr"
Jul 28 17:24:46.845027 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.844203    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/341c4c28-6384-46c0-964b-cc2a892e8b5f-node-exporter-tls\") pod \"node-exporter-g2cmr\" (UID: \"341c4c28-6384-46c0-964b-cc2a892e8b5f\") " pod="openshift-monitoring/node-exporter-g2cmr"
Jul 28 17:24:46.845027 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.844242    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/341c4c28-6384-46c0-964b-cc2a892e8b5f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g2cmr\" (UID: \"341c4c28-6384-46c0-964b-cc2a892e8b5f\") " pod="openshift-monitoring/node-exporter-g2cmr"
Jul 28 17:24:46.845027 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.844272    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/341c4c28-6384-46c0-964b-cc2a892e8b5f-node-exporter-wtmp\") pod \"node-exporter-g2cmr\" (UID: \"341c4c28-6384-46c0-964b-cc2a892e8b5f\") " pod="openshift-monitoring/node-exporter-g2cmr"
Jul 28 17:24:46.845027 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.844296    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/341c4c28-6384-46c0-964b-cc2a892e8b5f-node-exporter-accelerators-collector-config\") pod \"node-exporter-g2cmr\" (UID: \"341c4c28-6384-46c0-964b-cc2a892e8b5f\") " pod="openshift-monitoring/node-exporter-g2cmr"
Jul 28 17:24:46.845027 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.844323    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/341c4c28-6384-46c0-964b-cc2a892e8b5f-root\") pod \"node-exporter-g2cmr\" (UID: \"341c4c28-6384-46c0-964b-cc2a892e8b5f\") " pod="openshift-monitoring/node-exporter-g2cmr"
Jul 28 17:24:46.845027 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.844414    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/341c4c28-6384-46c0-964b-cc2a892e8b5f-root\") pod \"node-exporter-g2cmr\" (UID: \"341c4c28-6384-46c0-964b-cc2a892e8b5f\") " pod="openshift-monitoring/node-exporter-g2cmr"
Jul 28 17:24:46.845027 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.844741    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/341c4c28-6384-46c0-964b-cc2a892e8b5f-node-exporter-textfile\") pod \"node-exporter-g2cmr\" (UID: \"341c4c28-6384-46c0-964b-cc2a892e8b5f\") " pod="openshift-monitoring/node-exporter-g2cmr"
Jul 28 17:24:46.845027 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.844790    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/341c4c28-6384-46c0-964b-cc2a892e8b5f-sys\") pod \"node-exporter-g2cmr\" (UID: \"341c4c28-6384-46c0-964b-cc2a892e8b5f\") " pod="openshift-monitoring/node-exporter-g2cmr"
Jul 28 17:24:46.845934 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.845270    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/341c4c28-6384-46c0-964b-cc2a892e8b5f-metrics-client-ca\") pod \"node-exporter-g2cmr\" (UID: \"341c4c28-6384-46c0-964b-cc2a892e8b5f\") " pod="openshift-monitoring/node-exporter-g2cmr"
Jul 28 17:24:46.845934 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:46.845872    2543 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 17:24:46.845934 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:24:46.845934    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/341c4c28-6384-46c0-964b-cc2a892e8b5f-node-exporter-tls podName:341c4c28-6384-46c0-964b-cc2a892e8b5f nodeName:}" failed. No retries permitted until 2026-07-28 17:24:47.345915382 +0000 UTC m=+90.752189106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/341c4c28-6384-46c0-964b-cc2a892e8b5f-node-exporter-tls") pod "node-exporter-g2cmr" (UID: "341c4c28-6384-46c0-964b-cc2a892e8b5f") : secret "node-exporter-tls" not found
Jul 28 17:24:46.846271 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.846220    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/341c4c28-6384-46c0-964b-cc2a892e8b5f-node-exporter-accelerators-collector-config\") pod \"node-exporter-g2cmr\" (UID: \"341c4c28-6384-46c0-964b-cc2a892e8b5f\") " pod="openshift-monitoring/node-exporter-g2cmr"
Jul 28 17:24:46.847673 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.845399    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/341c4c28-6384-46c0-964b-cc2a892e8b5f-node-exporter-wtmp\") pod \"node-exporter-g2cmr\" (UID: \"341c4c28-6384-46c0-964b-cc2a892e8b5f\") " pod="openshift-monitoring/node-exporter-g2cmr"
Jul 28 17:24:46.848964 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.848913    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/341c4c28-6384-46c0-964b-cc2a892e8b5f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g2cmr\" (UID: \"341c4c28-6384-46c0-964b-cc2a892e8b5f\") " pod="openshift-monitoring/node-exporter-g2cmr"
Jul 28 17:24:46.855278 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:46.855257    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4ldc\" (UniqueName: \"kubernetes.io/projected/341c4c28-6384-46c0-964b-cc2a892e8b5f-kube-api-access-z4ldc\") pod \"node-exporter-g2cmr\" (UID: \"341c4c28-6384-46c0-964b-cc2a892e8b5f\") " pod="openshift-monitoring/node-exporter-g2cmr"
Jul 28 17:24:47.347612 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:47.347581    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/341c4c28-6384-46c0-964b-cc2a892e8b5f-node-exporter-tls\") pod \"node-exporter-g2cmr\" (UID: \"341c4c28-6384-46c0-964b-cc2a892e8b5f\") " pod="openshift-monitoring/node-exporter-g2cmr"
Jul 28 17:24:47.349845 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:47.349821    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/341c4c28-6384-46c0-964b-cc2a892e8b5f-node-exporter-tls\") pod \"node-exporter-g2cmr\" (UID: \"341c4c28-6384-46c0-964b-cc2a892e8b5f\") " pod="openshift-monitoring/node-exporter-g2cmr"
Jul 28 17:24:47.558818 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:47.558770    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-g2cmr"
Jul 28 17:24:47.570533 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:24:47.570510    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod341c4c28_6384_46c0_964b_cc2a892e8b5f.slice/crio-d85e77e0f9ae79ea821b85f9221a2cfb3f96e9b8bc857517a4e24b8b33541c77 WatchSource:0}: Error finding container d85e77e0f9ae79ea821b85f9221a2cfb3f96e9b8bc857517a4e24b8b33541c77: Status 404 returned error can't find the container with id d85e77e0f9ae79ea821b85f9221a2cfb3f96e9b8bc857517a4e24b8b33541c77
Jul 28 17:24:47.598451 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:47.598172    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g2cmr" event={"ID":"341c4c28-6384-46c0-964b-cc2a892e8b5f","Type":"ContainerStarted","Data":"d85e77e0f9ae79ea821b85f9221a2cfb3f96e9b8bc857517a4e24b8b33541c77"}
Jul 28 17:24:47.763278 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:47.763251    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:24:47.767834 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:47.767817    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:47.771150 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:47.771113    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 17:24:47.771428 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:47.771411    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 17:24:47.772674 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:47.772658    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 17:24:47.772731 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:47.772674    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 17:24:47.775313 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:47.775295    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 17:24:47.775713 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:47.775692    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 17:24:47.775713 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:47.775706    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 17:24:47.775878 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:47.775709    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 17:24:47.775878 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:47.775746    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8prvr\""
Jul 28 17:24:47.776435 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:47.776418    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 17:24:47.789180 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:47.789160    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:24:47.953249 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:47.953174    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m9kk\" (UniqueName: \"kubernetes.io/projected/3933c0c1-cbfe-48ab-a824-02a85a224c7b-kube-api-access-8m9kk\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:47.953249 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:47.953223    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:47.953450 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:47.953261    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3933c0c1-cbfe-48ab-a824-02a85a224c7b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:47.953450 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:47.953329    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3933c0c1-cbfe-48ab-a824-02a85a224c7b-config-out\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:47.953450 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:47.953357    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:47.953450 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:47.953381    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:47.953450 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:47.953408    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-config-volume\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:47.953450 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:47.953436    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3933c0c1-cbfe-48ab-a824-02a85a224c7b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:47.953742 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:47.953459    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3933c0c1-cbfe-48ab-a824-02a85a224c7b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:47.953742 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:47.953544    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:47.953742 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:47.953598    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:47.953742 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:47.953628    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-web-config\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:47.953742 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:47.953662    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3933c0c1-cbfe-48ab-a824-02a85a224c7b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:48.054314 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:48.054279    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:48.054496 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:48.054331    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:48.054496 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:48.054361    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-web-config\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:48.054496 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:48.054389    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3933c0c1-cbfe-48ab-a824-02a85a224c7b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:48.054496 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:48.054435    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m9kk\" (UniqueName: \"kubernetes.io/projected/3933c0c1-cbfe-48ab-a824-02a85a224c7b-kube-api-access-8m9kk\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:48.054496 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:48.054466    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:48.054758 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:48.054497    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3933c0c1-cbfe-48ab-a824-02a85a224c7b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:48.054758 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:48.054533    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3933c0c1-cbfe-48ab-a824-02a85a224c7b-config-out\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:48.054758 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:48.054561    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:48.054758 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:48.054587    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:48.054758 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:48.054613    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-config-volume\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:48.054758 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:48.054645    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3933c0c1-cbfe-48ab-a824-02a85a224c7b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:48.054758 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:48.054694    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3933c0c1-cbfe-48ab-a824-02a85a224c7b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:48.055167 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:48.055119    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3933c0c1-cbfe-48ab-a824-02a85a224c7b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:48.055360 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:48.055337    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3933c0c1-cbfe-48ab-a824-02a85a224c7b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:48.055739 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:48.055628    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3933c0c1-cbfe-48ab-a824-02a85a224c7b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:48.058449 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:48.057710    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-web-config\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:48.058449 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:48.057840    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:48.058449 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:48.057944    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3933c0c1-cbfe-48ab-a824-02a85a224c7b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:48.058449 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:48.058386    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:48.059159 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:48.058900    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:48.059159 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:48.058944    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3933c0c1-cbfe-48ab-a824-02a85a224c7b-config-out\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:48.059377 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:48.059358    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:48.059570 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:48.059551    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-config-volume\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:48.060513 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:48.060488    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:48.067312 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:48.067293    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m9kk\" (UniqueName: \"kubernetes.io/projected/3933c0c1-cbfe-48ab-a824-02a85a224c7b-kube-api-access-8m9kk\") pod \"alertmanager-main-0\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:48.077202 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:48.077182    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:24:48.225028 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:48.224954    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:24:48.363550 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:24:48.363516    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3933c0c1_cbfe_48ab_a824_02a85a224c7b.slice/crio-0714f42cfc10f7baf5669d38ddd09a58d04b986a8bdad16660c7d5d5364bff56 WatchSource:0}: Error finding container 0714f42cfc10f7baf5669d38ddd09a58d04b986a8bdad16660c7d5d5364bff56: Status 404 returned error can't find the container with id 0714f42cfc10f7baf5669d38ddd09a58d04b986a8bdad16660c7d5d5364bff56
Jul 28 17:24:48.602567 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:48.602536    2543 generic.go:356] "Generic (PLEG): container finished" podID="341c4c28-6384-46c0-964b-cc2a892e8b5f" containerID="012e9dd56c8fd016e82dc3d7c3dec2d692498ea4516168f2f2ae2872664e3c67" exitCode=0
Jul 28 17:24:48.603003 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:48.602623    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g2cmr" event={"ID":"341c4c28-6384-46c0-964b-cc2a892e8b5f","Type":"ContainerDied","Data":"012e9dd56c8fd016e82dc3d7c3dec2d692498ea4516168f2f2ae2872664e3c67"}
Jul 28 17:24:48.603800 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:48.603783    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3933c0c1-cbfe-48ab-a824-02a85a224c7b","Type":"ContainerStarted","Data":"0714f42cfc10f7baf5669d38ddd09a58d04b986a8bdad16660c7d5d5364bff56"}
Jul 28 17:24:49.610176 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:49.610121    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g2cmr" event={"ID":"341c4c28-6384-46c0-964b-cc2a892e8b5f","Type":"ContainerStarted","Data":"7929de3fe72e7ed2b6a5eff563c5cdaaa4db50d5c64c51985beeca71935c6115"}
Jul 28 17:24:49.610618 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:49.610197    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g2cmr" event={"ID":"341c4c28-6384-46c0-964b-cc2a892e8b5f","Type":"ContainerStarted","Data":"826fe88adc257693c873e9fff4a94e903f2717d09ea979bdd8de0df3ff9378d5"}
Jul 28 17:24:49.611889 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:49.611861    2543 generic.go:356] "Generic (PLEG): container finished" podID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerID="b7205f86691ca526181d7d26ff9ef264c63e83173ad858d3583637b73f2f02ca" exitCode=0
Jul 28 17:24:49.611984 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:49.611916    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3933c0c1-cbfe-48ab-a824-02a85a224c7b","Type":"ContainerDied","Data":"b7205f86691ca526181d7d26ff9ef264c63e83173ad858d3583637b73f2f02ca"}
Jul 28 17:24:49.631561 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:49.631516    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-g2cmr" podStartSLOduration=2.733653526 podStartE2EDuration="3.63150157s" totalImagesPullingTime="835.108311ms" totalInitContainerRuntime="62.739733ms" isStatefulPod=true podCreationTimestamp="2026-07-28 17:24:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:49.62990841 +0000 UTC m=+93.036182150" watchObservedRunningTime="2026-07-28 17:24:49.63150157 +0000 UTC m=+93.037775311"
Jul 28 17:24:50.988796 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:50.988756    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-789948b4bd-tbvt4"]
Jul 28 17:24:50.993907 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:50.993885    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-789948b4bd-tbvt4"
Jul 28 17:24:50.997707 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:50.997681    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 28 17:24:50.997969 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:50.997951    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 17:24:50.998064 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:50.997976    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 28 17:24:50.998663 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:50.998510    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-m46fq\""
Jul 28 17:24:50.998663 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:50.998512    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-b7a6p97k4qnre\""
Jul 28 17:24:50.998663 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:50.998591    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 28 17:24:51.004691 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.004668    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-789948b4bd-tbvt4"]
Jul 28 17:24:51.079198 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.079164    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9853b4ab-2da8-411e-8f52-089d275dedb0-secret-metrics-server-tls\") pod \"metrics-server-789948b4bd-tbvt4\" (UID: \"9853b4ab-2da8-411e-8f52-089d275dedb0\") " pod="openshift-monitoring/metrics-server-789948b4bd-tbvt4"
Jul 28 17:24:51.079360 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.079204    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9853b4ab-2da8-411e-8f52-089d275dedb0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-789948b4bd-tbvt4\" (UID: \"9853b4ab-2da8-411e-8f52-089d275dedb0\") " pod="openshift-monitoring/metrics-server-789948b4bd-tbvt4"
Jul 28 17:24:51.079360 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.079229    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9853b4ab-2da8-411e-8f52-089d275dedb0-audit-log\") pod \"metrics-server-789948b4bd-tbvt4\" (UID: \"9853b4ab-2da8-411e-8f52-089d275dedb0\") " pod="openshift-monitoring/metrics-server-789948b4bd-tbvt4"
Jul 28 17:24:51.079360 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.079261    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9853b4ab-2da8-411e-8f52-089d275dedb0-metrics-server-audit-profiles\") pod \"metrics-server-789948b4bd-tbvt4\" (UID: \"9853b4ab-2da8-411e-8f52-089d275dedb0\") " pod="openshift-monitoring/metrics-server-789948b4bd-tbvt4"
Jul 28 17:24:51.079360 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.079282    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w85h4\" (UniqueName: \"kubernetes.io/projected/9853b4ab-2da8-411e-8f52-089d275dedb0-kube-api-access-w85h4\") pod \"metrics-server-789948b4bd-tbvt4\" (UID: \"9853b4ab-2da8-411e-8f52-089d275dedb0\") " pod="openshift-monitoring/metrics-server-789948b4bd-tbvt4"
Jul 28 17:24:51.079360 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.079300    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9853b4ab-2da8-411e-8f52-089d275dedb0-client-ca-bundle\") pod \"metrics-server-789948b4bd-tbvt4\" (UID: \"9853b4ab-2da8-411e-8f52-089d275dedb0\") " pod="openshift-monitoring/metrics-server-789948b4bd-tbvt4"
Jul 28 17:24:51.079360 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.079324    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9853b4ab-2da8-411e-8f52-089d275dedb0-secret-metrics-server-client-certs\") pod \"metrics-server-789948b4bd-tbvt4\" (UID: \"9853b4ab-2da8-411e-8f52-089d275dedb0\") " pod="openshift-monitoring/metrics-server-789948b4bd-tbvt4"
Jul 28 17:24:51.179765 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.179742    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9853b4ab-2da8-411e-8f52-089d275dedb0-secret-metrics-server-tls\") pod \"metrics-server-789948b4bd-tbvt4\" (UID: \"9853b4ab-2da8-411e-8f52-089d275dedb0\") " pod="openshift-monitoring/metrics-server-789948b4bd-tbvt4"
Jul 28 17:24:51.179830 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.179781    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9853b4ab-2da8-411e-8f52-089d275dedb0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-789948b4bd-tbvt4\" (UID: \"9853b4ab-2da8-411e-8f52-089d275dedb0\") " pod="openshift-monitoring/metrics-server-789948b4bd-tbvt4"
Jul 28 17:24:51.179830 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.179803    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9853b4ab-2da8-411e-8f52-089d275dedb0-audit-log\") pod \"metrics-server-789948b4bd-tbvt4\" (UID: \"9853b4ab-2da8-411e-8f52-089d275dedb0\") " pod="openshift-monitoring/metrics-server-789948b4bd-tbvt4"
Jul 28 17:24:51.179830 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.179825    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9853b4ab-2da8-411e-8f52-089d275dedb0-metrics-server-audit-profiles\") pod \"metrics-server-789948b4bd-tbvt4\" (UID: \"9853b4ab-2da8-411e-8f52-089d275dedb0\") " pod="openshift-monitoring/metrics-server-789948b4bd-tbvt4"
Jul 28 17:24:51.180037 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.179852    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w85h4\" (UniqueName: \"kubernetes.io/projected/9853b4ab-2da8-411e-8f52-089d275dedb0-kube-api-access-w85h4\") pod \"metrics-server-789948b4bd-tbvt4\" (UID: \"9853b4ab-2da8-411e-8f52-089d275dedb0\") " pod="openshift-monitoring/metrics-server-789948b4bd-tbvt4"
Jul 28 17:24:51.180037 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.179882    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9853b4ab-2da8-411e-8f52-089d275dedb0-client-ca-bundle\") pod \"metrics-server-789948b4bd-tbvt4\" (UID: \"9853b4ab-2da8-411e-8f52-089d275dedb0\") " pod="openshift-monitoring/metrics-server-789948b4bd-tbvt4"
Jul 28 17:24:51.180037 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.179917    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9853b4ab-2da8-411e-8f52-089d275dedb0-secret-metrics-server-client-certs\") pod \"metrics-server-789948b4bd-tbvt4\" (UID: \"9853b4ab-2da8-411e-8f52-089d275dedb0\") " pod="openshift-monitoring/metrics-server-789948b4bd-tbvt4"
Jul 28 17:24:51.180217 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.180203    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9853b4ab-2da8-411e-8f52-089d275dedb0-audit-log\") pod \"metrics-server-789948b4bd-tbvt4\" (UID: \"9853b4ab-2da8-411e-8f52-089d275dedb0\") " pod="openshift-monitoring/metrics-server-789948b4bd-tbvt4"
Jul 28 17:24:51.180686 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.180660    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9853b4ab-2da8-411e-8f52-089d275dedb0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-789948b4bd-tbvt4\" (UID: \"9853b4ab-2da8-411e-8f52-089d275dedb0\") " pod="openshift-monitoring/metrics-server-789948b4bd-tbvt4"
Jul 28 17:24:51.180859 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.180838    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9853b4ab-2da8-411e-8f52-089d275dedb0-metrics-server-audit-profiles\") pod \"metrics-server-789948b4bd-tbvt4\" (UID: \"9853b4ab-2da8-411e-8f52-089d275dedb0\") " pod="openshift-monitoring/metrics-server-789948b4bd-tbvt4"
Jul 28 17:24:51.182738 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.182705    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9853b4ab-2da8-411e-8f52-089d275dedb0-secret-metrics-server-client-certs\") pod \"metrics-server-789948b4bd-tbvt4\" (UID: \"9853b4ab-2da8-411e-8f52-089d275dedb0\") " pod="openshift-monitoring/metrics-server-789948b4bd-tbvt4"
Jul 28 17:24:51.182912 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.182885    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9853b4ab-2da8-411e-8f52-089d275dedb0-client-ca-bundle\") pod \"metrics-server-789948b4bd-tbvt4\" (UID: \"9853b4ab-2da8-411e-8f52-089d275dedb0\") " pod="openshift-monitoring/metrics-server-789948b4bd-tbvt4"
Jul 28 17:24:51.182976 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.182911    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9853b4ab-2da8-411e-8f52-089d275dedb0-secret-metrics-server-tls\") pod \"metrics-server-789948b4bd-tbvt4\" (UID: \"9853b4ab-2da8-411e-8f52-089d275dedb0\") " pod="openshift-monitoring/metrics-server-789948b4bd-tbvt4"
Jul 28 17:24:51.189451 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.189430    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w85h4\" (UniqueName: \"kubernetes.io/projected/9853b4ab-2da8-411e-8f52-089d275dedb0-kube-api-access-w85h4\") pod \"metrics-server-789948b4bd-tbvt4\" (UID: \"9853b4ab-2da8-411e-8f52-089d275dedb0\") " pod="openshift-monitoring/metrics-server-789948b4bd-tbvt4"
Jul 28 17:24:51.304797 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.304774    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-789948b4bd-tbvt4"
Jul 28 17:24:51.459566 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.459483    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-789948b4bd-tbvt4"]
Jul 28 17:24:51.466214 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:24:51.465235    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9853b4ab_2da8_411e_8f52_089d275dedb0.slice/crio-289c6476a1c3c5d1b75391d1cae33012e41e5e1d8be6b6c9ed0523de16d04c64 WatchSource:0}: Error finding container 289c6476a1c3c5d1b75391d1cae33012e41e5e1d8be6b6c9ed0523de16d04c64: Status 404 returned error can't find the container with id 289c6476a1c3c5d1b75391d1cae33012e41e5e1d8be6b6c9ed0523de16d04c64
Jul 28 17:24:51.623413 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.623379    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3933c0c1-cbfe-48ab-a824-02a85a224c7b","Type":"ContainerStarted","Data":"5656277fc9de31340e765938aae8d41f23f55a5172d8d10ecfc4176a11960441"}
Jul 28 17:24:51.623546 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.623452    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3933c0c1-cbfe-48ab-a824-02a85a224c7b","Type":"ContainerStarted","Data":"c4966596b6f6772627239d5d4e6a1d3c8185b9b1f65be35258828dc8b1964250"}
Jul 28 17:24:51.623546 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.623484    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3933c0c1-cbfe-48ab-a824-02a85a224c7b","Type":"ContainerStarted","Data":"817a5b4466b951268542e3e29c449f6cc9db6c5ba3fac00f6301933402fbe14a"}
Jul 28 17:24:51.623546 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.623520    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3933c0c1-cbfe-48ab-a824-02a85a224c7b","Type":"ContainerStarted","Data":"44b7b9fda6e230b94b83a7716d7cf6c6c5cf3f4d60d3c715deb68776dd2a5ce3"}
Jul 28 17:24:51.623679 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.623546    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3933c0c1-cbfe-48ab-a824-02a85a224c7b","Type":"ContainerStarted","Data":"dbf37094182d57da4400a7496af54f1aa7fa812f5e2b99756b881a37bc5e38f8"}
Jul 28 17:24:51.624484 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.624460    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-789948b4bd-tbvt4" event={"ID":"9853b4ab-2da8-411e-8f52-089d275dedb0","Type":"ContainerStarted","Data":"289c6476a1c3c5d1b75391d1cae33012e41e5e1d8be6b6c9ed0523de16d04c64"}
Jul 28 17:24:51.928091 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.928019    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5478597cd8-pw7rm"]
Jul 28 17:24:51.931703 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.931688    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5478597cd8-pw7rm"
Jul 28 17:24:51.934996 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.934939    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 28 17:24:51.935163 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.934970    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 28 17:24:51.935812 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.935793    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-mk7dm\""
Jul 28 17:24:51.935932 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.935853    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 28 17:24:51.935932 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.935904    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 28 17:24:51.936079 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.936071    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 28 17:24:51.942263 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.942245    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 28 17:24:51.961254 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.961233    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5478597cd8-pw7rm"]
Jul 28 17:24:51.986633 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.986605    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af758b14-72fa-48eb-bb20-4dc7c85693b7-serving-certs-ca-bundle\") pod \"telemeter-client-5478597cd8-pw7rm\" (UID: \"af758b14-72fa-48eb-bb20-4dc7c85693b7\") " pod="openshift-monitoring/telemeter-client-5478597cd8-pw7rm"
Jul 28 17:24:51.986793 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.986670    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af758b14-72fa-48eb-bb20-4dc7c85693b7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5478597cd8-pw7rm\" (UID: \"af758b14-72fa-48eb-bb20-4dc7c85693b7\") " pod="openshift-monitoring/telemeter-client-5478597cd8-pw7rm"
Jul 28 17:24:51.986793 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.986701    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qrzr\" (UniqueName: \"kubernetes.io/projected/af758b14-72fa-48eb-bb20-4dc7c85693b7-kube-api-access-7qrzr\") pod \"telemeter-client-5478597cd8-pw7rm\" (UID: \"af758b14-72fa-48eb-bb20-4dc7c85693b7\") " pod="openshift-monitoring/telemeter-client-5478597cd8-pw7rm"
Jul 28 17:24:51.986793 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.986740    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/af758b14-72fa-48eb-bb20-4dc7c85693b7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5478597cd8-pw7rm\" (UID: \"af758b14-72fa-48eb-bb20-4dc7c85693b7\") " pod="openshift-monitoring/telemeter-client-5478597cd8-pw7rm"
Jul 28 17:24:51.986793 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.986777    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/af758b14-72fa-48eb-bb20-4dc7c85693b7-federate-client-tls\") pod \"telemeter-client-5478597cd8-pw7rm\" (UID: \"af758b14-72fa-48eb-bb20-4dc7c85693b7\") " pod="openshift-monitoring/telemeter-client-5478597cd8-pw7rm"
Jul 28 17:24:51.986976 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.986819    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/af758b14-72fa-48eb-bb20-4dc7c85693b7-secret-telemeter-client\") pod \"telemeter-client-5478597cd8-pw7rm\" (UID: \"af758b14-72fa-48eb-bb20-4dc7c85693b7\") " pod="openshift-monitoring/telemeter-client-5478597cd8-pw7rm"
Jul 28 17:24:51.986976 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.986896    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/af758b14-72fa-48eb-bb20-4dc7c85693b7-telemeter-client-tls\") pod \"telemeter-client-5478597cd8-pw7rm\" (UID: \"af758b14-72fa-48eb-bb20-4dc7c85693b7\") " pod="openshift-monitoring/telemeter-client-5478597cd8-pw7rm"
Jul 28 17:24:51.986976 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:51.986964    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af758b14-72fa-48eb-bb20-4dc7c85693b7-metrics-client-ca\") pod \"telemeter-client-5478597cd8-pw7rm\" (UID: \"af758b14-72fa-48eb-bb20-4dc7c85693b7\") " pod="openshift-monitoring/telemeter-client-5478597cd8-pw7rm"
Jul 28 17:24:52.087542 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:52.087500    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af758b14-72fa-48eb-bb20-4dc7c85693b7-metrics-client-ca\") pod \"telemeter-client-5478597cd8-pw7rm\" (UID: \"af758b14-72fa-48eb-bb20-4dc7c85693b7\") " pod="openshift-monitoring/telemeter-client-5478597cd8-pw7rm"
Jul 28 17:24:52.087976 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:52.087568    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af758b14-72fa-48eb-bb20-4dc7c85693b7-serving-certs-ca-bundle\") pod \"telemeter-client-5478597cd8-pw7rm\" (UID: \"af758b14-72fa-48eb-bb20-4dc7c85693b7\") " pod="openshift-monitoring/telemeter-client-5478597cd8-pw7rm"
Jul 28 17:24:52.087976 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:52.087607    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af758b14-72fa-48eb-bb20-4dc7c85693b7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5478597cd8-pw7rm\" (UID: \"af758b14-72fa-48eb-bb20-4dc7c85693b7\") " pod="openshift-monitoring/telemeter-client-5478597cd8-pw7rm"
Jul 28 17:24:52.087976 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:52.087629    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qrzr\" (UniqueName: \"kubernetes.io/projected/af758b14-72fa-48eb-bb20-4dc7c85693b7-kube-api-access-7qrzr\") pod \"telemeter-client-5478597cd8-pw7rm\" (UID: \"af758b14-72fa-48eb-bb20-4dc7c85693b7\") " pod="openshift-monitoring/telemeter-client-5478597cd8-pw7rm"
Jul 28 17:24:52.087976 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:52.087659    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/af758b14-72fa-48eb-bb20-4dc7c85693b7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5478597cd8-pw7rm\" (UID: \"af758b14-72fa-48eb-bb20-4dc7c85693b7\") " pod="openshift-monitoring/telemeter-client-5478597cd8-pw7rm"
Jul 28 17:24:52.087976 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:52.087889    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/af758b14-72fa-48eb-bb20-4dc7c85693b7-federate-client-tls\") pod \"telemeter-client-5478597cd8-pw7rm\" (UID: \"af758b14-72fa-48eb-bb20-4dc7c85693b7\") " pod="openshift-monitoring/telemeter-client-5478597cd8-pw7rm"
Jul 28 17:24:52.087976 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:52.087931    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/af758b14-72fa-48eb-bb20-4dc7c85693b7-secret-telemeter-client\") pod \"telemeter-client-5478597cd8-pw7rm\" (UID: \"af758b14-72fa-48eb-bb20-4dc7c85693b7\") " pod="openshift-monitoring/telemeter-client-5478597cd8-pw7rm"
Jul 28 17:24:52.087976 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:52.087959    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/af758b14-72fa-48eb-bb20-4dc7c85693b7-telemeter-client-tls\") pod \"telemeter-client-5478597cd8-pw7rm\" (UID: \"af758b14-72fa-48eb-bb20-4dc7c85693b7\") " pod="openshift-monitoring/telemeter-client-5478597cd8-pw7rm"
Jul 28 17:24:52.088388 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:52.088364    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af758b14-72fa-48eb-bb20-4dc7c85693b7-metrics-client-ca\") pod \"telemeter-client-5478597cd8-pw7rm\" (UID: \"af758b14-72fa-48eb-bb20-4dc7c85693b7\") " pod="openshift-monitoring/telemeter-client-5478597cd8-pw7rm"
Jul 28 17:24:52.088455 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:52.088371    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af758b14-72fa-48eb-bb20-4dc7c85693b7-serving-certs-ca-bundle\") pod \"telemeter-client-5478597cd8-pw7rm\" (UID: \"af758b14-72fa-48eb-bb20-4dc7c85693b7\") " pod="openshift-monitoring/telemeter-client-5478597cd8-pw7rm"
Jul 28 17:24:52.088752 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:52.088717    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af758b14-72fa-48eb-bb20-4dc7c85693b7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5478597cd8-pw7rm\" (UID: \"af758b14-72fa-48eb-bb20-4dc7c85693b7\") " pod="openshift-monitoring/telemeter-client-5478597cd8-pw7rm"
Jul 28 17:24:52.090650 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:52.090608    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/af758b14-72fa-48eb-bb20-4dc7c85693b7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5478597cd8-pw7rm\" (UID: \"af758b14-72fa-48eb-bb20-4dc7c85693b7\") " pod="openshift-monitoring/telemeter-client-5478597cd8-pw7rm"
Jul 28 17:24:52.091148 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:52.091107    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/af758b14-72fa-48eb-bb20-4dc7c85693b7-telemeter-client-tls\") pod \"telemeter-client-5478597cd8-pw7rm\" (UID: \"af758b14-72fa-48eb-bb20-4dc7c85693b7\") " pod="openshift-monitoring/telemeter-client-5478597cd8-pw7rm"
Jul 28 17:24:52.091248 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:52.091107    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/af758b14-72fa-48eb-bb20-4dc7c85693b7-federate-client-tls\") pod \"telemeter-client-5478597cd8-pw7rm\" (UID: \"af758b14-72fa-48eb-bb20-4dc7c85693b7\") " pod="openshift-monitoring/telemeter-client-5478597cd8-pw7rm"
Jul 28 17:24:52.091248 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:52.091203    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/af758b14-72fa-48eb-bb20-4dc7c85693b7-secret-telemeter-client\") pod \"telemeter-client-5478597cd8-pw7rm\" (UID: \"af758b14-72fa-48eb-bb20-4dc7c85693b7\") " pod="openshift-monitoring/telemeter-client-5478597cd8-pw7rm"
Jul 28 17:24:52.105749 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:52.105719    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qrzr\" (UniqueName: \"kubernetes.io/projected/af758b14-72fa-48eb-bb20-4dc7c85693b7-kube-api-access-7qrzr\") pod \"telemeter-client-5478597cd8-pw7rm\" (UID: \"af758b14-72fa-48eb-bb20-4dc7c85693b7\") " pod="openshift-monitoring/telemeter-client-5478597cd8-pw7rm"
Jul 28 17:24:52.241980 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:52.241949    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5478597cd8-pw7rm"
Jul 28 17:24:52.410580 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:52.410549    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5478597cd8-pw7rm"]
Jul 28 17:24:52.414447 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:24:52.414408    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf758b14_72fa_48eb_bb20_4dc7c85693b7.slice/crio-5b8ddcffc547ab6ee596e8aebc0e124f366e846abdd00238ecf826269593141a WatchSource:0}: Error finding container 5b8ddcffc547ab6ee596e8aebc0e124f366e846abdd00238ecf826269593141a: Status 404 returned error can't find the container with id 5b8ddcffc547ab6ee596e8aebc0e124f366e846abdd00238ecf826269593141a
Jul 28 17:24:52.630054 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:52.629974    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5478597cd8-pw7rm" event={"ID":"af758b14-72fa-48eb-bb20-4dc7c85693b7","Type":"ContainerStarted","Data":"5b8ddcffc547ab6ee596e8aebc0e124f366e846abdd00238ecf826269593141a"}
Jul 28 17:24:52.636655 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:52.636627    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3933c0c1-cbfe-48ab-a824-02a85a224c7b","Type":"ContainerStarted","Data":"727b8557aa5f1d0822c70747fc22a374ca179be5ab4a71aae9f39f67722bf8f0"}
Jul 28 17:24:52.676667 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:52.676608    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.2143464330000002 podStartE2EDuration="5.67658842s" totalImagesPullingTime="3.443583235s" totalInitContainerRuntime="18.658752ms" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:52.674509723 +0000 UTC m=+96.080783464" watchObservedRunningTime="2026-07-28 17:24:52.67658842 +0000 UTC m=+96.082862160"
Jul 28 17:24:53.646294 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:53.646197    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-789948b4bd-tbvt4" event={"ID":"9853b4ab-2da8-411e-8f52-089d275dedb0","Type":"ContainerStarted","Data":"af304f0dd953adbe706308e3313fd8fcb5343ee1281a82c1bb11a69435935f63"}
Jul 28 17:24:53.674073 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:53.674025    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-789948b4bd-tbvt4" podStartSLOduration=1.837142914 podStartE2EDuration="3.674012294s" totalImagesPullingTime="1.83686938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:53.673807342 +0000 UTC m=+97.080081084" watchObservedRunningTime="2026-07-28 17:24:53.674012294 +0000 UTC m=+97.080286094"
Jul 28 17:24:54.212058 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:54.212028    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f968994d7-4jr7x"]
Jul 28 17:24:54.215543 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:54.215514    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f968994d7-4jr7x"
Jul 28 17:24:54.223121 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:54.223095    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 17:24:54.231280 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:54.231224    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f968994d7-4jr7x"]
Jul 28 17:24:54.310827 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:54.310790    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa2cde03-a473-40f2-ac59-b383486afcc2-trusted-ca-bundle\") pod \"console-f968994d7-4jr7x\" (UID: \"aa2cde03-a473-40f2-ac59-b383486afcc2\") " pod="openshift-console/console-f968994d7-4jr7x"
Jul 28 17:24:54.310998 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:54.310882    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wvhs\" (UniqueName: \"kubernetes.io/projected/aa2cde03-a473-40f2-ac59-b383486afcc2-kube-api-access-8wvhs\") pod \"console-f968994d7-4jr7x\" (UID: \"aa2cde03-a473-40f2-ac59-b383486afcc2\") " pod="openshift-console/console-f968994d7-4jr7x"
Jul 28 17:24:54.310998 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:54.310915    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa2cde03-a473-40f2-ac59-b383486afcc2-console-oauth-config\") pod \"console-f968994d7-4jr7x\" (UID: \"aa2cde03-a473-40f2-ac59-b383486afcc2\") " pod="openshift-console/console-f968994d7-4jr7x"
Jul 28 17:24:54.310998 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:54.310942    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa2cde03-a473-40f2-ac59-b383486afcc2-oauth-serving-cert\") pod \"console-f968994d7-4jr7x\" (UID: \"aa2cde03-a473-40f2-ac59-b383486afcc2\") " pod="openshift-console/console-f968994d7-4jr7x"
Jul 28 17:24:54.311200 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:54.311018    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa2cde03-a473-40f2-ac59-b383486afcc2-service-ca\") pod \"console-f968994d7-4jr7x\" (UID: \"aa2cde03-a473-40f2-ac59-b383486afcc2\") " pod="openshift-console/console-f968994d7-4jr7x"
Jul 28 17:24:54.311200 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:54.311074    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa2cde03-a473-40f2-ac59-b383486afcc2-console-config\") pod \"console-f968994d7-4jr7x\" (UID: \"aa2cde03-a473-40f2-ac59-b383486afcc2\") " pod="openshift-console/console-f968994d7-4jr7x"
Jul 28 17:24:54.311200 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:54.311102    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa2cde03-a473-40f2-ac59-b383486afcc2-console-serving-cert\") pod \"console-f968994d7-4jr7x\" (UID: \"aa2cde03-a473-40f2-ac59-b383486afcc2\") " pod="openshift-console/console-f968994d7-4jr7x"
Jul 28 17:24:54.412452 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:54.412408    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa2cde03-a473-40f2-ac59-b383486afcc2-console-oauth-config\") pod \"console-f968994d7-4jr7x\" (UID: \"aa2cde03-a473-40f2-ac59-b383486afcc2\") " pod="openshift-console/console-f968994d7-4jr7x"
Jul 28 17:24:54.412640 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:54.412477    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa2cde03-a473-40f2-ac59-b383486afcc2-oauth-serving-cert\") pod \"console-f968994d7-4jr7x\" (UID: \"aa2cde03-a473-40f2-ac59-b383486afcc2\") " pod="openshift-console/console-f968994d7-4jr7x"
Jul 28 17:24:54.412640 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:54.412515    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa2cde03-a473-40f2-ac59-b383486afcc2-service-ca\") pod \"console-f968994d7-4jr7x\" (UID: \"aa2cde03-a473-40f2-ac59-b383486afcc2\") " pod="openshift-console/console-f968994d7-4jr7x"
Jul 28 17:24:54.412640 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:54.412540    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa2cde03-a473-40f2-ac59-b383486afcc2-console-config\") pod \"console-f968994d7-4jr7x\" (UID: \"aa2cde03-a473-40f2-ac59-b383486afcc2\") " pod="openshift-console/console-f968994d7-4jr7x"
Jul 28 17:24:54.412640 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:54.412561    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa2cde03-a473-40f2-ac59-b383486afcc2-console-serving-cert\") pod \"console-f968994d7-4jr7x\" (UID: \"aa2cde03-a473-40f2-ac59-b383486afcc2\") " pod="openshift-console/console-f968994d7-4jr7x"
Jul 28 17:24:54.412640 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:54.412585    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa2cde03-a473-40f2-ac59-b383486afcc2-trusted-ca-bundle\") pod \"console-f968994d7-4jr7x\" (UID: \"aa2cde03-a473-40f2-ac59-b383486afcc2\") " pod="openshift-console/console-f968994d7-4jr7x"
Jul 28 17:24:54.412898 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:54.412652    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wvhs\" (UniqueName: \"kubernetes.io/projected/aa2cde03-a473-40f2-ac59-b383486afcc2-kube-api-access-8wvhs\") pod \"console-f968994d7-4jr7x\" (UID: \"aa2cde03-a473-40f2-ac59-b383486afcc2\") " pod="openshift-console/console-f968994d7-4jr7x"
Jul 28 17:24:54.413431 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:54.413397    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa2cde03-a473-40f2-ac59-b383486afcc2-service-ca\") pod \"console-f968994d7-4jr7x\" (UID: \"aa2cde03-a473-40f2-ac59-b383486afcc2\") " pod="openshift-console/console-f968994d7-4jr7x"
Jul 28 17:24:54.413557 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:54.413488    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa2cde03-a473-40f2-ac59-b383486afcc2-oauth-serving-cert\") pod \"console-f968994d7-4jr7x\" (UID: \"aa2cde03-a473-40f2-ac59-b383486afcc2\") " pod="openshift-console/console-f968994d7-4jr7x"
Jul 28 17:24:54.413557 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:54.413537    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa2cde03-a473-40f2-ac59-b383486afcc2-console-config\") pod \"console-f968994d7-4jr7x\" (UID: \"aa2cde03-a473-40f2-ac59-b383486afcc2\") " pod="openshift-console/console-f968994d7-4jr7x"
Jul 28 17:24:54.413816 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:54.413795    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa2cde03-a473-40f2-ac59-b383486afcc2-trusted-ca-bundle\") pod \"console-f968994d7-4jr7x\" (UID: \"aa2cde03-a473-40f2-ac59-b383486afcc2\") " pod="openshift-console/console-f968994d7-4jr7x"
Jul 28 17:24:54.415471 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:54.415444    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa2cde03-a473-40f2-ac59-b383486afcc2-console-serving-cert\") pod \"console-f968994d7-4jr7x\" (UID: \"aa2cde03-a473-40f2-ac59-b383486afcc2\") " pod="openshift-console/console-f968994d7-4jr7x"
Jul 28 17:24:54.415573 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:54.415547    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa2cde03-a473-40f2-ac59-b383486afcc2-console-oauth-config\") pod \"console-f968994d7-4jr7x\" (UID: \"aa2cde03-a473-40f2-ac59-b383486afcc2\") " pod="openshift-console/console-f968994d7-4jr7x"
Jul 28 17:24:54.422598 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:54.422552    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wvhs\" (UniqueName: \"kubernetes.io/projected/aa2cde03-a473-40f2-ac59-b383486afcc2-kube-api-access-8wvhs\") pod \"console-f968994d7-4jr7x\" (UID: \"aa2cde03-a473-40f2-ac59-b383486afcc2\") " pod="openshift-console/console-f968994d7-4jr7x"
Jul 28 17:24:54.526476 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:54.526445    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f968994d7-4jr7x"
Jul 28 17:24:54.799574 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:54.799553    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f968994d7-4jr7x"]
Jul 28 17:24:54.802318 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:24:54.802285    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa2cde03_a473_40f2_ac59_b383486afcc2.slice/crio-c77f169fff69f2fdf233be1c1a3671299467f124ac5e4478b7f1f728d6273b34 WatchSource:0}: Error finding container c77f169fff69f2fdf233be1c1a3671299467f124ac5e4478b7f1f728d6273b34: Status 404 returned error can't find the container with id c77f169fff69f2fdf233be1c1a3671299467f124ac5e4478b7f1f728d6273b34
Jul 28 17:24:55.657007 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:55.656888    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5478597cd8-pw7rm" event={"ID":"af758b14-72fa-48eb-bb20-4dc7c85693b7","Type":"ContainerStarted","Data":"bf2079e52f48f0ac99c9e41132b16226f76f2e2d8de53e3eb82d04186a1d89a0"}
Jul 28 17:24:55.657007 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:55.656959    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5478597cd8-pw7rm" event={"ID":"af758b14-72fa-48eb-bb20-4dc7c85693b7","Type":"ContainerStarted","Data":"fbc3897408c03b0b49a53d0fa1c38b3b347325891e16c7a55b24af7c790d46fd"}
Jul 28 17:24:55.657007 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:55.656987    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5478597cd8-pw7rm" event={"ID":"af758b14-72fa-48eb-bb20-4dc7c85693b7","Type":"ContainerStarted","Data":"a1fce80c31fa3d063aa1eb7d0cab7621ea9228d9f05a509318a365ba40349da3"}
Jul 28 17:24:55.658558 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:55.658482    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f968994d7-4jr7x" event={"ID":"aa2cde03-a473-40f2-ac59-b383486afcc2","Type":"ContainerStarted","Data":"506f2779f24b3a900a6ac991790603820f502287a3940fa567ff8eba4103f443"}
Jul 28 17:24:55.658669 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:55.658576    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f968994d7-4jr7x" event={"ID":"aa2cde03-a473-40f2-ac59-b383486afcc2","Type":"ContainerStarted","Data":"c77f169fff69f2fdf233be1c1a3671299467f124ac5e4478b7f1f728d6273b34"}
Jul 28 17:24:55.682757 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:55.682699    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5478597cd8-pw7rm" podStartSLOduration=2.372938491 podStartE2EDuration="4.682677866s" totalImagesPullingTime="2.309739375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:55.681660327 +0000 UTC m=+99.087934068" watchObservedRunningTime="2026-07-28 17:24:55.682677866 +0000 UTC m=+99.088951607"
Jul 28 17:24:55.707051 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:55.707008    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f968994d7-4jr7x" podStartSLOduration=1.706997908 podStartE2EDuration="1.706997908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:55.705202071 +0000 UTC m=+99.111475812" watchObservedRunningTime="2026-07-28 17:24:55.706997908 +0000 UTC m=+99.113271648"
Jul 28 17:24:58.508269 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:58.508240    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9c8w8"
Jul 28 17:24:59.278874 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:59.278845    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f968994d7-4jr7x"]
Jul 28 17:24:59.317707 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:59.317673    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b6bb7dc96-cwkx6"]
Jul 28 17:24:59.350984 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:59.350952    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b6bb7dc96-cwkx6"]
Jul 28 17:24:59.351122 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:59.351071    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b6bb7dc96-cwkx6"
Jul 28 17:24:59.460316 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:59.460291    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-trusted-ca-bundle\") pod \"console-6b6bb7dc96-cwkx6\" (UID: \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\") " pod="openshift-console/console-6b6bb7dc96-cwkx6"
Jul 28 17:24:59.460476 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:59.460340    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-console-serving-cert\") pod \"console-6b6bb7dc96-cwkx6\" (UID: \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\") " pod="openshift-console/console-6b6bb7dc96-cwkx6"
Jul 28 17:24:59.460476 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:59.460407    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f9pc\" (UniqueName: \"kubernetes.io/projected/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-kube-api-access-9f9pc\") pod \"console-6b6bb7dc96-cwkx6\" (UID: \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\") " pod="openshift-console/console-6b6bb7dc96-cwkx6"
Jul 28 17:24:59.460476 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:59.460427    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-service-ca\") pod \"console-6b6bb7dc96-cwkx6\" (UID: \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\") " pod="openshift-console/console-6b6bb7dc96-cwkx6"
Jul 28 17:24:59.460600 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:59.460510    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-console-oauth-config\") pod \"console-6b6bb7dc96-cwkx6\" (UID: \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\") " pod="openshift-console/console-6b6bb7dc96-cwkx6"
Jul 28 17:24:59.460600 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:59.460536    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-console-config\") pod \"console-6b6bb7dc96-cwkx6\" (UID: \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\") " pod="openshift-console/console-6b6bb7dc96-cwkx6"
Jul 28 17:24:59.460600 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:59.460555    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-oauth-serving-cert\") pod \"console-6b6bb7dc96-cwkx6\" (UID: \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\") " pod="openshift-console/console-6b6bb7dc96-cwkx6"
Jul 28 17:24:59.561222 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:59.561119    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-console-serving-cert\") pod \"console-6b6bb7dc96-cwkx6\" (UID: \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\") " pod="openshift-console/console-6b6bb7dc96-cwkx6"
Jul 28 17:24:59.561703 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:59.561278    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f9pc\" (UniqueName: \"kubernetes.io/projected/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-kube-api-access-9f9pc\") pod \"console-6b6bb7dc96-cwkx6\" (UID: \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\") " pod="openshift-console/console-6b6bb7dc96-cwkx6"
Jul 28 17:24:59.561703 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:59.561306    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-service-ca\") pod \"console-6b6bb7dc96-cwkx6\" (UID: \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\") " pod="openshift-console/console-6b6bb7dc96-cwkx6"
Jul 28 17:24:59.561703 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:59.561338    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-console-oauth-config\") pod \"console-6b6bb7dc96-cwkx6\" (UID: \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\") " pod="openshift-console/console-6b6bb7dc96-cwkx6"
Jul 28 17:24:59.561703 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:59.561355    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-console-config\") pod \"console-6b6bb7dc96-cwkx6\" (UID: \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\") " pod="openshift-console/console-6b6bb7dc96-cwkx6"
Jul 28 17:24:59.561703 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:59.561496    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-oauth-serving-cert\") pod \"console-6b6bb7dc96-cwkx6\" (UID: \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\") " pod="openshift-console/console-6b6bb7dc96-cwkx6"
Jul 28 17:24:59.561703 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:59.561629    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-trusted-ca-bundle\") pod \"console-6b6bb7dc96-cwkx6\" (UID: \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\") " pod="openshift-console/console-6b6bb7dc96-cwkx6"
Jul 28 17:24:59.562250 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:59.562224    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-service-ca\") pod \"console-6b6bb7dc96-cwkx6\" (UID: \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\") " pod="openshift-console/console-6b6bb7dc96-cwkx6"
Jul 28 17:24:59.562353 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:59.562225    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-console-config\") pod \"console-6b6bb7dc96-cwkx6\" (UID: \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\") " pod="openshift-console/console-6b6bb7dc96-cwkx6"
Jul 28 17:24:59.562409 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:59.562399    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-oauth-serving-cert\") pod \"console-6b6bb7dc96-cwkx6\" (UID: \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\") " pod="openshift-console/console-6b6bb7dc96-cwkx6"
Jul 28 17:24:59.562471 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:59.562453    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-trusted-ca-bundle\") pod \"console-6b6bb7dc96-cwkx6\" (UID: \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\") " pod="openshift-console/console-6b6bb7dc96-cwkx6"
Jul 28 17:24:59.563721 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:59.563698    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-console-serving-cert\") pod \"console-6b6bb7dc96-cwkx6\" (UID: \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\") " pod="openshift-console/console-6b6bb7dc96-cwkx6"
Jul 28 17:24:59.563839 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:59.563735    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-console-oauth-config\") pod \"console-6b6bb7dc96-cwkx6\" (UID: \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\") " pod="openshift-console/console-6b6bb7dc96-cwkx6"
Jul 28 17:24:59.570648 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:59.570624    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f9pc\" (UniqueName: \"kubernetes.io/projected/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-kube-api-access-9f9pc\") pod \"console-6b6bb7dc96-cwkx6\" (UID: \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\") " pod="openshift-console/console-6b6bb7dc96-cwkx6"
Jul 28 17:24:59.660197 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:59.660161    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b6bb7dc96-cwkx6"
Jul 28 17:24:59.816919 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:24:59.816841    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b6bb7dc96-cwkx6"]
Jul 28 17:24:59.820488 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:24:59.820461    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f8f9d16_dbd2_4981_b459_217b8f46b5c6.slice/crio-4f0c226ead79d1b980096a5ba44d0e530759734f96479b7ea9b0d51056f53e8c WatchSource:0}: Error finding container 4f0c226ead79d1b980096a5ba44d0e530759734f96479b7ea9b0d51056f53e8c: Status 404 returned error can't find the container with id 4f0c226ead79d1b980096a5ba44d0e530759734f96479b7ea9b0d51056f53e8c
Jul 28 17:25:00.571407 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:00.571384    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5b7ff5c8fb-j995z"
Jul 28 17:25:00.678101 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:00.678069    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b6bb7dc96-cwkx6" event={"ID":"2f8f9d16-dbd2-4981-b459-217b8f46b5c6","Type":"ContainerStarted","Data":"19dbfede0d0144e2d9be9ae3348e12056dfb66fca1ec4a999330cfb231881ef6"}
Jul 28 17:25:00.678300 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:00.678123    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b6bb7dc96-cwkx6" event={"ID":"2f8f9d16-dbd2-4981-b459-217b8f46b5c6","Type":"ContainerStarted","Data":"4f0c226ead79d1b980096a5ba44d0e530759734f96479b7ea9b0d51056f53e8c"}
Jul 28 17:25:00.696791 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:00.696742    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b6bb7dc96-cwkx6" podStartSLOduration=1.696728206 podStartE2EDuration="1.696728206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:25:00.695835199 +0000 UTC m=+104.102108961" watchObservedRunningTime="2026-07-28 17:25:00.696728206 +0000 UTC m=+104.103001947"
Jul 28 17:25:04.526956 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:04.526913    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f968994d7-4jr7x"
Jul 28 17:25:09.661260 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:09.661230    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b6bb7dc96-cwkx6"
Jul 28 17:25:09.661773 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:09.661514    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b6bb7dc96-cwkx6"
Jul 28 17:25:09.666062 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:09.666038    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b6bb7dc96-cwkx6"
Jul 28 17:25:09.716987 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:09.716961    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b6bb7dc96-cwkx6"
Jul 28 17:25:09.765772 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:09.765736    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64cb7bf944-dh52l"]
Jul 28 17:25:11.305380 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:11.305347    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-789948b4bd-tbvt4"
Jul 28 17:25:11.305893 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:11.305409    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-789948b4bd-tbvt4"
Jul 28 17:25:24.297867 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:24.297802    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-f968994d7-4jr7x" podUID="aa2cde03-a473-40f2-ac59-b383486afcc2" containerName="console" containerID="cri-o://506f2779f24b3a900a6ac991790603820f502287a3940fa567ff8eba4103f443" gracePeriod=15
Jul 28 17:25:24.771612 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:24.771583    2543 generic.go:356] "Generic (PLEG): container finished" podID="aa2cde03-a473-40f2-ac59-b383486afcc2" containerID="506f2779f24b3a900a6ac991790603820f502287a3940fa567ff8eba4103f443" exitCode=0
Jul 28 17:25:24.771764 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:24.771664    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f968994d7-4jr7x" event={"ID":"aa2cde03-a473-40f2-ac59-b383486afcc2","Type":"ContainerDied","Data":"506f2779f24b3a900a6ac991790603820f502287a3940fa567ff8eba4103f443"}
Jul 28 17:25:25.052724 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:25.052699    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f968994d7-4jr7x"
Jul 28 17:25:25.184351 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:25.184278    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa2cde03-a473-40f2-ac59-b383486afcc2-console-config\") pod \"aa2cde03-a473-40f2-ac59-b383486afcc2\" (UID: \"aa2cde03-a473-40f2-ac59-b383486afcc2\") "
Jul 28 17:25:25.184351 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:25.184309    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa2cde03-a473-40f2-ac59-b383486afcc2-trusted-ca-bundle\") pod \"aa2cde03-a473-40f2-ac59-b383486afcc2\" (UID: \"aa2cde03-a473-40f2-ac59-b383486afcc2\") "
Jul 28 17:25:25.184351 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:25.184328    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa2cde03-a473-40f2-ac59-b383486afcc2-console-oauth-config\") pod \"aa2cde03-a473-40f2-ac59-b383486afcc2\" (UID: \"aa2cde03-a473-40f2-ac59-b383486afcc2\") "
Jul 28 17:25:25.184629 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:25.184361    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa2cde03-a473-40f2-ac59-b383486afcc2-service-ca\") pod \"aa2cde03-a473-40f2-ac59-b383486afcc2\" (UID: \"aa2cde03-a473-40f2-ac59-b383486afcc2\") "
Jul 28 17:25:25.184629 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:25.184486    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa2cde03-a473-40f2-ac59-b383486afcc2-oauth-serving-cert\") pod \"aa2cde03-a473-40f2-ac59-b383486afcc2\" (UID: \"aa2cde03-a473-40f2-ac59-b383486afcc2\") "
Jul 28 17:25:25.184629 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:25.184612    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa2cde03-a473-40f2-ac59-b383486afcc2-console-serving-cert\") pod \"aa2cde03-a473-40f2-ac59-b383486afcc2\" (UID: \"aa2cde03-a473-40f2-ac59-b383486afcc2\") "
Jul 28 17:25:25.184813 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:25.184651    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wvhs\" (UniqueName: \"kubernetes.io/projected/aa2cde03-a473-40f2-ac59-b383486afcc2-kube-api-access-8wvhs\") pod \"aa2cde03-a473-40f2-ac59-b383486afcc2\" (UID: \"aa2cde03-a473-40f2-ac59-b383486afcc2\") "
Jul 28 17:25:25.184813 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:25.184684    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa2cde03-a473-40f2-ac59-b383486afcc2-console-config" (OuterVolumeSpecName: "console-config") pod "aa2cde03-a473-40f2-ac59-b383486afcc2" (UID: "aa2cde03-a473-40f2-ac59-b383486afcc2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:25:25.184813 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:25.184696    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa2cde03-a473-40f2-ac59-b383486afcc2-service-ca" (OuterVolumeSpecName: "service-ca") pod "aa2cde03-a473-40f2-ac59-b383486afcc2" (UID: "aa2cde03-a473-40f2-ac59-b383486afcc2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:25:25.184813 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:25.184727    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa2cde03-a473-40f2-ac59-b383486afcc2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "aa2cde03-a473-40f2-ac59-b383486afcc2" (UID: "aa2cde03-a473-40f2-ac59-b383486afcc2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:25:25.184997 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:25.184963    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa2cde03-a473-40f2-ac59-b383486afcc2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "aa2cde03-a473-40f2-ac59-b383486afcc2" (UID: "aa2cde03-a473-40f2-ac59-b383486afcc2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:25:25.185086 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:25.185072    2543 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa2cde03-a473-40f2-ac59-b383486afcc2-oauth-serving-cert\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:25:25.185158 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:25.185089    2543 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa2cde03-a473-40f2-ac59-b383486afcc2-console-config\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:25:25.185158 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:25.185098    2543 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa2cde03-a473-40f2-ac59-b383486afcc2-trusted-ca-bundle\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:25:25.185158 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:25.185106    2543 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa2cde03-a473-40f2-ac59-b383486afcc2-service-ca\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:25:25.186618 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:25.186592    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa2cde03-a473-40f2-ac59-b383486afcc2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "aa2cde03-a473-40f2-ac59-b383486afcc2" (UID: "aa2cde03-a473-40f2-ac59-b383486afcc2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:25:25.186718 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:25.186695    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa2cde03-a473-40f2-ac59-b383486afcc2-kube-api-access-8wvhs" (OuterVolumeSpecName: "kube-api-access-8wvhs") pod "aa2cde03-a473-40f2-ac59-b383486afcc2" (UID: "aa2cde03-a473-40f2-ac59-b383486afcc2"). InnerVolumeSpecName "kube-api-access-8wvhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:25:25.186757 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:25.186714    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa2cde03-a473-40f2-ac59-b383486afcc2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "aa2cde03-a473-40f2-ac59-b383486afcc2" (UID: "aa2cde03-a473-40f2-ac59-b383486afcc2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:25:25.286277 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:25.286247    2543 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa2cde03-a473-40f2-ac59-b383486afcc2-console-serving-cert\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:25:25.286277 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:25.286273    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wvhs\" (UniqueName: \"kubernetes.io/projected/aa2cde03-a473-40f2-ac59-b383486afcc2-kube-api-access-8wvhs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:25:25.286277 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:25.286283    2543 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa2cde03-a473-40f2-ac59-b383486afcc2-console-oauth-config\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:25:25.776349 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:25.776319    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f968994d7-4jr7x"
Jul 28 17:25:25.776783 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:25.776351    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f968994d7-4jr7x" event={"ID":"aa2cde03-a473-40f2-ac59-b383486afcc2","Type":"ContainerDied","Data":"c77f169fff69f2fdf233be1c1a3671299467f124ac5e4478b7f1f728d6273b34"}
Jul 28 17:25:25.776783 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:25.776413    2543 scope.go:117] "RemoveContainer" containerID="506f2779f24b3a900a6ac991790603820f502287a3940fa567ff8eba4103f443"
Jul 28 17:25:25.797370 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:25.797347    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f968994d7-4jr7x"]
Jul 28 17:25:25.799459 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:25.799439    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f968994d7-4jr7x"]
Jul 28 17:25:25.891025 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:25.890987    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c1c169-0fd2-4e73-a497-0b69fe85d11f-metrics-certs\") pod \"network-metrics-daemon-sjtrc\" (UID: \"a2c1c169-0fd2-4e73-a497-0b69fe85d11f\") " pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:25:25.893247 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:25.893227    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c1c169-0fd2-4e73-a497-0b69fe85d11f-metrics-certs\") pod \"network-metrics-daemon-sjtrc\" (UID: \"a2c1c169-0fd2-4e73-a497-0b69fe85d11f\") " pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:25:25.965326 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:25.965299    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dpv4x\""
Jul 28 17:25:25.973181 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:25.973163    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjtrc"
Jul 28 17:25:26.096492 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:26.096463    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sjtrc"]
Jul 28 17:25:26.099847 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:25:26.099811    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c1c169_0fd2_4e73_a497_0b69fe85d11f.slice/crio-8be5019938756bdf1c5615edd45eae96d3e548cf3321e4084123dc175f25e294 WatchSource:0}: Error finding container 8be5019938756bdf1c5615edd45eae96d3e548cf3321e4084123dc175f25e294: Status 404 returned error can't find the container with id 8be5019938756bdf1c5615edd45eae96d3e548cf3321e4084123dc175f25e294
Jul 28 17:25:26.781900 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:26.781865    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sjtrc" event={"ID":"a2c1c169-0fd2-4e73-a497-0b69fe85d11f","Type":"ContainerStarted","Data":"8be5019938756bdf1c5615edd45eae96d3e548cf3321e4084123dc175f25e294"}
Jul 28 17:25:27.155824 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:27.155791    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa2cde03-a473-40f2-ac59-b383486afcc2" path="/var/lib/kubelet/pods/aa2cde03-a473-40f2-ac59-b383486afcc2/volumes"
Jul 28 17:25:27.788007 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:27.787968    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sjtrc" event={"ID":"a2c1c169-0fd2-4e73-a497-0b69fe85d11f","Type":"ContainerStarted","Data":"86c45f7b66481de6a798e5cafe7aa244009fbdf5297aea982b5b4d1ff550088b"}
Jul 28 17:25:27.788527 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:27.788037    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sjtrc" event={"ID":"a2c1c169-0fd2-4e73-a497-0b69fe85d11f","Type":"ContainerStarted","Data":"357b21bdac3146c86857b8bd9da26a345b63a47d7f8b98cd80d4de099ea8a7d7"}
Jul 28 17:25:27.811190 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:27.811111    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sjtrc" podStartSLOduration=129.992399639 podStartE2EDuration="2m10.81109636s" totalImagesPullingTime="818.696721ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:25:27.810404107 +0000 UTC m=+131.216677849" watchObservedRunningTime="2026-07-28 17:25:27.81109636 +0000 UTC m=+131.217370100"
Jul 28 17:25:30.803303 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:30.803272    2543 generic.go:356] "Generic (PLEG): container finished" podID="6a0600d3-cc7b-4caf-a420-e669a2c09715" containerID="d69676a8ccc2d6278cc9d7ff67da37c002053689f69106057312485c87932200" exitCode=0
Jul 28 17:25:30.803804 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:30.803317    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s" event={"ID":"6a0600d3-cc7b-4caf-a420-e669a2c09715","Type":"ContainerDied","Data":"d69676a8ccc2d6278cc9d7ff67da37c002053689f69106057312485c87932200"}
Jul 28 17:25:30.803804 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:30.803701    2543 scope.go:117] "RemoveContainer" containerID="d69676a8ccc2d6278cc9d7ff67da37c002053689f69106057312485c87932200"
Jul 28 17:25:31.310845 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:31.310817    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-789948b4bd-tbvt4"
Jul 28 17:25:31.315020 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:31.314998    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-789948b4bd-tbvt4"
Jul 28 17:25:31.809674 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:31.809637    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s" event={"ID":"6a0600d3-cc7b-4caf-a420-e669a2c09715","Type":"ContainerStarted","Data":"7e66acee229dc9e54a4da72363a7005c80d21333f313ff4fd4e9743cb257e440"}
Jul 28 17:25:34.786843 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:34.786800    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-64cb7bf944-dh52l" podUID="1980fe9a-8463-4a8d-8462-d591761ffdf1" containerName="console" containerID="cri-o://3cc87b01ebfb259ddb1ecc08115b21860780fd3948e65055eedfe36269cb43c3" gracePeriod=15
Jul 28 17:25:35.032940 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:35.032914    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64cb7bf944-dh52l"
Jul 28 17:25:35.075268 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:35.075200    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1980fe9a-8463-4a8d-8462-d591761ffdf1-console-oauth-config\") pod \"1980fe9a-8463-4a8d-8462-d591761ffdf1\" (UID: \"1980fe9a-8463-4a8d-8462-d591761ffdf1\") "
Jul 28 17:25:35.075268 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:35.075253    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1980fe9a-8463-4a8d-8462-d591761ffdf1-oauth-serving-cert\") pod \"1980fe9a-8463-4a8d-8462-d591761ffdf1\" (UID: \"1980fe9a-8463-4a8d-8462-d591761ffdf1\") "
Jul 28 17:25:35.075438 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:35.075280    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1980fe9a-8463-4a8d-8462-d591761ffdf1-service-ca\") pod \"1980fe9a-8463-4a8d-8462-d591761ffdf1\" (UID: \"1980fe9a-8463-4a8d-8462-d591761ffdf1\") "
Jul 28 17:25:35.075438 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:35.075319    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1980fe9a-8463-4a8d-8462-d591761ffdf1-console-serving-cert\") pod \"1980fe9a-8463-4a8d-8462-d591761ffdf1\" (UID: \"1980fe9a-8463-4a8d-8462-d591761ffdf1\") "
Jul 28 17:25:35.075438 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:35.075342    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6fq6\" (UniqueName: \"kubernetes.io/projected/1980fe9a-8463-4a8d-8462-d591761ffdf1-kube-api-access-f6fq6\") pod \"1980fe9a-8463-4a8d-8462-d591761ffdf1\" (UID: \"1980fe9a-8463-4a8d-8462-d591761ffdf1\") "
Jul 28 17:25:35.075583 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:35.075492    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1980fe9a-8463-4a8d-8462-d591761ffdf1-console-config\") pod \"1980fe9a-8463-4a8d-8462-d591761ffdf1\" (UID: \"1980fe9a-8463-4a8d-8462-d591761ffdf1\") "
Jul 28 17:25:35.075762 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:35.075724    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1980fe9a-8463-4a8d-8462-d591761ffdf1-service-ca" (OuterVolumeSpecName: "service-ca") pod "1980fe9a-8463-4a8d-8462-d591761ffdf1" (UID: "1980fe9a-8463-4a8d-8462-d591761ffdf1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:25:35.075762 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:35.075732    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1980fe9a-8463-4a8d-8462-d591761ffdf1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1980fe9a-8463-4a8d-8462-d591761ffdf1" (UID: "1980fe9a-8463-4a8d-8462-d591761ffdf1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:25:35.076075 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:35.076039    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1980fe9a-8463-4a8d-8462-d591761ffdf1-console-config" (OuterVolumeSpecName: "console-config") pod "1980fe9a-8463-4a8d-8462-d591761ffdf1" (UID: "1980fe9a-8463-4a8d-8462-d591761ffdf1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:25:35.077458 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:35.077425    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1980fe9a-8463-4a8d-8462-d591761ffdf1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1980fe9a-8463-4a8d-8462-d591761ffdf1" (UID: "1980fe9a-8463-4a8d-8462-d591761ffdf1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:25:35.077558 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:35.077481    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1980fe9a-8463-4a8d-8462-d591761ffdf1-kube-api-access-f6fq6" (OuterVolumeSpecName: "kube-api-access-f6fq6") pod "1980fe9a-8463-4a8d-8462-d591761ffdf1" (UID: "1980fe9a-8463-4a8d-8462-d591761ffdf1"). InnerVolumeSpecName "kube-api-access-f6fq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:25:35.077558 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:35.077521    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1980fe9a-8463-4a8d-8462-d591761ffdf1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1980fe9a-8463-4a8d-8462-d591761ffdf1" (UID: "1980fe9a-8463-4a8d-8462-d591761ffdf1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:25:35.176254 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:35.176227    2543 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1980fe9a-8463-4a8d-8462-d591761ffdf1-oauth-serving-cert\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:25:35.176254 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:35.176250    2543 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1980fe9a-8463-4a8d-8462-d591761ffdf1-service-ca\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:25:35.176422 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:35.176265    2543 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1980fe9a-8463-4a8d-8462-d591761ffdf1-console-serving-cert\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:25:35.176422 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:35.176277    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6fq6\" (UniqueName: \"kubernetes.io/projected/1980fe9a-8463-4a8d-8462-d591761ffdf1-kube-api-access-f6fq6\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:25:35.176422 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:35.176287    2543 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1980fe9a-8463-4a8d-8462-d591761ffdf1-console-config\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:25:35.176422 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:35.176296    2543 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1980fe9a-8463-4a8d-8462-d591761ffdf1-console-oauth-config\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:25:35.830185 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:35.830148    2543 generic.go:356] "Generic (PLEG): container finished" podID="1980fe9a-8463-4a8d-8462-d591761ffdf1" containerID="3cc87b01ebfb259ddb1ecc08115b21860780fd3948e65055eedfe36269cb43c3" exitCode=0
Jul 28 17:25:35.830683 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:35.830200    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64cb7bf944-dh52l"
Jul 28 17:25:35.830683 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:35.830233    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64cb7bf944-dh52l" event={"ID":"1980fe9a-8463-4a8d-8462-d591761ffdf1","Type":"ContainerDied","Data":"3cc87b01ebfb259ddb1ecc08115b21860780fd3948e65055eedfe36269cb43c3"}
Jul 28 17:25:35.830683 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:35.830300    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64cb7bf944-dh52l" event={"ID":"1980fe9a-8463-4a8d-8462-d591761ffdf1","Type":"ContainerDied","Data":"c7670277097614b4c3dc500d592d4010f3f034c733c4cb1eabe709add30dcc56"}
Jul 28 17:25:35.830683 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:35.830330    2543 scope.go:117] "RemoveContainer" containerID="3cc87b01ebfb259ddb1ecc08115b21860780fd3948e65055eedfe36269cb43c3"
Jul 28 17:25:35.840373 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:35.839593    2543 scope.go:117] "RemoveContainer" containerID="3cc87b01ebfb259ddb1ecc08115b21860780fd3948e65055eedfe36269cb43c3"
Jul 28 17:25:35.840526 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:25:35.840498    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cc87b01ebfb259ddb1ecc08115b21860780fd3948e65055eedfe36269cb43c3\": container with ID starting with 3cc87b01ebfb259ddb1ecc08115b21860780fd3948e65055eedfe36269cb43c3 not found: ID does not exist" containerID="3cc87b01ebfb259ddb1ecc08115b21860780fd3948e65055eedfe36269cb43c3"
Jul 28 17:25:35.840575 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:35.840542    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cc87b01ebfb259ddb1ecc08115b21860780fd3948e65055eedfe36269cb43c3"} err="failed to get container status \"3cc87b01ebfb259ddb1ecc08115b21860780fd3948e65055eedfe36269cb43c3\": rpc error: code = NotFound desc = could not find container \"3cc87b01ebfb259ddb1ecc08115b21860780fd3948e65055eedfe36269cb43c3\": container with ID starting with 3cc87b01ebfb259ddb1ecc08115b21860780fd3948e65055eedfe36269cb43c3 not found: ID does not exist"
Jul 28 17:25:35.847400 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:35.847372    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64cb7bf944-dh52l"]
Jul 28 17:25:35.849398 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:35.849376    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64cb7bf944-dh52l"]
Jul 28 17:25:37.151619 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:25:37.151589    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1980fe9a-8463-4a8d-8462-d591761ffdf1" path="/var/lib/kubelet/pods/1980fe9a-8463-4a8d-8462-d591761ffdf1/volumes"
Jul 28 17:26:07.146241 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:07.146200    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:26:07.148229 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:07.148197    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerName="alertmanager" containerID="cri-o://dbf37094182d57da4400a7496af54f1aa7fa812f5e2b99756b881a37bc5e38f8" gracePeriod=120
Jul 28 17:26:07.148388 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:07.148268    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerName="prom-label-proxy" containerID="cri-o://727b8557aa5f1d0822c70747fc22a374ca179be5ab4a71aae9f39f67722bf8f0" gracePeriod=120
Jul 28 17:26:07.148388 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:07.148296    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerName="kube-rbac-proxy-metric" containerID="cri-o://5656277fc9de31340e765938aae8d41f23f55a5172d8d10ecfc4176a11960441" gracePeriod=120
Jul 28 17:26:07.148388 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:07.148327    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerName="config-reloader" containerID="cri-o://44b7b9fda6e230b94b83a7716d7cf6c6c5cf3f4d60d3c715deb68776dd2a5ce3" gracePeriod=120
Jul 28 17:26:07.148554 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:07.148268    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerName="kube-rbac-proxy-web" containerID="cri-o://817a5b4466b951268542e3e29c449f6cc9db6c5ba3fac00f6301933402fbe14a" gracePeriod=120
Jul 28 17:26:07.148554 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:07.148272    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerName="kube-rbac-proxy" containerID="cri-o://c4966596b6f6772627239d5d4e6a1d3c8185b9b1f65be35258828dc8b1964250" gracePeriod=120
Jul 28 17:26:07.950300 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:07.950266    2543 generic.go:356] "Generic (PLEG): container finished" podID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerID="727b8557aa5f1d0822c70747fc22a374ca179be5ab4a71aae9f39f67722bf8f0" exitCode=0
Jul 28 17:26:07.950300 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:07.950292    2543 generic.go:356] "Generic (PLEG): container finished" podID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerID="c4966596b6f6772627239d5d4e6a1d3c8185b9b1f65be35258828dc8b1964250" exitCode=0
Jul 28 17:26:07.950300 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:07.950298    2543 generic.go:356] "Generic (PLEG): container finished" podID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerID="44b7b9fda6e230b94b83a7716d7cf6c6c5cf3f4d60d3c715deb68776dd2a5ce3" exitCode=0
Jul 28 17:26:07.950300 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:07.950304    2543 generic.go:356] "Generic (PLEG): container finished" podID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerID="dbf37094182d57da4400a7496af54f1aa7fa812f5e2b99756b881a37bc5e38f8" exitCode=0
Jul 28 17:26:07.950556 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:07.950337    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3933c0c1-cbfe-48ab-a824-02a85a224c7b","Type":"ContainerDied","Data":"727b8557aa5f1d0822c70747fc22a374ca179be5ab4a71aae9f39f67722bf8f0"}
Jul 28 17:26:07.950556 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:07.950442    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3933c0c1-cbfe-48ab-a824-02a85a224c7b","Type":"ContainerDied","Data":"c4966596b6f6772627239d5d4e6a1d3c8185b9b1f65be35258828dc8b1964250"}
Jul 28 17:26:07.950556 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:07.950470    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3933c0c1-cbfe-48ab-a824-02a85a224c7b","Type":"ContainerDied","Data":"44b7b9fda6e230b94b83a7716d7cf6c6c5cf3f4d60d3c715deb68776dd2a5ce3"}
Jul 28 17:26:07.950556 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:07.950494    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3933c0c1-cbfe-48ab-a824-02a85a224c7b","Type":"ContainerDied","Data":"dbf37094182d57da4400a7496af54f1aa7fa812f5e2b99756b881a37bc5e38f8"}
Jul 28 17:26:08.393550 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.393529    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:08.546387 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.546361    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3933c0c1-cbfe-48ab-a824-02a85a224c7b-metrics-client-ca\") pod \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") "
Jul 28 17:26:08.546549 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.546399    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") "
Jul 28 17:26:08.546549 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.546435    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8m9kk\" (UniqueName: \"kubernetes.io/projected/3933c0c1-cbfe-48ab-a824-02a85a224c7b-kube-api-access-8m9kk\") pod \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") "
Jul 28 17:26:08.546549 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.546461    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3933c0c1-cbfe-48ab-a824-02a85a224c7b-tls-assets\") pod \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") "
Jul 28 17:26:08.546549 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.546506    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3933c0c1-cbfe-48ab-a824-02a85a224c7b-alertmanager-main-db\") pod \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") "
Jul 28 17:26:08.546549 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.546530    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-web-config\") pod \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") "
Jul 28 17:26:08.546847 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.546563    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-cluster-tls-config\") pod \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") "
Jul 28 17:26:08.546847 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.546586    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-config-volume\") pod \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") "
Jul 28 17:26:08.546847 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.546631    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3933c0c1-cbfe-48ab-a824-02a85a224c7b-alertmanager-trusted-ca-bundle\") pod \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") "
Jul 28 17:26:08.546847 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.546662    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-secret-alertmanager-kube-rbac-proxy\") pod \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") "
Jul 28 17:26:08.546847 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.546713    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-secret-alertmanager-kube-rbac-proxy-web\") pod \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") "
Jul 28 17:26:08.546847 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.546753    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3933c0c1-cbfe-48ab-a824-02a85a224c7b-config-out\") pod \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") "
Jul 28 17:26:08.546847 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.546798    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-secret-alertmanager-main-tls\") pod \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\" (UID: \"3933c0c1-cbfe-48ab-a824-02a85a224c7b\") "
Jul 28 17:26:08.547220 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.546865    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3933c0c1-cbfe-48ab-a824-02a85a224c7b-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "3933c0c1-cbfe-48ab-a824-02a85a224c7b" (UID: "3933c0c1-cbfe-48ab-a824-02a85a224c7b"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:26:08.547220 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.547059    2543 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3933c0c1-cbfe-48ab-a824-02a85a224c7b-metrics-client-ca\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:26:08.547326 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.547286    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3933c0c1-cbfe-48ab-a824-02a85a224c7b-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "3933c0c1-cbfe-48ab-a824-02a85a224c7b" (UID: "3933c0c1-cbfe-48ab-a824-02a85a224c7b"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:26:08.548214 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.548183    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3933c0c1-cbfe-48ab-a824-02a85a224c7b-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "3933c0c1-cbfe-48ab-a824-02a85a224c7b" (UID: "3933c0c1-cbfe-48ab-a824-02a85a224c7b"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:26:08.549639 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.549611    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "3933c0c1-cbfe-48ab-a824-02a85a224c7b" (UID: "3933c0c1-cbfe-48ab-a824-02a85a224c7b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:26:08.549778 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.549750    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3933c0c1-cbfe-48ab-a824-02a85a224c7b-kube-api-access-8m9kk" (OuterVolumeSpecName: "kube-api-access-8m9kk") pod "3933c0c1-cbfe-48ab-a824-02a85a224c7b" (UID: "3933c0c1-cbfe-48ab-a824-02a85a224c7b"). InnerVolumeSpecName "kube-api-access-8m9kk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:26:08.550260 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.550207    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "3933c0c1-cbfe-48ab-a824-02a85a224c7b" (UID: "3933c0c1-cbfe-48ab-a824-02a85a224c7b"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:26:08.550464 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.550429    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3933c0c1-cbfe-48ab-a824-02a85a224c7b-config-out" (OuterVolumeSpecName: "config-out") pod "3933c0c1-cbfe-48ab-a824-02a85a224c7b" (UID: "3933c0c1-cbfe-48ab-a824-02a85a224c7b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:26:08.550574 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.550508    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "3933c0c1-cbfe-48ab-a824-02a85a224c7b" (UID: "3933c0c1-cbfe-48ab-a824-02a85a224c7b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:26:08.550746 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.550720    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "3933c0c1-cbfe-48ab-a824-02a85a224c7b" (UID: "3933c0c1-cbfe-48ab-a824-02a85a224c7b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:26:08.550849 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.550813    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3933c0c1-cbfe-48ab-a824-02a85a224c7b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "3933c0c1-cbfe-48ab-a824-02a85a224c7b" (UID: "3933c0c1-cbfe-48ab-a824-02a85a224c7b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:26:08.551122 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.551105    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-config-volume" (OuterVolumeSpecName: "config-volume") pod "3933c0c1-cbfe-48ab-a824-02a85a224c7b" (UID: "3933c0c1-cbfe-48ab-a824-02a85a224c7b"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:26:08.553697 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.553666    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "3933c0c1-cbfe-48ab-a824-02a85a224c7b" (UID: "3933c0c1-cbfe-48ab-a824-02a85a224c7b"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:26:08.560292 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.560272    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-web-config" (OuterVolumeSpecName: "web-config") pod "3933c0c1-cbfe-48ab-a824-02a85a224c7b" (UID: "3933c0c1-cbfe-48ab-a824-02a85a224c7b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:26:08.648107 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.648079    2543 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:26:08.648107 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.648104    2543 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3933c0c1-cbfe-48ab-a824-02a85a224c7b-config-out\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:26:08.648107 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.648114    2543 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-secret-alertmanager-main-tls\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:26:08.648299 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.648124    2543 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:26:08.648299 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.648150    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m9kk\" (UniqueName: \"kubernetes.io/projected/3933c0c1-cbfe-48ab-a824-02a85a224c7b-kube-api-access-8m9kk\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:26:08.648299 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.648159    2543 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3933c0c1-cbfe-48ab-a824-02a85a224c7b-tls-assets\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:26:08.648299 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.648168    2543 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3933c0c1-cbfe-48ab-a824-02a85a224c7b-alertmanager-main-db\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:26:08.648299 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.648176    2543 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-web-config\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:26:08.648299 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.648183    2543 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-cluster-tls-config\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:26:08.648299 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.648190    2543 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-config-volume\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:26:08.648299 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.648198    2543 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3933c0c1-cbfe-48ab-a824-02a85a224c7b-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:26:08.648299 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.648207    2543 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3933c0c1-cbfe-48ab-a824-02a85a224c7b-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:26:08.956960 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.956858    2543 generic.go:356] "Generic (PLEG): container finished" podID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerID="5656277fc9de31340e765938aae8d41f23f55a5172d8d10ecfc4176a11960441" exitCode=0
Jul 28 17:26:08.956960 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.956884    2543 generic.go:356] "Generic (PLEG): container finished" podID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerID="817a5b4466b951268542e3e29c449f6cc9db6c5ba3fac00f6301933402fbe14a" exitCode=0
Jul 28 17:26:08.956960 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.956926    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:08.956960 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.956947    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3933c0c1-cbfe-48ab-a824-02a85a224c7b","Type":"ContainerDied","Data":"5656277fc9de31340e765938aae8d41f23f55a5172d8d10ecfc4176a11960441"}
Jul 28 17:26:08.957196 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.957019    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3933c0c1-cbfe-48ab-a824-02a85a224c7b","Type":"ContainerDied","Data":"817a5b4466b951268542e3e29c449f6cc9db6c5ba3fac00f6301933402fbe14a"}
Jul 28 17:26:08.957196 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.957048    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3933c0c1-cbfe-48ab-a824-02a85a224c7b","Type":"ContainerDied","Data":"0714f42cfc10f7baf5669d38ddd09a58d04b986a8bdad16660c7d5d5364bff56"}
Jul 28 17:26:08.957196 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.957087    2543 scope.go:117] "RemoveContainer" containerID="727b8557aa5f1d0822c70747fc22a374ca179be5ab4a71aae9f39f67722bf8f0"
Jul 28 17:26:08.965268 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.965252    2543 scope.go:117] "RemoveContainer" containerID="5656277fc9de31340e765938aae8d41f23f55a5172d8d10ecfc4176a11960441"
Jul 28 17:26:08.971964 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.971946    2543 scope.go:117] "RemoveContainer" containerID="c4966596b6f6772627239d5d4e6a1d3c8185b9b1f65be35258828dc8b1964250"
Jul 28 17:26:08.978654 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.978637    2543 scope.go:117] "RemoveContainer" containerID="817a5b4466b951268542e3e29c449f6cc9db6c5ba3fac00f6301933402fbe14a"
Jul 28 17:26:08.984885 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.984872    2543 scope.go:117] "RemoveContainer" containerID="44b7b9fda6e230b94b83a7716d7cf6c6c5cf3f4d60d3c715deb68776dd2a5ce3"
Jul 28 17:26:08.991408 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.991383    2543 scope.go:117] "RemoveContainer" containerID="dbf37094182d57da4400a7496af54f1aa7fa812f5e2b99756b881a37bc5e38f8"
Jul 28 17:26:08.997541 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.997522    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:26:08.998551 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:08.998531    2543 scope.go:117] "RemoveContainer" containerID="b7205f86691ca526181d7d26ff9ef264c63e83173ad858d3583637b73f2f02ca"
Jul 28 17:26:09.006150 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.005109    2543 scope.go:117] "RemoveContainer" containerID="727b8557aa5f1d0822c70747fc22a374ca179be5ab4a71aae9f39f67722bf8f0"
Jul 28 17:26:09.006350 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:26:09.006331    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"727b8557aa5f1d0822c70747fc22a374ca179be5ab4a71aae9f39f67722bf8f0\": container with ID starting with 727b8557aa5f1d0822c70747fc22a374ca179be5ab4a71aae9f39f67722bf8f0 not found: ID does not exist" containerID="727b8557aa5f1d0822c70747fc22a374ca179be5ab4a71aae9f39f67722bf8f0"
Jul 28 17:26:09.006411 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.006358    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727b8557aa5f1d0822c70747fc22a374ca179be5ab4a71aae9f39f67722bf8f0"} err="failed to get container status \"727b8557aa5f1d0822c70747fc22a374ca179be5ab4a71aae9f39f67722bf8f0\": rpc error: code = NotFound desc = could not find container \"727b8557aa5f1d0822c70747fc22a374ca179be5ab4a71aae9f39f67722bf8f0\": container with ID starting with 727b8557aa5f1d0822c70747fc22a374ca179be5ab4a71aae9f39f67722bf8f0 not found: ID does not exist"
Jul 28 17:26:09.006411 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.006378    2543 scope.go:117] "RemoveContainer" containerID="5656277fc9de31340e765938aae8d41f23f55a5172d8d10ecfc4176a11960441"
Jul 28 17:26:09.006651 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:26:09.006634    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5656277fc9de31340e765938aae8d41f23f55a5172d8d10ecfc4176a11960441\": container with ID starting with 5656277fc9de31340e765938aae8d41f23f55a5172d8d10ecfc4176a11960441 not found: ID does not exist" containerID="5656277fc9de31340e765938aae8d41f23f55a5172d8d10ecfc4176a11960441"
Jul 28 17:26:09.006700 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.006659    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5656277fc9de31340e765938aae8d41f23f55a5172d8d10ecfc4176a11960441"} err="failed to get container status \"5656277fc9de31340e765938aae8d41f23f55a5172d8d10ecfc4176a11960441\": rpc error: code = NotFound desc = could not find container \"5656277fc9de31340e765938aae8d41f23f55a5172d8d10ecfc4176a11960441\": container with ID starting with 5656277fc9de31340e765938aae8d41f23f55a5172d8d10ecfc4176a11960441 not found: ID does not exist"
Jul 28 17:26:09.006700 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.006675    2543 scope.go:117] "RemoveContainer" containerID="c4966596b6f6772627239d5d4e6a1d3c8185b9b1f65be35258828dc8b1964250"
Jul 28 17:26:09.006905 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:26:09.006886    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4966596b6f6772627239d5d4e6a1d3c8185b9b1f65be35258828dc8b1964250\": container with ID starting with c4966596b6f6772627239d5d4e6a1d3c8185b9b1f65be35258828dc8b1964250 not found: ID does not exist" containerID="c4966596b6f6772627239d5d4e6a1d3c8185b9b1f65be35258828dc8b1964250"
Jul 28 17:26:09.006987 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.006915    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4966596b6f6772627239d5d4e6a1d3c8185b9b1f65be35258828dc8b1964250"} err="failed to get container status \"c4966596b6f6772627239d5d4e6a1d3c8185b9b1f65be35258828dc8b1964250\": rpc error: code = NotFound desc = could not find container \"c4966596b6f6772627239d5d4e6a1d3c8185b9b1f65be35258828dc8b1964250\": container with ID starting with c4966596b6f6772627239d5d4e6a1d3c8185b9b1f65be35258828dc8b1964250 not found: ID does not exist"
Jul 28 17:26:09.006987 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.006943    2543 scope.go:117] "RemoveContainer" containerID="817a5b4466b951268542e3e29c449f6cc9db6c5ba3fac00f6301933402fbe14a"
Jul 28 17:26:09.007216 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:26:09.007192    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"817a5b4466b951268542e3e29c449f6cc9db6c5ba3fac00f6301933402fbe14a\": container with ID starting with 817a5b4466b951268542e3e29c449f6cc9db6c5ba3fac00f6301933402fbe14a not found: ID does not exist" containerID="817a5b4466b951268542e3e29c449f6cc9db6c5ba3fac00f6301933402fbe14a"
Jul 28 17:26:09.007281 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.007223    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"817a5b4466b951268542e3e29c449f6cc9db6c5ba3fac00f6301933402fbe14a"} err="failed to get container status \"817a5b4466b951268542e3e29c449f6cc9db6c5ba3fac00f6301933402fbe14a\": rpc error: code = NotFound desc = could not find container \"817a5b4466b951268542e3e29c449f6cc9db6c5ba3fac00f6301933402fbe14a\": container with ID starting with 817a5b4466b951268542e3e29c449f6cc9db6c5ba3fac00f6301933402fbe14a not found: ID does not exist"
Jul 28 17:26:09.007281 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.007239    2543 scope.go:117] "RemoveContainer" containerID="44b7b9fda6e230b94b83a7716d7cf6c6c5cf3f4d60d3c715deb68776dd2a5ce3"
Jul 28 17:26:09.007504 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.007484    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:26:09.007550 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:26:09.007492    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44b7b9fda6e230b94b83a7716d7cf6c6c5cf3f4d60d3c715deb68776dd2a5ce3\": container with ID starting with 44b7b9fda6e230b94b83a7716d7cf6c6c5cf3f4d60d3c715deb68776dd2a5ce3 not found: ID does not exist" containerID="44b7b9fda6e230b94b83a7716d7cf6c6c5cf3f4d60d3c715deb68776dd2a5ce3"
Jul 28 17:26:09.007586 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.007543    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44b7b9fda6e230b94b83a7716d7cf6c6c5cf3f4d60d3c715deb68776dd2a5ce3"} err="failed to get container status \"44b7b9fda6e230b94b83a7716d7cf6c6c5cf3f4d60d3c715deb68776dd2a5ce3\": rpc error: code = NotFound desc = could not find container \"44b7b9fda6e230b94b83a7716d7cf6c6c5cf3f4d60d3c715deb68776dd2a5ce3\": container with ID starting with 44b7b9fda6e230b94b83a7716d7cf6c6c5cf3f4d60d3c715deb68776dd2a5ce3 not found: ID does not exist"
Jul 28 17:26:09.007586 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.007558    2543 scope.go:117] "RemoveContainer" containerID="dbf37094182d57da4400a7496af54f1aa7fa812f5e2b99756b881a37bc5e38f8"
Jul 28 17:26:09.007801 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:26:09.007785    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbf37094182d57da4400a7496af54f1aa7fa812f5e2b99756b881a37bc5e38f8\": container with ID starting with dbf37094182d57da4400a7496af54f1aa7fa812f5e2b99756b881a37bc5e38f8 not found: ID does not exist" containerID="dbf37094182d57da4400a7496af54f1aa7fa812f5e2b99756b881a37bc5e38f8"
Jul 28 17:26:09.007852 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.007804    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbf37094182d57da4400a7496af54f1aa7fa812f5e2b99756b881a37bc5e38f8"} err="failed to get container status \"dbf37094182d57da4400a7496af54f1aa7fa812f5e2b99756b881a37bc5e38f8\": rpc error: code = NotFound desc = could not find container \"dbf37094182d57da4400a7496af54f1aa7fa812f5e2b99756b881a37bc5e38f8\": container with ID starting with dbf37094182d57da4400a7496af54f1aa7fa812f5e2b99756b881a37bc5e38f8 not found: ID does not exist"
Jul 28 17:26:09.007852 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.007817    2543 scope.go:117] "RemoveContainer" containerID="b7205f86691ca526181d7d26ff9ef264c63e83173ad858d3583637b73f2f02ca"
Jul 28 17:26:09.008048 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:26:09.008032    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7205f86691ca526181d7d26ff9ef264c63e83173ad858d3583637b73f2f02ca\": container with ID starting with b7205f86691ca526181d7d26ff9ef264c63e83173ad858d3583637b73f2f02ca not found: ID does not exist" containerID="b7205f86691ca526181d7d26ff9ef264c63e83173ad858d3583637b73f2f02ca"
Jul 28 17:26:09.008087 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.008052    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7205f86691ca526181d7d26ff9ef264c63e83173ad858d3583637b73f2f02ca"} err="failed to get container status \"b7205f86691ca526181d7d26ff9ef264c63e83173ad858d3583637b73f2f02ca\": rpc error: code = NotFound desc = could not find container \"b7205f86691ca526181d7d26ff9ef264c63e83173ad858d3583637b73f2f02ca\": container with ID starting with b7205f86691ca526181d7d26ff9ef264c63e83173ad858d3583637b73f2f02ca not found: ID does not exist"
Jul 28 17:26:09.008087 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.008071    2543 scope.go:117] "RemoveContainer" containerID="727b8557aa5f1d0822c70747fc22a374ca179be5ab4a71aae9f39f67722bf8f0"
Jul 28 17:26:09.008276 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.008262    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727b8557aa5f1d0822c70747fc22a374ca179be5ab4a71aae9f39f67722bf8f0"} err="failed to get container status \"727b8557aa5f1d0822c70747fc22a374ca179be5ab4a71aae9f39f67722bf8f0\": rpc error: code = NotFound desc = could not find container \"727b8557aa5f1d0822c70747fc22a374ca179be5ab4a71aae9f39f67722bf8f0\": container with ID starting with 727b8557aa5f1d0822c70747fc22a374ca179be5ab4a71aae9f39f67722bf8f0 not found: ID does not exist"
Jul 28 17:26:09.008326 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.008276    2543 scope.go:117] "RemoveContainer" containerID="5656277fc9de31340e765938aae8d41f23f55a5172d8d10ecfc4176a11960441"
Jul 28 17:26:09.008479 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.008460    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5656277fc9de31340e765938aae8d41f23f55a5172d8d10ecfc4176a11960441"} err="failed to get container status \"5656277fc9de31340e765938aae8d41f23f55a5172d8d10ecfc4176a11960441\": rpc error: code = NotFound desc = could not find container \"5656277fc9de31340e765938aae8d41f23f55a5172d8d10ecfc4176a11960441\": container with ID starting with 5656277fc9de31340e765938aae8d41f23f55a5172d8d10ecfc4176a11960441 not found: ID does not exist"
Jul 28 17:26:09.008520 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.008479    2543 scope.go:117] "RemoveContainer" containerID="c4966596b6f6772627239d5d4e6a1d3c8185b9b1f65be35258828dc8b1964250"
Jul 28 17:26:09.008700 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.008679    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4966596b6f6772627239d5d4e6a1d3c8185b9b1f65be35258828dc8b1964250"} err="failed to get container status \"c4966596b6f6772627239d5d4e6a1d3c8185b9b1f65be35258828dc8b1964250\": rpc error: code = NotFound desc = could not find container \"c4966596b6f6772627239d5d4e6a1d3c8185b9b1f65be35258828dc8b1964250\": container with ID starting with c4966596b6f6772627239d5d4e6a1d3c8185b9b1f65be35258828dc8b1964250 not found: ID does not exist"
Jul 28 17:26:09.008753 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.008700    2543 scope.go:117] "RemoveContainer" containerID="817a5b4466b951268542e3e29c449f6cc9db6c5ba3fac00f6301933402fbe14a"
Jul 28 17:26:09.008909 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.008894    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"817a5b4466b951268542e3e29c449f6cc9db6c5ba3fac00f6301933402fbe14a"} err="failed to get container status \"817a5b4466b951268542e3e29c449f6cc9db6c5ba3fac00f6301933402fbe14a\": rpc error: code = NotFound desc = could not find container \"817a5b4466b951268542e3e29c449f6cc9db6c5ba3fac00f6301933402fbe14a\": container with ID starting with 817a5b4466b951268542e3e29c449f6cc9db6c5ba3fac00f6301933402fbe14a not found: ID does not exist"
Jul 28 17:26:09.008949 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.008908    2543 scope.go:117] "RemoveContainer" containerID="44b7b9fda6e230b94b83a7716d7cf6c6c5cf3f4d60d3c715deb68776dd2a5ce3"
Jul 28 17:26:09.009113 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.009093    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44b7b9fda6e230b94b83a7716d7cf6c6c5cf3f4d60d3c715deb68776dd2a5ce3"} err="failed to get container status \"44b7b9fda6e230b94b83a7716d7cf6c6c5cf3f4d60d3c715deb68776dd2a5ce3\": rpc error: code = NotFound desc = could not find container \"44b7b9fda6e230b94b83a7716d7cf6c6c5cf3f4d60d3c715deb68776dd2a5ce3\": container with ID starting with 44b7b9fda6e230b94b83a7716d7cf6c6c5cf3f4d60d3c715deb68776dd2a5ce3 not found: ID does not exist"
Jul 28 17:26:09.009186 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.009114    2543 scope.go:117] "RemoveContainer" containerID="dbf37094182d57da4400a7496af54f1aa7fa812f5e2b99756b881a37bc5e38f8"
Jul 28 17:26:09.009353 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.009334    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbf37094182d57da4400a7496af54f1aa7fa812f5e2b99756b881a37bc5e38f8"} err="failed to get container status \"dbf37094182d57da4400a7496af54f1aa7fa812f5e2b99756b881a37bc5e38f8\": rpc error: code = NotFound desc = could not find container \"dbf37094182d57da4400a7496af54f1aa7fa812f5e2b99756b881a37bc5e38f8\": container with ID starting with dbf37094182d57da4400a7496af54f1aa7fa812f5e2b99756b881a37bc5e38f8 not found: ID does not exist"
Jul 28 17:26:09.009403 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.009353    2543 scope.go:117] "RemoveContainer" containerID="b7205f86691ca526181d7d26ff9ef264c63e83173ad858d3583637b73f2f02ca"
Jul 28 17:26:09.009584 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.009563    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7205f86691ca526181d7d26ff9ef264c63e83173ad858d3583637b73f2f02ca"} err="failed to get container status \"b7205f86691ca526181d7d26ff9ef264c63e83173ad858d3583637b73f2f02ca\": rpc error: code = NotFound desc = could not find container \"b7205f86691ca526181d7d26ff9ef264c63e83173ad858d3583637b73f2f02ca\": container with ID starting with b7205f86691ca526181d7d26ff9ef264c63e83173ad858d3583637b73f2f02ca not found: ID does not exist"
Jul 28 17:26:09.084008 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.083981    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:26:09.084342 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.084320    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerName="kube-rbac-proxy"
Jul 28 17:26:09.084342 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.084342    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerName="kube-rbac-proxy"
Jul 28 17:26:09.084489 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.084361    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa2cde03-a473-40f2-ac59-b383486afcc2" containerName="console"
Jul 28 17:26:09.084489 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.084369    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa2cde03-a473-40f2-ac59-b383486afcc2" containerName="console"
Jul 28 17:26:09.084489 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.084384    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerName="alertmanager"
Jul 28 17:26:09.084489 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.084392    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerName="alertmanager"
Jul 28 17:26:09.084489 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.084401    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerName="config-reloader"
Jul 28 17:26:09.084489 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.084410    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerName="config-reloader"
Jul 28 17:26:09.084489 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.084422    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerName="init-config-reloader"
Jul 28 17:26:09.084489 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.084431    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerName="init-config-reloader"
Jul 28 17:26:09.084489 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.084443    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerName="prom-label-proxy"
Jul 28 17:26:09.084489 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.084454    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerName="prom-label-proxy"
Jul 28 17:26:09.084489 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.084467    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerName="kube-rbac-proxy-web"
Jul 28 17:26:09.084489 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.084476    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerName="kube-rbac-proxy-web"
Jul 28 17:26:09.084489 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.084493    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1980fe9a-8463-4a8d-8462-d591761ffdf1" containerName="console"
Jul 28 17:26:09.085081 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.084501    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="1980fe9a-8463-4a8d-8462-d591761ffdf1" containerName="console"
Jul 28 17:26:09.085081 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.084513    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerName="kube-rbac-proxy-metric"
Jul 28 17:26:09.085081 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.084523    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerName="kube-rbac-proxy-metric"
Jul 28 17:26:09.085081 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.084595    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerName="prom-label-proxy"
Jul 28 17:26:09.085081 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.084608    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="1980fe9a-8463-4a8d-8462-d591761ffdf1" containerName="console"
Jul 28 17:26:09.085081 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.084617    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa2cde03-a473-40f2-ac59-b383486afcc2" containerName="console"
Jul 28 17:26:09.085081 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.084628    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerName="alertmanager"
Jul 28 17:26:09.085081 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.084636    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerName="kube-rbac-proxy-web"
Jul 28 17:26:09.085081 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.084646    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerName="kube-rbac-proxy"
Jul 28 17:26:09.085081 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.084653    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerName="kube-rbac-proxy-metric"
Jul 28 17:26:09.085081 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.084663    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" containerName="config-reloader"
Jul 28 17:26:09.090094 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.090073    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.097420 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.097404    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 17:26:09.098484 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.098466    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8prvr\""
Jul 28 17:26:09.098538 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.098502    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 17:26:09.100218 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.100201    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 17:26:09.100282 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.100230    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 17:26:09.100871 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.100855    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 17:26:09.100909 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.100878    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 17:26:09.100990 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.100976    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 17:26:09.106730 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.106712    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 17:26:09.108669 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.108657    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 17:26:09.120075 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.120050    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:26:09.151093 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.151067    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3933c0c1-cbfe-48ab-a824-02a85a224c7b" path="/var/lib/kubelet/pods/3933c0c1-cbfe-48ab-a824-02a85a224c7b/volumes"
Jul 28 17:26:09.253663 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.253629    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/573ee9e7-316a-4218-bafe-70345e39c3bd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.253807 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.253686    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/573ee9e7-316a-4218-bafe-70345e39c3bd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.253807 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.253725    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/573ee9e7-316a-4218-bafe-70345e39c3bd-config-volume\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.253807 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.253748    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/573ee9e7-316a-4218-bafe-70345e39c3bd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.253807 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.253793    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/573ee9e7-316a-4218-bafe-70345e39c3bd-web-config\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.254011 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.253820    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/573ee9e7-316a-4218-bafe-70345e39c3bd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.254011 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.253855    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/573ee9e7-316a-4218-bafe-70345e39c3bd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.254011 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.253880    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/573ee9e7-316a-4218-bafe-70345e39c3bd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.254011 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.253913    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/573ee9e7-316a-4218-bafe-70345e39c3bd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.254011 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.253938    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/573ee9e7-316a-4218-bafe-70345e39c3bd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.254011 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.254004    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/573ee9e7-316a-4218-bafe-70345e39c3bd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.254291 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.254028    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk5c5\" (UniqueName: \"kubernetes.io/projected/573ee9e7-316a-4218-bafe-70345e39c3bd-kube-api-access-pk5c5\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.254291 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.254089    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/573ee9e7-316a-4218-bafe-70345e39c3bd-config-out\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.354651 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.354617    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/573ee9e7-316a-4218-bafe-70345e39c3bd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.354813 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.354669    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/573ee9e7-316a-4218-bafe-70345e39c3bd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.354813 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.354735    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/573ee9e7-316a-4218-bafe-70345e39c3bd-config-volume\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.354813 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.354762    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/573ee9e7-316a-4218-bafe-70345e39c3bd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.354813 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.354791    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/573ee9e7-316a-4218-bafe-70345e39c3bd-web-config\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.355015 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.354820    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/573ee9e7-316a-4218-bafe-70345e39c3bd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.355015 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.354853    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/573ee9e7-316a-4218-bafe-70345e39c3bd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.355015 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.354875    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/573ee9e7-316a-4218-bafe-70345e39c3bd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.355015 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.354912    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/573ee9e7-316a-4218-bafe-70345e39c3bd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.355015 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.354935    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/573ee9e7-316a-4218-bafe-70345e39c3bd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.355015 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.354970    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/573ee9e7-316a-4218-bafe-70345e39c3bd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.355015 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.354994    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk5c5\" (UniqueName: \"kubernetes.io/projected/573ee9e7-316a-4218-bafe-70345e39c3bd-kube-api-access-pk5c5\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.355376 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.355035    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/573ee9e7-316a-4218-bafe-70345e39c3bd-config-out\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.355376 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.355253    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/573ee9e7-316a-4218-bafe-70345e39c3bd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.355861 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.355837    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/573ee9e7-316a-4218-bafe-70345e39c3bd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.357708 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.357676    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/573ee9e7-316a-4218-bafe-70345e39c3bd-config-out\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.358080 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.357829    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/573ee9e7-316a-4218-bafe-70345e39c3bd-web-config\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.358080 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.357829    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/573ee9e7-316a-4218-bafe-70345e39c3bd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.358080 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.357923    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/573ee9e7-316a-4218-bafe-70345e39c3bd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.358080 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.357956    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/573ee9e7-316a-4218-bafe-70345e39c3bd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.358395 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.358374    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/573ee9e7-316a-4218-bafe-70345e39c3bd-config-volume\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.358452 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.358443    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/573ee9e7-316a-4218-bafe-70345e39c3bd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.358507 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.358453    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/573ee9e7-316a-4218-bafe-70345e39c3bd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.358556 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.358538    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/573ee9e7-316a-4218-bafe-70345e39c3bd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.359803 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.359782    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/573ee9e7-316a-4218-bafe-70345e39c3bd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.367414 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.367395    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk5c5\" (UniqueName: \"kubernetes.io/projected/573ee9e7-316a-4218-bafe-70345e39c3bd-kube-api-access-pk5c5\") pod \"alertmanager-main-0\" (UID: \"573ee9e7-316a-4218-bafe-70345e39c3bd\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.398393 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.398357    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:26:09.549649 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.549615    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:26:09.962743 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.962710    2543 generic.go:356] "Generic (PLEG): container finished" podID="573ee9e7-316a-4218-bafe-70345e39c3bd" containerID="918826161c5d9bda5162661027539271f9bec192dc2eaacbdedd2ca575a5fd9d" exitCode=0
Jul 28 17:26:09.962921 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.962768    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"573ee9e7-316a-4218-bafe-70345e39c3bd","Type":"ContainerDied","Data":"918826161c5d9bda5162661027539271f9bec192dc2eaacbdedd2ca575a5fd9d"}
Jul 28 17:26:09.962921 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:09.962848    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"573ee9e7-316a-4218-bafe-70345e39c3bd","Type":"ContainerStarted","Data":"d1ef83b3ba1c7e2d7269f8c9a5710b1685206675587c4ad31bff8a255b6b83fb"}
Jul 28 17:26:10.969342 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:10.969303    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"573ee9e7-316a-4218-bafe-70345e39c3bd","Type":"ContainerStarted","Data":"8f40b95ae592b60e14cd2b9aebe7443ccd1837d255fac863e143cddfef664c15"}
Jul 28 17:26:10.969724 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:10.969389    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"573ee9e7-316a-4218-bafe-70345e39c3bd","Type":"ContainerStarted","Data":"e04c09d6cdcd795e7abe6aff9e6ae14c2928008bcbb677ec7630453b9999d21b"}
Jul 28 17:26:10.969724 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:10.969433    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"573ee9e7-316a-4218-bafe-70345e39c3bd","Type":"ContainerStarted","Data":"63dcac30651cf3849e734e4cf3c0f3fda6f58efa2163f31d689f356755c029ff"}
Jul 28 17:26:10.969724 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:10.969469    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"573ee9e7-316a-4218-bafe-70345e39c3bd","Type":"ContainerStarted","Data":"4331b7054363bee30a9a7604e10ac46af750ca67f26a8b872ba24175a782fbde"}
Jul 28 17:26:10.969724 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:10.969504    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"573ee9e7-316a-4218-bafe-70345e39c3bd","Type":"ContainerStarted","Data":"92d76c3d3c967d720d46c9d3694466b4385fd63eae2ac668821b12acef518134"}
Jul 28 17:26:10.969724 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:10.969542    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"573ee9e7-316a-4218-bafe-70345e39c3bd","Type":"ContainerStarted","Data":"ca2ae5d5680930a39d707eb071d47da918f55fed0a254b7526616aba940f7d17"}
Jul 28 17:26:11.001567 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:11.001486    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.9837454559999999 podStartE2EDuration="2.001468926s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.72347ms" isStatefulPod=false podCreationTimestamp="2026-07-28 17:26:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:26:10.999921635 +0000 UTC m=+174.406195377" watchObservedRunningTime="2026-07-28 17:26:11.001468926 +0000 UTC m=+174.407742668"
Jul 28 17:26:11.162317 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:11.162287    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bb5d8fd44-hr7k8"]
Jul 28 17:26:11.165396 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:11.165381    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb5d8fd44-hr7k8"
Jul 28 17:26:11.178584 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:11.178561    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bb5d8fd44-hr7k8"]
Jul 28 17:26:11.271251 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:11.271223    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-console-serving-cert\") pod \"console-bb5d8fd44-hr7k8\" (UID: \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\") " pod="openshift-console/console-bb5d8fd44-hr7k8"
Jul 28 17:26:11.271412 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:11.271254    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-oauth-serving-cert\") pod \"console-bb5d8fd44-hr7k8\" (UID: \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\") " pod="openshift-console/console-bb5d8fd44-hr7k8"
Jul 28 17:26:11.271412 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:11.271280    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-service-ca\") pod \"console-bb5d8fd44-hr7k8\" (UID: \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\") " pod="openshift-console/console-bb5d8fd44-hr7k8"
Jul 28 17:26:11.271412 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:11.271387    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-console-config\") pod \"console-bb5d8fd44-hr7k8\" (UID: \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\") " pod="openshift-console/console-bb5d8fd44-hr7k8"
Jul 28 17:26:11.271582 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:11.271413    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nrtr\" (UniqueName: \"kubernetes.io/projected/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-kube-api-access-6nrtr\") pod \"console-bb5d8fd44-hr7k8\" (UID: \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\") " pod="openshift-console/console-bb5d8fd44-hr7k8"
Jul 28 17:26:11.271582 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:11.271469    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-console-oauth-config\") pod \"console-bb5d8fd44-hr7k8\" (UID: \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\") " pod="openshift-console/console-bb5d8fd44-hr7k8"
Jul 28 17:26:11.271582 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:11.271517    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-trusted-ca-bundle\") pod \"console-bb5d8fd44-hr7k8\" (UID: \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\") " pod="openshift-console/console-bb5d8fd44-hr7k8"
Jul 28 17:26:11.372917 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:11.372880    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-console-serving-cert\") pod \"console-bb5d8fd44-hr7k8\" (UID: \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\") " pod="openshift-console/console-bb5d8fd44-hr7k8"
Jul 28 17:26:11.372917 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:11.372917    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-oauth-serving-cert\") pod \"console-bb5d8fd44-hr7k8\" (UID: \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\") " pod="openshift-console/console-bb5d8fd44-hr7k8"
Jul 28 17:26:11.373202 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:11.372935    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-service-ca\") pod \"console-bb5d8fd44-hr7k8\" (UID: \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\") " pod="openshift-console/console-bb5d8fd44-hr7k8"
Jul 28 17:26:11.373202 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:11.372970    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-console-config\") pod \"console-bb5d8fd44-hr7k8\" (UID: \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\") " pod="openshift-console/console-bb5d8fd44-hr7k8"
Jul 28 17:26:11.373202 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:11.372986    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nrtr\" (UniqueName: \"kubernetes.io/projected/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-kube-api-access-6nrtr\") pod \"console-bb5d8fd44-hr7k8\" (UID: \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\") " pod="openshift-console/console-bb5d8fd44-hr7k8"
Jul 28 17:26:11.373202 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:11.373104    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-console-oauth-config\") pod \"console-bb5d8fd44-hr7k8\" (UID: \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\") " pod="openshift-console/console-bb5d8fd44-hr7k8"
Jul 28 17:26:11.373202 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:11.373196    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-trusted-ca-bundle\") pod \"console-bb5d8fd44-hr7k8\" (UID: \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\") " pod="openshift-console/console-bb5d8fd44-hr7k8"
Jul 28 17:26:11.373780 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:11.373755    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-oauth-serving-cert\") pod \"console-bb5d8fd44-hr7k8\" (UID: \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\") " pod="openshift-console/console-bb5d8fd44-hr7k8"
Jul 28 17:26:11.373898 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:11.373802    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-service-ca\") pod \"console-bb5d8fd44-hr7k8\" (UID: \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\") " pod="openshift-console/console-bb5d8fd44-hr7k8"
Jul 28 17:26:11.373898 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:11.373802    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-console-config\") pod \"console-bb5d8fd44-hr7k8\" (UID: \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\") " pod="openshift-console/console-bb5d8fd44-hr7k8"
Jul 28 17:26:11.374121 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:11.374095    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-trusted-ca-bundle\") pod \"console-bb5d8fd44-hr7k8\" (UID: \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\") " pod="openshift-console/console-bb5d8fd44-hr7k8"
Jul 28 17:26:11.375579 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:11.375549    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-console-serving-cert\") pod \"console-bb5d8fd44-hr7k8\" (UID: \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\") " pod="openshift-console/console-bb5d8fd44-hr7k8"
Jul 28 17:26:11.375687 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:11.375652    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-console-oauth-config\") pod \"console-bb5d8fd44-hr7k8\" (UID: \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\") " pod="openshift-console/console-bb5d8fd44-hr7k8"
Jul 28 17:26:11.381058 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:11.381027    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nrtr\" (UniqueName: \"kubernetes.io/projected/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-kube-api-access-6nrtr\") pod \"console-bb5d8fd44-hr7k8\" (UID: \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\") " pod="openshift-console/console-bb5d8fd44-hr7k8"
Jul 28 17:26:11.474219 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:11.474188    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb5d8fd44-hr7k8"
Jul 28 17:26:11.601416 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:11.601393    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bb5d8fd44-hr7k8"]
Jul 28 17:26:11.603648 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:26:11.603623    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a0bd377_c6b3_4361_bcb2_ed02ea99659a.slice/crio-4370a787bff300af91948b25c765241fd5b378f6c681b0e7b670bea49303622f WatchSource:0}: Error finding container 4370a787bff300af91948b25c765241fd5b378f6c681b0e7b670bea49303622f: Status 404 returned error can't find the container with id 4370a787bff300af91948b25c765241fd5b378f6c681b0e7b670bea49303622f
Jul 28 17:26:11.981485 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:11.981390    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb5d8fd44-hr7k8" event={"ID":"7a0bd377-c6b3-4361-bcb2-ed02ea99659a","Type":"ContainerStarted","Data":"b98fa01be5e450f2f4601ed86e4cdb187bf82976bc6b4b5357600fa8f525d1cf"}
Jul 28 17:26:11.981485 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:11.981469    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb5d8fd44-hr7k8" event={"ID":"7a0bd377-c6b3-4361-bcb2-ed02ea99659a","Type":"ContainerStarted","Data":"4370a787bff300af91948b25c765241fd5b378f6c681b0e7b670bea49303622f"}
Jul 28 17:26:12.000646 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:12.000597    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bb5d8fd44-hr7k8" podStartSLOduration=1.000581554 podStartE2EDuration="1.000581554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:26:11.9992948 +0000 UTC m=+175.405568564" watchObservedRunningTime="2026-07-28 17:26:12.000581554 +0000 UTC m=+175.406855295"
Jul 28 17:26:21.474721 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:21.474689    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bb5d8fd44-hr7k8"
Jul 28 17:26:21.475129 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:21.474740    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bb5d8fd44-hr7k8"
Jul 28 17:26:21.479426 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:21.479404    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bb5d8fd44-hr7k8"
Jul 28 17:26:22.024647 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:22.024621    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bb5d8fd44-hr7k8"
Jul 28 17:26:22.085563 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:22.085532    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b6bb7dc96-cwkx6"]
Jul 28 17:26:47.106221 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:47.106185    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b6bb7dc96-cwkx6" podUID="2f8f9d16-dbd2-4981-b459-217b8f46b5c6" containerName="console" containerID="cri-o://19dbfede0d0144e2d9be9ae3348e12056dfb66fca1ec4a999330cfb231881ef6" gracePeriod=15
Jul 28 17:26:47.354205 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:47.354183    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b6bb7dc96-cwkx6"
Jul 28 17:26:47.463904 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:47.463831    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-console-serving-cert\") pod \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\" (UID: \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\") "
Jul 28 17:26:47.463904 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:47.463869    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9f9pc\" (UniqueName: \"kubernetes.io/projected/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-kube-api-access-9f9pc\") pod \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\" (UID: \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\") "
Jul 28 17:26:47.463904 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:47.463891    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-console-config\") pod \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\" (UID: \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\") "
Jul 28 17:26:47.464123 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:47.464101    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-service-ca\") pod \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\" (UID: \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\") "
Jul 28 17:26:47.464232 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:47.464216    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-oauth-serving-cert\") pod \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\" (UID: \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\") "
Jul 28 17:26:47.464281 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:47.464253    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-console-oauth-config\") pod \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\" (UID: \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\") "
Jul 28 17:26:47.464317 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:47.464292    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-trusted-ca-bundle\") pod \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\" (UID: \"2f8f9d16-dbd2-4981-b459-217b8f46b5c6\") "
Jul 28 17:26:47.464401 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:47.464379    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-console-config" (OuterVolumeSpecName: "console-config") pod "2f8f9d16-dbd2-4981-b459-217b8f46b5c6" (UID: "2f8f9d16-dbd2-4981-b459-217b8f46b5c6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:26:47.464544 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:47.464507    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-service-ca" (OuterVolumeSpecName: "service-ca") pod "2f8f9d16-dbd2-4981-b459-217b8f46b5c6" (UID: "2f8f9d16-dbd2-4981-b459-217b8f46b5c6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:26:47.464612 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:47.464544    2543 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-console-config\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:26:47.464670 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:47.464515    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2f8f9d16-dbd2-4981-b459-217b8f46b5c6" (UID: "2f8f9d16-dbd2-4981-b459-217b8f46b5c6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:26:47.464732 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:47.464712    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2f8f9d16-dbd2-4981-b459-217b8f46b5c6" (UID: "2f8f9d16-dbd2-4981-b459-217b8f46b5c6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:26:47.465997 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:47.465973    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-kube-api-access-9f9pc" (OuterVolumeSpecName: "kube-api-access-9f9pc") pod "2f8f9d16-dbd2-4981-b459-217b8f46b5c6" (UID: "2f8f9d16-dbd2-4981-b459-217b8f46b5c6"). InnerVolumeSpecName "kube-api-access-9f9pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:26:47.465997 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:47.465985    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2f8f9d16-dbd2-4981-b459-217b8f46b5c6" (UID: "2f8f9d16-dbd2-4981-b459-217b8f46b5c6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:26:47.466203 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:47.466181    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2f8f9d16-dbd2-4981-b459-217b8f46b5c6" (UID: "2f8f9d16-dbd2-4981-b459-217b8f46b5c6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:26:47.565803 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:47.565773    2543 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-console-serving-cert\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:26:47.565803 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:47.565798    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f9pc\" (UniqueName: \"kubernetes.io/projected/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-kube-api-access-9f9pc\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:26:47.565803 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:47.565809    2543 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-service-ca\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:26:47.565990 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:47.565817    2543 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-oauth-serving-cert\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:26:47.565990 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:47.565825    2543 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-console-oauth-config\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:26:47.565990 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:47.565833    2543 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f8f9d16-dbd2-4981-b459-217b8f46b5c6-trusted-ca-bundle\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:26:48.115393 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:48.115361    2543 generic.go:356] "Generic (PLEG): container finished" podID="2f8f9d16-dbd2-4981-b459-217b8f46b5c6" containerID="19dbfede0d0144e2d9be9ae3348e12056dfb66fca1ec4a999330cfb231881ef6" exitCode=0
Jul 28 17:26:48.115813 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:48.115409    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b6bb7dc96-cwkx6"
Jul 28 17:26:48.115813 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:48.115415    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b6bb7dc96-cwkx6" event={"ID":"2f8f9d16-dbd2-4981-b459-217b8f46b5c6","Type":"ContainerDied","Data":"19dbfede0d0144e2d9be9ae3348e12056dfb66fca1ec4a999330cfb231881ef6"}
Jul 28 17:26:48.115813 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:48.115454    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b6bb7dc96-cwkx6" event={"ID":"2f8f9d16-dbd2-4981-b459-217b8f46b5c6","Type":"ContainerDied","Data":"4f0c226ead79d1b980096a5ba44d0e530759734f96479b7ea9b0d51056f53e8c"}
Jul 28 17:26:48.115813 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:48.115474    2543 scope.go:117] "RemoveContainer" containerID="19dbfede0d0144e2d9be9ae3348e12056dfb66fca1ec4a999330cfb231881ef6"
Jul 28 17:26:48.123647 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:48.123619    2543 scope.go:117] "RemoveContainer" containerID="19dbfede0d0144e2d9be9ae3348e12056dfb66fca1ec4a999330cfb231881ef6"
Jul 28 17:26:48.123901 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:26:48.123883    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19dbfede0d0144e2d9be9ae3348e12056dfb66fca1ec4a999330cfb231881ef6\": container with ID starting with 19dbfede0d0144e2d9be9ae3348e12056dfb66fca1ec4a999330cfb231881ef6 not found: ID does not exist" containerID="19dbfede0d0144e2d9be9ae3348e12056dfb66fca1ec4a999330cfb231881ef6"
Jul 28 17:26:48.123955 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:48.123911    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19dbfede0d0144e2d9be9ae3348e12056dfb66fca1ec4a999330cfb231881ef6"} err="failed to get container status \"19dbfede0d0144e2d9be9ae3348e12056dfb66fca1ec4a999330cfb231881ef6\": rpc error: code = NotFound desc = could not find container \"19dbfede0d0144e2d9be9ae3348e12056dfb66fca1ec4a999330cfb231881ef6\": container with ID starting with 19dbfede0d0144e2d9be9ae3348e12056dfb66fca1ec4a999330cfb231881ef6 not found: ID does not exist"
Jul 28 17:26:48.136219 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:48.136194    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b6bb7dc96-cwkx6"]
Jul 28 17:26:48.139487 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:48.139469    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b6bb7dc96-cwkx6"]
Jul 28 17:26:49.151686 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:26:49.151652    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f8f9d16-dbd2-4981-b459-217b8f46b5c6" path="/var/lib/kubelet/pods/2f8f9d16-dbd2-4981-b459-217b8f46b5c6/volumes"
Jul 28 17:27:30.171966 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:30.171925    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kvwgh"]
Jul 28 17:27:30.172541 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:30.172365    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f8f9d16-dbd2-4981-b459-217b8f46b5c6" containerName="console"
Jul 28 17:27:30.172541 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:30.172383    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f8f9d16-dbd2-4981-b459-217b8f46b5c6" containerName="console"
Jul 28 17:27:30.172541 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:30.172469    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f8f9d16-dbd2-4981-b459-217b8f46b5c6" containerName="console"
Jul 28 17:27:30.176873 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:30.176852    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kvwgh"
Jul 28 17:27:30.179464 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:30.179437    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vmfb2\""
Jul 28 17:27:30.179577 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:30.179486    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 28 17:27:30.180044 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:30.180028    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 28 17:27:30.184319 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:30.184294    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kvwgh"]
Jul 28 17:27:30.212740 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:30.212709    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d35e3cb8-142e-40e3-b796-a68cead121ea-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kvwgh\" (UID: \"d35e3cb8-142e-40e3-b796-a68cead121ea\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kvwgh"
Jul 28 17:27:30.212878 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:30.212767    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m27s7\" (UniqueName: \"kubernetes.io/projected/d35e3cb8-142e-40e3-b796-a68cead121ea-kube-api-access-m27s7\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kvwgh\" (UID: \"d35e3cb8-142e-40e3-b796-a68cead121ea\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kvwgh"
Jul 28 17:27:30.212878 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:30.212824    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d35e3cb8-142e-40e3-b796-a68cead121ea-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kvwgh\" (UID: \"d35e3cb8-142e-40e3-b796-a68cead121ea\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kvwgh"
Jul 28 17:27:30.313633 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:30.313597    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d35e3cb8-142e-40e3-b796-a68cead121ea-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kvwgh\" (UID: \"d35e3cb8-142e-40e3-b796-a68cead121ea\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kvwgh"
Jul 28 17:27:30.313791 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:30.313642    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d35e3cb8-142e-40e3-b796-a68cead121ea-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kvwgh\" (UID: \"d35e3cb8-142e-40e3-b796-a68cead121ea\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kvwgh"
Jul 28 17:27:30.313791 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:30.313688    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m27s7\" (UniqueName: \"kubernetes.io/projected/d35e3cb8-142e-40e3-b796-a68cead121ea-kube-api-access-m27s7\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kvwgh\" (UID: \"d35e3cb8-142e-40e3-b796-a68cead121ea\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kvwgh"
Jul 28 17:27:30.314002 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:30.313980    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d35e3cb8-142e-40e3-b796-a68cead121ea-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kvwgh\" (UID: \"d35e3cb8-142e-40e3-b796-a68cead121ea\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kvwgh"
Jul 28 17:27:30.314076 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:30.314044    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d35e3cb8-142e-40e3-b796-a68cead121ea-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kvwgh\" (UID: \"d35e3cb8-142e-40e3-b796-a68cead121ea\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kvwgh"
Jul 28 17:27:30.321905 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:30.321883    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m27s7\" (UniqueName: \"kubernetes.io/projected/d35e3cb8-142e-40e3-b796-a68cead121ea-kube-api-access-m27s7\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kvwgh\" (UID: \"d35e3cb8-142e-40e3-b796-a68cead121ea\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kvwgh"
Jul 28 17:27:30.486376 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:30.486293    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kvwgh"
Jul 28 17:27:30.628094 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:30.628049    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kvwgh"]
Jul 28 17:27:30.630680 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:27:30.630649    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd35e3cb8_142e_40e3_b796_a68cead121ea.slice/crio-8a86d1a4b9775e6c6276155622990598a5ac5c791743fe33c57ec91f941d1917 WatchSource:0}: Error finding container 8a86d1a4b9775e6c6276155622990598a5ac5c791743fe33c57ec91f941d1917: Status 404 returned error can't find the container with id 8a86d1a4b9775e6c6276155622990598a5ac5c791743fe33c57ec91f941d1917
Jul 28 17:27:31.275320 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:31.275257    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kvwgh" event={"ID":"d35e3cb8-142e-40e3-b796-a68cead121ea","Type":"ContainerStarted","Data":"8a86d1a4b9775e6c6276155622990598a5ac5c791743fe33c57ec91f941d1917"}
Jul 28 17:27:36.296533 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:36.296493    2543 generic.go:356] "Generic (PLEG): container finished" podID="d35e3cb8-142e-40e3-b796-a68cead121ea" containerID="3b657f8234e8213992c638d31998e23d4ee8289b8f250cb2107ca37405c8a98f" exitCode=0
Jul 28 17:27:36.296980 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:36.296577    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kvwgh" event={"ID":"d35e3cb8-142e-40e3-b796-a68cead121ea","Type":"ContainerDied","Data":"3b657f8234e8213992c638d31998e23d4ee8289b8f250cb2107ca37405c8a98f"}
Jul 28 17:27:38.309235 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:38.309199    2543 generic.go:356] "Generic (PLEG): container finished" podID="d35e3cb8-142e-40e3-b796-a68cead121ea" containerID="52abbd43ab7b26f1601f88f8ef1da7896e02d6ea36842a316af4114b12456ab9" exitCode=0
Jul 28 17:27:38.309690 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:38.309269    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kvwgh" event={"ID":"d35e3cb8-142e-40e3-b796-a68cead121ea","Type":"ContainerDied","Data":"52abbd43ab7b26f1601f88f8ef1da7896e02d6ea36842a316af4114b12456ab9"}
Jul 28 17:27:44.334414 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:44.334382    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kvwgh" event={"ID":"d35e3cb8-142e-40e3-b796-a68cead121ea","Type":"ContainerStarted","Data":"9367c1c754555eea4278f2a4ae1b9755410c84cd586ffe82c2c967e8474f5d70"}
Jul 28 17:27:44.366355 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:44.366301    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kvwgh" podStartSLOduration=1.6156604140000002 podStartE2EDuration="14.366284157s" totalImagesPullingTime="12.750623743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:27:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:27:44.364376455 +0000 UTC m=+267.770650196" watchObservedRunningTime="2026-07-28 17:27:44.366284157 +0000 UTC m=+267.772557898"
Jul 28 17:27:45.340303 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:45.340267    2543 generic.go:356] "Generic (PLEG): container finished" podID="d35e3cb8-142e-40e3-b796-a68cead121ea" containerID="9367c1c754555eea4278f2a4ae1b9755410c84cd586ffe82c2c967e8474f5d70" exitCode=0
Jul 28 17:27:45.340801 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:45.340347    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kvwgh" event={"ID":"d35e3cb8-142e-40e3-b796-a68cead121ea","Type":"ContainerDied","Data":"9367c1c754555eea4278f2a4ae1b9755410c84cd586ffe82c2c967e8474f5d70"}
Jul 28 17:27:46.467358 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:46.467332    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kvwgh"
Jul 28 17:27:46.564615 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:46.564582    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d35e3cb8-142e-40e3-b796-a68cead121ea-util\") pod \"d35e3cb8-142e-40e3-b796-a68cead121ea\" (UID: \"d35e3cb8-142e-40e3-b796-a68cead121ea\") "
Jul 28 17:27:46.564800 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:46.564626    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m27s7\" (UniqueName: \"kubernetes.io/projected/d35e3cb8-142e-40e3-b796-a68cead121ea-kube-api-access-m27s7\") pod \"d35e3cb8-142e-40e3-b796-a68cead121ea\" (UID: \"d35e3cb8-142e-40e3-b796-a68cead121ea\") "
Jul 28 17:27:46.564800 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:46.564676    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d35e3cb8-142e-40e3-b796-a68cead121ea-bundle\") pod \"d35e3cb8-142e-40e3-b796-a68cead121ea\" (UID: \"d35e3cb8-142e-40e3-b796-a68cead121ea\") "
Jul 28 17:27:46.565379 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:46.565354    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d35e3cb8-142e-40e3-b796-a68cead121ea-bundle" (OuterVolumeSpecName: "bundle") pod "d35e3cb8-142e-40e3-b796-a68cead121ea" (UID: "d35e3cb8-142e-40e3-b796-a68cead121ea"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:27:46.566884 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:46.566857    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d35e3cb8-142e-40e3-b796-a68cead121ea-kube-api-access-m27s7" (OuterVolumeSpecName: "kube-api-access-m27s7") pod "d35e3cb8-142e-40e3-b796-a68cead121ea" (UID: "d35e3cb8-142e-40e3-b796-a68cead121ea"). InnerVolumeSpecName "kube-api-access-m27s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:27:46.569364 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:46.569336    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d35e3cb8-142e-40e3-b796-a68cead121ea-util" (OuterVolumeSpecName: "util") pod "d35e3cb8-142e-40e3-b796-a68cead121ea" (UID: "d35e3cb8-142e-40e3-b796-a68cead121ea"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:27:46.665483 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:46.665393    2543 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d35e3cb8-142e-40e3-b796-a68cead121ea-util\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:27:46.665483 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:46.665426    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m27s7\" (UniqueName: \"kubernetes.io/projected/d35e3cb8-142e-40e3-b796-a68cead121ea-kube-api-access-m27s7\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:27:46.665483 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:46.665443    2543 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d35e3cb8-142e-40e3-b796-a68cead121ea-bundle\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:27:47.349728 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:47.349698    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kvwgh"
Jul 28 17:27:47.349894 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:47.349703    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5kvwgh" event={"ID":"d35e3cb8-142e-40e3-b796-a68cead121ea","Type":"ContainerDied","Data":"8a86d1a4b9775e6c6276155622990598a5ac5c791743fe33c57ec91f941d1917"}
Jul 28 17:27:47.349894 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:47.349817    2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a86d1a4b9775e6c6276155622990598a5ac5c791743fe33c57ec91f941d1917"
Jul 28 17:27:55.010998 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:55.010966    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-x7b8m"]
Jul 28 17:27:55.011470 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:55.011296    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d35e3cb8-142e-40e3-b796-a68cead121ea" containerName="pull"
Jul 28 17:27:55.011470 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:55.011309    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="d35e3cb8-142e-40e3-b796-a68cead121ea" containerName="pull"
Jul 28 17:27:55.011470 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:55.011318    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d35e3cb8-142e-40e3-b796-a68cead121ea" containerName="util"
Jul 28 17:27:55.011470 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:55.011323    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="d35e3cb8-142e-40e3-b796-a68cead121ea" containerName="util"
Jul 28 17:27:55.011470 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:55.011337    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d35e3cb8-142e-40e3-b796-a68cead121ea" containerName="extract"
Jul 28 17:27:55.011470 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:55.011344    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="d35e3cb8-142e-40e3-b796-a68cead121ea" containerName="extract"
Jul 28 17:27:55.011470 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:55.011390    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="d35e3cb8-142e-40e3-b796-a68cead121ea" containerName="extract"
Jul 28 17:27:55.065535 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:55.065505    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-x7b8m"]
Jul 28 17:27:55.065681 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:55.065598    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-x7b8m"
Jul 28 17:27:55.070041 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:55.070020    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 28 17:27:55.070809 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:55.070779    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-jr8pf\""
Jul 28 17:27:55.070918 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:55.070811    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 28 17:27:55.075318 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:55.075304    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 28 17:27:55.131178 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:55.131149    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/2cb839f7-a9e3-4e5d-bc9f-4e2e346619fb-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-x7b8m\" (UID: \"2cb839f7-a9e3-4e5d-bc9f-4e2e346619fb\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-x7b8m"
Jul 28 17:27:55.131304 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:55.131198    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7zl7\" (UniqueName: \"kubernetes.io/projected/2cb839f7-a9e3-4e5d-bc9f-4e2e346619fb-kube-api-access-h7zl7\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-x7b8m\" (UID: \"2cb839f7-a9e3-4e5d-bc9f-4e2e346619fb\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-x7b8m"
Jul 28 17:27:55.231778 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:55.231745    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/2cb839f7-a9e3-4e5d-bc9f-4e2e346619fb-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-x7b8m\" (UID: \"2cb839f7-a9e3-4e5d-bc9f-4e2e346619fb\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-x7b8m"
Jul 28 17:27:55.231937 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:55.231786    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7zl7\" (UniqueName: \"kubernetes.io/projected/2cb839f7-a9e3-4e5d-bc9f-4e2e346619fb-kube-api-access-h7zl7\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-x7b8m\" (UID: \"2cb839f7-a9e3-4e5d-bc9f-4e2e346619fb\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-x7b8m"
Jul 28 17:27:55.234170 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:55.234125    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/2cb839f7-a9e3-4e5d-bc9f-4e2e346619fb-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-x7b8m\" (UID: \"2cb839f7-a9e3-4e5d-bc9f-4e2e346619fb\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-x7b8m"
Jul 28 17:27:55.245792 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:55.245760    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7zl7\" (UniqueName: \"kubernetes.io/projected/2cb839f7-a9e3-4e5d-bc9f-4e2e346619fb-kube-api-access-h7zl7\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-x7b8m\" (UID: \"2cb839f7-a9e3-4e5d-bc9f-4e2e346619fb\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-x7b8m"
Jul 28 17:27:55.375365 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:55.375285    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-x7b8m"
Jul 28 17:27:55.558966 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:55.558938    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-x7b8m"]
Jul 28 17:27:55.560652 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:27:55.560624    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb839f7_a9e3_4e5d_bc9f_4e2e346619fb.slice/crio-5756b8b7d6064ea9b10371f92d24f7bd110ca2c8ebe0e14ca1f42d896c2a6fe1 WatchSource:0}: Error finding container 5756b8b7d6064ea9b10371f92d24f7bd110ca2c8ebe0e14ca1f42d896c2a6fe1: Status 404 returned error can't find the container with id 5756b8b7d6064ea9b10371f92d24f7bd110ca2c8ebe0e14ca1f42d896c2a6fe1
Jul 28 17:27:56.386282 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:56.386239    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-x7b8m" event={"ID":"2cb839f7-a9e3-4e5d-bc9f-4e2e346619fb","Type":"ContainerStarted","Data":"5756b8b7d6064ea9b10371f92d24f7bd110ca2c8ebe0e14ca1f42d896c2a6fe1"}
Jul 28 17:27:58.396687 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:58.396602    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-x7b8m" event={"ID":"2cb839f7-a9e3-4e5d-bc9f-4e2e346619fb","Type":"ContainerStarted","Data":"0cca3afe53e12ac447808c89471d1ae50463915d0e0e672d0b33d16d10bf4fc7"}
Jul 28 17:27:58.397039 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:58.396740    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-x7b8m"
Jul 28 17:27:58.455124 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:58.455067    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-x7b8m" podStartSLOduration=1.925035475 podStartE2EDuration="4.455049201s" totalImagesPullingTime="2.530013726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:27:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:27:58.454664439 +0000 UTC m=+281.860938181" watchObservedRunningTime="2026-07-28 17:27:58.455049201 +0000 UTC m=+281.861322942"
Jul 28 17:27:59.192678 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:59.192648    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-gjxrd"]
Jul 28 17:27:59.194768 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:59.194751    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-gjxrd"
Jul 28 17:27:59.199179 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:59.199159    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 28 17:27:59.199438 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:59.199424    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-5lwzx\""
Jul 28 17:27:59.199969 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:59.199944    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 28 17:27:59.218061 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:59.218037    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-gjxrd"]
Jul 28 17:27:59.266978 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:59.266944    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7f8facda-e7d8-4aad-b6ee-069f8eaa962b-certificates\") pod \"keda-operator-77f648d694-gjxrd\" (UID: \"7f8facda-e7d8-4aad-b6ee-069f8eaa962b\") " pod="openshift-keda/keda-operator-77f648d694-gjxrd"
Jul 28 17:27:59.267124 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:59.267008    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/7f8facda-e7d8-4aad-b6ee-069f8eaa962b-cabundle0\") pod \"keda-operator-77f648d694-gjxrd\" (UID: \"7f8facda-e7d8-4aad-b6ee-069f8eaa962b\") " pod="openshift-keda/keda-operator-77f648d694-gjxrd"
Jul 28 17:27:59.267124 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:59.267068    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cglx5\" (UniqueName: \"kubernetes.io/projected/7f8facda-e7d8-4aad-b6ee-069f8eaa962b-kube-api-access-cglx5\") pod \"keda-operator-77f648d694-gjxrd\" (UID: \"7f8facda-e7d8-4aad-b6ee-069f8eaa962b\") " pod="openshift-keda/keda-operator-77f648d694-gjxrd"
Jul 28 17:27:59.368316 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:59.368282    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7f8facda-e7d8-4aad-b6ee-069f8eaa962b-certificates\") pod \"keda-operator-77f648d694-gjxrd\" (UID: \"7f8facda-e7d8-4aad-b6ee-069f8eaa962b\") " pod="openshift-keda/keda-operator-77f648d694-gjxrd"
Jul 28 17:27:59.368462 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:59.368341    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/7f8facda-e7d8-4aad-b6ee-069f8eaa962b-cabundle0\") pod \"keda-operator-77f648d694-gjxrd\" (UID: \"7f8facda-e7d8-4aad-b6ee-069f8eaa962b\") " pod="openshift-keda/keda-operator-77f648d694-gjxrd"
Jul 28 17:27:59.368462 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:59.368384    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cglx5\" (UniqueName: \"kubernetes.io/projected/7f8facda-e7d8-4aad-b6ee-069f8eaa962b-kube-api-access-cglx5\") pod \"keda-operator-77f648d694-gjxrd\" (UID: \"7f8facda-e7d8-4aad-b6ee-069f8eaa962b\") " pod="openshift-keda/keda-operator-77f648d694-gjxrd"
Jul 28 17:27:59.368548 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:27:59.368456    2543 secret.go:281] references non-existent secret key: ca.crt
Jul 28 17:27:59.368548 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:27:59.368480    2543 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 28 17:27:59.368548 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:27:59.368492    2543 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-gjxrd: references non-existent secret key: ca.crt
Jul 28 17:27:59.368657 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:27:59.368554    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f8facda-e7d8-4aad-b6ee-069f8eaa962b-certificates podName:7f8facda-e7d8-4aad-b6ee-069f8eaa962b nodeName:}" failed. No retries permitted until 2026-07-28 17:27:59.868533262 +0000 UTC m=+283.274806995 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/7f8facda-e7d8-4aad-b6ee-069f8eaa962b-certificates") pod "keda-operator-77f648d694-gjxrd" (UID: "7f8facda-e7d8-4aad-b6ee-069f8eaa962b") : references non-existent secret key: ca.crt
Jul 28 17:27:59.369065 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:59.369048    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/7f8facda-e7d8-4aad-b6ee-069f8eaa962b-cabundle0\") pod \"keda-operator-77f648d694-gjxrd\" (UID: \"7f8facda-e7d8-4aad-b6ee-069f8eaa962b\") " pod="openshift-keda/keda-operator-77f648d694-gjxrd"
Jul 28 17:27:59.382619 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:59.382583    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cglx5\" (UniqueName: \"kubernetes.io/projected/7f8facda-e7d8-4aad-b6ee-069f8eaa962b-kube-api-access-cglx5\") pod \"keda-operator-77f648d694-gjxrd\" (UID: \"7f8facda-e7d8-4aad-b6ee-069f8eaa962b\") " pod="openshift-keda/keda-operator-77f648d694-gjxrd"
Jul 28 17:27:59.874206 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:59.874171    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7f8facda-e7d8-4aad-b6ee-069f8eaa962b-certificates\") pod \"keda-operator-77f648d694-gjxrd\" (UID: \"7f8facda-e7d8-4aad-b6ee-069f8eaa962b\") " pod="openshift-keda/keda-operator-77f648d694-gjxrd"
Jul 28 17:27:59.876619 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:27:59.876590    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7f8facda-e7d8-4aad-b6ee-069f8eaa962b-certificates\") pod \"keda-operator-77f648d694-gjxrd\" (UID: \"7f8facda-e7d8-4aad-b6ee-069f8eaa962b\") " pod="openshift-keda/keda-operator-77f648d694-gjxrd"
Jul 28 17:28:00.105023 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:28:00.104988    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-gjxrd"
Jul 28 17:28:00.244534 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:28:00.244503    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-gjxrd"]
Jul 28 17:28:00.248084 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:28:00.248056    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f8facda_e7d8_4aad_b6ee_069f8eaa962b.slice/crio-6b793513b69156d570bc92d9503418f00cd4dd18c6b10fa7fe9f463a3ef5d41f WatchSource:0}: Error finding container 6b793513b69156d570bc92d9503418f00cd4dd18c6b10fa7fe9f463a3ef5d41f: Status 404 returned error can't find the container with id 6b793513b69156d570bc92d9503418f00cd4dd18c6b10fa7fe9f463a3ef5d41f
Jul 28 17:28:00.406490 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:28:00.406412    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-gjxrd" event={"ID":"7f8facda-e7d8-4aad-b6ee-069f8eaa962b","Type":"ContainerStarted","Data":"6b793513b69156d570bc92d9503418f00cd4dd18c6b10fa7fe9f463a3ef5d41f"}
Jul 28 17:28:03.421692 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:28:03.421659    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-gjxrd" event={"ID":"7f8facda-e7d8-4aad-b6ee-069f8eaa962b","Type":"ContainerStarted","Data":"397652c42dfeb1d1a51c8cfe1cc1cadcdf7d97a004a33e2cbe97664cc0c69709"}
Jul 28 17:28:03.422167 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:28:03.421754    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-gjxrd"
Jul 28 17:28:03.467957 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:28:03.467912    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-gjxrd" podStartSLOduration=1.445209294 podStartE2EDuration="4.467897674s" totalImagesPullingTime="3.02268838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:27:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:28:03.466868324 +0000 UTC m=+286.873142078" watchObservedRunningTime="2026-07-28 17:28:03.467897674 +0000 UTC m=+286.874171414"
Jul 28 17:28:16.929513 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:28:16.929487    2543 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 17:28:19.404355 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:28:19.404322    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-x7b8m"
Jul 28 17:28:24.428096 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:28:24.428068    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-gjxrd"
Jul 28 17:29:24.861554 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:24.861470    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-4n9rn"]
Jul 28 17:29:24.864302 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:24.864278    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cd4d44c67-4n9rn"
Jul 28 17:29:24.866874 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:24.866852    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-mp8lc\""
Jul 28 17:29:24.867009 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:24.866927    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 17:29:24.867690 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:24.867669    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 17:29:24.867690 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:24.867684    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 28 17:29:24.871995 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:24.871972    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/720b74c0-3bc1-42a8-9ba7-63d612b5044d-cert\") pod \"kserve-controller-manager-cd4d44c67-4n9rn\" (UID: \"720b74c0-3bc1-42a8-9ba7-63d612b5044d\") " pod="kserve/kserve-controller-manager-cd4d44c67-4n9rn"
Jul 28 17:29:24.872110 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:24.872052    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwj94\" (UniqueName: \"kubernetes.io/projected/720b74c0-3bc1-42a8-9ba7-63d612b5044d-kube-api-access-gwj94\") pod \"kserve-controller-manager-cd4d44c67-4n9rn\" (UID: \"720b74c0-3bc1-42a8-9ba7-63d612b5044d\") " pod="kserve/kserve-controller-manager-cd4d44c67-4n9rn"
Jul 28 17:29:24.877732 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:24.877712    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-4n9rn"]
Jul 28 17:29:24.899390 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:24.899369    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-545d98d4c7-5zmwp"]
Jul 28 17:29:24.901937 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:24.901916    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-545d98d4c7-5zmwp"
Jul 28 17:29:24.904241 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:24.904220    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 28 17:29:24.904342 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:24.904239    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-n544f\""
Jul 28 17:29:24.918534 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:24.918506    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-545d98d4c7-5zmwp"]
Jul 28 17:29:24.972814 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:24.972775    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/720b74c0-3bc1-42a8-9ba7-63d612b5044d-cert\") pod \"kserve-controller-manager-cd4d44c67-4n9rn\" (UID: \"720b74c0-3bc1-42a8-9ba7-63d612b5044d\") " pod="kserve/kserve-controller-manager-cd4d44c67-4n9rn"
Jul 28 17:29:24.972972 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:24.972839    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cdz8\" (UniqueName: \"kubernetes.io/projected/0ef61165-c90b-4f43-b6e8-43b9bb0909bd-kube-api-access-9cdz8\") pod \"llmisvc-controller-manager-545d98d4c7-5zmwp\" (UID: \"0ef61165-c90b-4f43-b6e8-43b9bb0909bd\") " pod="kserve/llmisvc-controller-manager-545d98d4c7-5zmwp"
Jul 28 17:29:24.972972 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:29:24.972888    2543 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 28 17:29:24.972972 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:24.972899    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ef61165-c90b-4f43-b6e8-43b9bb0909bd-cert\") pod \"llmisvc-controller-manager-545d98d4c7-5zmwp\" (UID: \"0ef61165-c90b-4f43-b6e8-43b9bb0909bd\") " pod="kserve/llmisvc-controller-manager-545d98d4c7-5zmwp"
Jul 28 17:29:24.972972 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:29:24.972945    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/720b74c0-3bc1-42a8-9ba7-63d612b5044d-cert podName:720b74c0-3bc1-42a8-9ba7-63d612b5044d nodeName:}" failed. No retries permitted until 2026-07-28 17:29:25.472928688 +0000 UTC m=+368.879202411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/720b74c0-3bc1-42a8-9ba7-63d612b5044d-cert") pod "kserve-controller-manager-cd4d44c67-4n9rn" (UID: "720b74c0-3bc1-42a8-9ba7-63d612b5044d") : secret "kserve-webhook-server-cert" not found
Jul 28 17:29:24.973145 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:24.973005    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwj94\" (UniqueName: \"kubernetes.io/projected/720b74c0-3bc1-42a8-9ba7-63d612b5044d-kube-api-access-gwj94\") pod \"kserve-controller-manager-cd4d44c67-4n9rn\" (UID: \"720b74c0-3bc1-42a8-9ba7-63d612b5044d\") " pod="kserve/kserve-controller-manager-cd4d44c67-4n9rn"
Jul 28 17:29:24.988390 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:24.988360    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwj94\" (UniqueName: \"kubernetes.io/projected/720b74c0-3bc1-42a8-9ba7-63d612b5044d-kube-api-access-gwj94\") pod \"kserve-controller-manager-cd4d44c67-4n9rn\" (UID: \"720b74c0-3bc1-42a8-9ba7-63d612b5044d\") " pod="kserve/kserve-controller-manager-cd4d44c67-4n9rn"
Jul 28 17:29:25.074061 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:25.074033    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cdz8\" (UniqueName: \"kubernetes.io/projected/0ef61165-c90b-4f43-b6e8-43b9bb0909bd-kube-api-access-9cdz8\") pod \"llmisvc-controller-manager-545d98d4c7-5zmwp\" (UID: \"0ef61165-c90b-4f43-b6e8-43b9bb0909bd\") " pod="kserve/llmisvc-controller-manager-545d98d4c7-5zmwp"
Jul 28 17:29:25.074255 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:25.074085    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ef61165-c90b-4f43-b6e8-43b9bb0909bd-cert\") pod \"llmisvc-controller-manager-545d98d4c7-5zmwp\" (UID: \"0ef61165-c90b-4f43-b6e8-43b9bb0909bd\") " pod="kserve/llmisvc-controller-manager-545d98d4c7-5zmwp"
Jul 28 17:29:25.076328 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:25.076303    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ef61165-c90b-4f43-b6e8-43b9bb0909bd-cert\") pod \"llmisvc-controller-manager-545d98d4c7-5zmwp\" (UID: \"0ef61165-c90b-4f43-b6e8-43b9bb0909bd\") " pod="kserve/llmisvc-controller-manager-545d98d4c7-5zmwp"
Jul 28 17:29:25.086128 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:25.086105    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cdz8\" (UniqueName: \"kubernetes.io/projected/0ef61165-c90b-4f43-b6e8-43b9bb0909bd-kube-api-access-9cdz8\") pod \"llmisvc-controller-manager-545d98d4c7-5zmwp\" (UID: \"0ef61165-c90b-4f43-b6e8-43b9bb0909bd\") " pod="kserve/llmisvc-controller-manager-545d98d4c7-5zmwp"
Jul 28 17:29:25.212279 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:25.212205    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-545d98d4c7-5zmwp"
Jul 28 17:29:25.338765 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:25.338736    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-545d98d4c7-5zmwp"]
Jul 28 17:29:25.342553 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:29:25.342527    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0ef61165_c90b_4f43_b6e8_43b9bb0909bd.slice/crio-dcc56d006bb2091f6b7ffb0a6e972993fd6c922b5a259153681b802a31f5c846 WatchSource:0}: Error finding container dcc56d006bb2091f6b7ffb0a6e972993fd6c922b5a259153681b802a31f5c846: Status 404 returned error can't find the container with id dcc56d006bb2091f6b7ffb0a6e972993fd6c922b5a259153681b802a31f5c846
Jul 28 17:29:25.344112 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:25.344094    2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:29:25.476891 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:25.476822    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/720b74c0-3bc1-42a8-9ba7-63d612b5044d-cert\") pod \"kserve-controller-manager-cd4d44c67-4n9rn\" (UID: \"720b74c0-3bc1-42a8-9ba7-63d612b5044d\") " pod="kserve/kserve-controller-manager-cd4d44c67-4n9rn"
Jul 28 17:29:25.479098 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:25.479073    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/720b74c0-3bc1-42a8-9ba7-63d612b5044d-cert\") pod \"kserve-controller-manager-cd4d44c67-4n9rn\" (UID: \"720b74c0-3bc1-42a8-9ba7-63d612b5044d\") " pod="kserve/kserve-controller-manager-cd4d44c67-4n9rn"
Jul 28 17:29:25.758525 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:25.758499    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-545d98d4c7-5zmwp" event={"ID":"0ef61165-c90b-4f43-b6e8-43b9bb0909bd","Type":"ContainerStarted","Data":"dcc56d006bb2091f6b7ffb0a6e972993fd6c922b5a259153681b802a31f5c846"}
Jul 28 17:29:25.774440 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:25.774418    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cd4d44c67-4n9rn"
Jul 28 17:29:25.910598 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:25.910561    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-4n9rn"]
Jul 28 17:29:25.912944 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:29:25.912918    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod720b74c0_3bc1_42a8_9ba7_63d612b5044d.slice/crio-5d3efdab74e83a03f8578ff919aca392080a82b8f132e61607670a88f3f75b53 WatchSource:0}: Error finding container 5d3efdab74e83a03f8578ff919aca392080a82b8f132e61607670a88f3f75b53: Status 404 returned error can't find the container with id 5d3efdab74e83a03f8578ff919aca392080a82b8f132e61607670a88f3f75b53
Jul 28 17:29:26.767284 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:26.767207    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cd4d44c67-4n9rn" event={"ID":"720b74c0-3bc1-42a8-9ba7-63d612b5044d","Type":"ContainerStarted","Data":"5d3efdab74e83a03f8578ff919aca392080a82b8f132e61607670a88f3f75b53"}
Jul 28 17:29:29.783164 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:29.783117    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-545d98d4c7-5zmwp" event={"ID":"0ef61165-c90b-4f43-b6e8-43b9bb0909bd","Type":"ContainerStarted","Data":"ee3641e8554bfb89f82e0467d513661ca3e6a80b258488785dd4027b271965bc"}
Jul 28 17:29:29.783617 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:29.783219    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-545d98d4c7-5zmwp"
Jul 28 17:29:29.784630 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:29.784606    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cd4d44c67-4n9rn" event={"ID":"720b74c0-3bc1-42a8-9ba7-63d612b5044d","Type":"ContainerStarted","Data":"11240730b56936a23668f2939af4d338ad027dfe9fe5d0441959d578c059f601"}
Jul 28 17:29:29.784737 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:29.784653    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-cd4d44c67-4n9rn"
Jul 28 17:29:29.802761 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:29.802717    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-545d98d4c7-5zmwp" podStartSLOduration=2.030501852 podStartE2EDuration="5.802703595s" totalImagesPullingTime="3.772201743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:29:29.802322302 +0000 UTC m=+373.208596043" watchObservedRunningTime="2026-07-28 17:29:29.802703595 +0000 UTC m=+373.208977337"
Jul 28 17:29:29.821099 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:29:29.821053    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-cd4d44c67-4n9rn" podStartSLOduration=2.618796334 podStartE2EDuration="5.821039015s" totalImagesPullingTime="3.202242681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:29:29.820481447 +0000 UTC m=+373.226755189" watchObservedRunningTime="2026-07-28 17:29:29.821039015 +0000 UTC m=+373.227312797"
Jul 28 17:30:00.792154 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:00.792099    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-545d98d4c7-5zmwp"
Jul 28 17:30:00.795052 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:00.795030    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-cd4d44c67-4n9rn"
Jul 28 17:30:09.398529 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:09.398494    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540npdqn"]
Jul 28 17:30:09.400864 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:09.400847    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540npdqn"
Jul 28 17:30:09.403451 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:09.403429    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 28 17:30:09.403582 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:09.403469    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vmfb2\""
Jul 28 17:30:09.404370 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:09.404351    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 28 17:30:09.411211 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:09.411185    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540npdqn"]
Jul 28 17:30:09.550300 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:09.550259    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83b4880a-83ba-41da-b9ef-78a2977ff2b0-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540npdqn\" (UID: \"83b4880a-83ba-41da-b9ef-78a2977ff2b0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540npdqn"
Jul 28 17:30:09.550300 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:09.550301    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83b4880a-83ba-41da-b9ef-78a2977ff2b0-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540npdqn\" (UID: \"83b4880a-83ba-41da-b9ef-78a2977ff2b0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540npdqn"
Jul 28 17:30:09.550507 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:09.550385    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkwdf\" (UniqueName: \"kubernetes.io/projected/83b4880a-83ba-41da-b9ef-78a2977ff2b0-kube-api-access-qkwdf\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540npdqn\" (UID: \"83b4880a-83ba-41da-b9ef-78a2977ff2b0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540npdqn"
Jul 28 17:30:09.651656 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:09.651587    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkwdf\" (UniqueName: \"kubernetes.io/projected/83b4880a-83ba-41da-b9ef-78a2977ff2b0-kube-api-access-qkwdf\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540npdqn\" (UID: \"83b4880a-83ba-41da-b9ef-78a2977ff2b0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540npdqn"
Jul 28 17:30:09.651744 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:09.651680    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83b4880a-83ba-41da-b9ef-78a2977ff2b0-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540npdqn\" (UID: \"83b4880a-83ba-41da-b9ef-78a2977ff2b0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540npdqn"
Jul 28 17:30:09.651744 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:09.651709    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83b4880a-83ba-41da-b9ef-78a2977ff2b0-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540npdqn\" (UID: \"83b4880a-83ba-41da-b9ef-78a2977ff2b0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540npdqn"
Jul 28 17:30:09.652109 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:09.652093    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83b4880a-83ba-41da-b9ef-78a2977ff2b0-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540npdqn\" (UID: \"83b4880a-83ba-41da-b9ef-78a2977ff2b0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540npdqn"
Jul 28 17:30:09.652169 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:09.652126    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83b4880a-83ba-41da-b9ef-78a2977ff2b0-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540npdqn\" (UID: \"83b4880a-83ba-41da-b9ef-78a2977ff2b0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540npdqn"
Jul 28 17:30:09.672705 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:09.672675    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkwdf\" (UniqueName: \"kubernetes.io/projected/83b4880a-83ba-41da-b9ef-78a2977ff2b0-kube-api-access-qkwdf\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540npdqn\" (UID: \"83b4880a-83ba-41da-b9ef-78a2977ff2b0\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540npdqn"
Jul 28 17:30:09.710356 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:09.710319    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540npdqn"
Jul 28 17:30:09.846479 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:09.846446    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540npdqn"]
Jul 28 17:30:09.849873 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:30:09.849844    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83b4880a_83ba_41da_b9ef_78a2977ff2b0.slice/crio-83f62fb0ca8c20581e780c14be9eef5d51d5d53b5fff1ba990a99e697e70a027 WatchSource:0}: Error finding container 83f62fb0ca8c20581e780c14be9eef5d51d5d53b5fff1ba990a99e697e70a027: Status 404 returned error can't find the container with id 83f62fb0ca8c20581e780c14be9eef5d51d5d53b5fff1ba990a99e697e70a027
Jul 28 17:30:09.951435 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:09.951404    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540npdqn" event={"ID":"83b4880a-83ba-41da-b9ef-78a2977ff2b0","Type":"ContainerStarted","Data":"66b11ad28ac8494f98c4074bd7ba85345688d68f2e4b882c5a6ae1cbedab73d3"}
Jul 28 17:30:09.951577 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:09.951451    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540npdqn" event={"ID":"83b4880a-83ba-41da-b9ef-78a2977ff2b0","Type":"ContainerStarted","Data":"83f62fb0ca8c20581e780c14be9eef5d51d5d53b5fff1ba990a99e697e70a027"}
Jul 28 17:30:10.957069 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:10.956988    2543 generic.go:356] "Generic (PLEG): container finished" podID="83b4880a-83ba-41da-b9ef-78a2977ff2b0" containerID="66b11ad28ac8494f98c4074bd7ba85345688d68f2e4b882c5a6ae1cbedab73d3" exitCode=0
Jul 28 17:30:10.957069 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:10.957038    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540npdqn" event={"ID":"83b4880a-83ba-41da-b9ef-78a2977ff2b0","Type":"ContainerDied","Data":"66b11ad28ac8494f98c4074bd7ba85345688d68f2e4b882c5a6ae1cbedab73d3"}
Jul 28 17:30:20.999912 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:20.999880    2543 generic.go:356] "Generic (PLEG): container finished" podID="83b4880a-83ba-41da-b9ef-78a2977ff2b0" containerID="444a71bd44e6f4e51482dbbf43bf5c93d5497a5a72af0119bbec7b9aa9ca266b" exitCode=0
Jul 28 17:30:21.000358 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:20.999951    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540npdqn" event={"ID":"83b4880a-83ba-41da-b9ef-78a2977ff2b0","Type":"ContainerDied","Data":"444a71bd44e6f4e51482dbbf43bf5c93d5497a5a72af0119bbec7b9aa9ca266b"}
Jul 28 17:30:22.005767 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:22.005730    2543 generic.go:356] "Generic (PLEG): container finished" podID="83b4880a-83ba-41da-b9ef-78a2977ff2b0" containerID="763d66033b2750f9ff49e5176acc11858f5deda92b482b0f5dad189bb10cceea" exitCode=0
Jul 28 17:30:22.006165 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:22.005817    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540npdqn" event={"ID":"83b4880a-83ba-41da-b9ef-78a2977ff2b0","Type":"ContainerDied","Data":"763d66033b2750f9ff49e5176acc11858f5deda92b482b0f5dad189bb10cceea"}
Jul 28 17:30:23.138063 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:23.138039    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540npdqn"
Jul 28 17:30:23.152815 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:23.152788    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkwdf\" (UniqueName: \"kubernetes.io/projected/83b4880a-83ba-41da-b9ef-78a2977ff2b0-kube-api-access-qkwdf\") pod \"83b4880a-83ba-41da-b9ef-78a2977ff2b0\" (UID: \"83b4880a-83ba-41da-b9ef-78a2977ff2b0\") "
Jul 28 17:30:23.152914 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:23.152822    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83b4880a-83ba-41da-b9ef-78a2977ff2b0-util\") pod \"83b4880a-83ba-41da-b9ef-78a2977ff2b0\" (UID: \"83b4880a-83ba-41da-b9ef-78a2977ff2b0\") "
Jul 28 17:30:23.152914 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:23.152861    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83b4880a-83ba-41da-b9ef-78a2977ff2b0-bundle\") pod \"83b4880a-83ba-41da-b9ef-78a2977ff2b0\" (UID: \"83b4880a-83ba-41da-b9ef-78a2977ff2b0\") "
Jul 28 17:30:23.153550 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:23.153524    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b4880a-83ba-41da-b9ef-78a2977ff2b0-bundle" (OuterVolumeSpecName: "bundle") pod "83b4880a-83ba-41da-b9ef-78a2977ff2b0" (UID: "83b4880a-83ba-41da-b9ef-78a2977ff2b0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:30:23.154738 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:23.154712    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b4880a-83ba-41da-b9ef-78a2977ff2b0-kube-api-access-qkwdf" (OuterVolumeSpecName: "kube-api-access-qkwdf") pod "83b4880a-83ba-41da-b9ef-78a2977ff2b0" (UID: "83b4880a-83ba-41da-b9ef-78a2977ff2b0"). InnerVolumeSpecName "kube-api-access-qkwdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:30:23.158303 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:23.158282    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b4880a-83ba-41da-b9ef-78a2977ff2b0-util" (OuterVolumeSpecName: "util") pod "83b4880a-83ba-41da-b9ef-78a2977ff2b0" (UID: "83b4880a-83ba-41da-b9ef-78a2977ff2b0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:30:23.254127 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:23.254092    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkwdf\" (UniqueName: \"kubernetes.io/projected/83b4880a-83ba-41da-b9ef-78a2977ff2b0-kube-api-access-qkwdf\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:30:23.254127 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:23.254119    2543 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83b4880a-83ba-41da-b9ef-78a2977ff2b0-util\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:30:23.254127 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:23.254128    2543 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83b4880a-83ba-41da-b9ef-78a2977ff2b0-bundle\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:30:24.016293 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:24.016260    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540npdqn" event={"ID":"83b4880a-83ba-41da-b9ef-78a2977ff2b0","Type":"ContainerDied","Data":"83f62fb0ca8c20581e780c14be9eef5d51d5d53b5fff1ba990a99e697e70a027"}
Jul 28 17:30:24.016460 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:24.016301    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540npdqn"
Jul 28 17:30:24.016460 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:24.016305    2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83f62fb0ca8c20581e780c14be9eef5d51d5d53b5fff1ba990a99e697e70a027"
Jul 28 17:30:36.592217 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:36.592176    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9wsxt"]
Jul 28 17:30:36.592809 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:36.592629    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83b4880a-83ba-41da-b9ef-78a2977ff2b0" containerName="util"
Jul 28 17:30:36.592809 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:36.592648    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="83b4880a-83ba-41da-b9ef-78a2977ff2b0" containerName="util"
Jul 28 17:30:36.592809 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:36.592664    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83b4880a-83ba-41da-b9ef-78a2977ff2b0" containerName="extract"
Jul 28 17:30:36.592809 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:36.592674    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="83b4880a-83ba-41da-b9ef-78a2977ff2b0" containerName="extract"
Jul 28 17:30:36.592809 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:36.592691    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83b4880a-83ba-41da-b9ef-78a2977ff2b0" containerName="pull"
Jul 28 17:30:36.592809 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:36.592700    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="83b4880a-83ba-41da-b9ef-78a2977ff2b0" containerName="pull"
Jul 28 17:30:36.592809 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:36.592776    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="83b4880a-83ba-41da-b9ef-78a2977ff2b0" containerName="extract"
Jul 28 17:30:36.597076 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:36.597053    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9wsxt"
Jul 28 17:30:36.599597 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:36.599573    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 28 17:30:36.600309 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:36.600296    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 28 17:30:36.600385 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:36.600297    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vmfb2\""
Jul 28 17:30:36.605071 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:36.605050    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9wsxt"]
Jul 28 17:30:36.661276 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:36.661245    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frvl5\" (UniqueName: \"kubernetes.io/projected/45818f79-2835-4d52-85a0-57217bcb8258-kube-api-access-frvl5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9wsxt\" (UID: \"45818f79-2835-4d52-85a0-57217bcb8258\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9wsxt"
Jul 28 17:30:36.661436 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:36.661311    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45818f79-2835-4d52-85a0-57217bcb8258-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9wsxt\" (UID: \"45818f79-2835-4d52-85a0-57217bcb8258\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9wsxt"
Jul 28 17:30:36.661436 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:36.661339    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45818f79-2835-4d52-85a0-57217bcb8258-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9wsxt\" (UID: \"45818f79-2835-4d52-85a0-57217bcb8258\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9wsxt"
Jul 28 17:30:36.762575 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:36.762534    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45818f79-2835-4d52-85a0-57217bcb8258-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9wsxt\" (UID: \"45818f79-2835-4d52-85a0-57217bcb8258\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9wsxt"
Jul 28 17:30:36.762696 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:36.762581    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45818f79-2835-4d52-85a0-57217bcb8258-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9wsxt\" (UID: \"45818f79-2835-4d52-85a0-57217bcb8258\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9wsxt"
Jul 28 17:30:36.762696 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:36.762627    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frvl5\" (UniqueName: \"kubernetes.io/projected/45818f79-2835-4d52-85a0-57217bcb8258-kube-api-access-frvl5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9wsxt\" (UID: \"45818f79-2835-4d52-85a0-57217bcb8258\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9wsxt"
Jul 28 17:30:36.762917 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:36.762897    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45818f79-2835-4d52-85a0-57217bcb8258-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9wsxt\" (UID: \"45818f79-2835-4d52-85a0-57217bcb8258\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9wsxt"
Jul 28 17:30:36.763007 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:36.762986    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45818f79-2835-4d52-85a0-57217bcb8258-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9wsxt\" (UID: \"45818f79-2835-4d52-85a0-57217bcb8258\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9wsxt"
Jul 28 17:30:36.771798 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:36.771773    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frvl5\" (UniqueName: \"kubernetes.io/projected/45818f79-2835-4d52-85a0-57217bcb8258-kube-api-access-frvl5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9wsxt\" (UID: \"45818f79-2835-4d52-85a0-57217bcb8258\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9wsxt"
Jul 28 17:30:36.906647 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:36.906565    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9wsxt"
Jul 28 17:30:37.041225 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:37.041200    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9wsxt"]
Jul 28 17:30:37.043240 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:30:37.043211    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45818f79_2835_4d52_85a0_57217bcb8258.slice/crio-b414a4067fc79e71545c434a1a9b7363dbc31a25d61a14b4bf7bb11ad3e3ef44 WatchSource:0}: Error finding container b414a4067fc79e71545c434a1a9b7363dbc31a25d61a14b4bf7bb11ad3e3ef44: Status 404 returned error can't find the container with id b414a4067fc79e71545c434a1a9b7363dbc31a25d61a14b4bf7bb11ad3e3ef44
Jul 28 17:30:37.075188 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:37.075160    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9wsxt" event={"ID":"45818f79-2835-4d52-85a0-57217bcb8258","Type":"ContainerStarted","Data":"b414a4067fc79e71545c434a1a9b7363dbc31a25d61a14b4bf7bb11ad3e3ef44"}
Jul 28 17:30:38.080869 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:38.080780    2543 generic.go:356] "Generic (PLEG): container finished" podID="45818f79-2835-4d52-85a0-57217bcb8258" containerID="c6ca1f8d932b732c0b1f6a73bef4c5b172326184f356113ba1caf2b031a5a547" exitCode=0
Jul 28 17:30:38.081329 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:30:38.080865    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9wsxt" event={"ID":"45818f79-2835-4d52-85a0-57217bcb8258","Type":"ContainerDied","Data":"c6ca1f8d932b732c0b1f6a73bef4c5b172326184f356113ba1caf2b031a5a547"}
Jul 28 17:31:01.185510 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:01.185479    2543 generic.go:356] "Generic (PLEG): container finished" podID="45818f79-2835-4d52-85a0-57217bcb8258" containerID="eb09832f847de981933759e60fdfb7a413c17563d4964bfd7893e424d0772b35" exitCode=0
Jul 28 17:31:01.185928 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:01.185550    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9wsxt" event={"ID":"45818f79-2835-4d52-85a0-57217bcb8258","Type":"ContainerDied","Data":"eb09832f847de981933759e60fdfb7a413c17563d4964bfd7893e424d0772b35"}
Jul 28 17:31:02.191889 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:02.191857    2543 generic.go:356] "Generic (PLEG): container finished" podID="45818f79-2835-4d52-85a0-57217bcb8258" containerID="673c6cf702d9519b471f5c174d28b6f29eb790e071e0fcc93ab3bf7c1267b495" exitCode=0
Jul 28 17:31:02.192344 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:02.191945    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9wsxt" event={"ID":"45818f79-2835-4d52-85a0-57217bcb8258","Type":"ContainerDied","Data":"673c6cf702d9519b471f5c174d28b6f29eb790e071e0fcc93ab3bf7c1267b495"}
Jul 28 17:31:03.318831 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:03.318805    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9wsxt"
Jul 28 17:31:03.488935 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:03.488905    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45818f79-2835-4d52-85a0-57217bcb8258-util\") pod \"45818f79-2835-4d52-85a0-57217bcb8258\" (UID: \"45818f79-2835-4d52-85a0-57217bcb8258\") "
Jul 28 17:31:03.489099 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:03.488960    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45818f79-2835-4d52-85a0-57217bcb8258-bundle\") pod \"45818f79-2835-4d52-85a0-57217bcb8258\" (UID: \"45818f79-2835-4d52-85a0-57217bcb8258\") "
Jul 28 17:31:03.489099 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:03.489005    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frvl5\" (UniqueName: \"kubernetes.io/projected/45818f79-2835-4d52-85a0-57217bcb8258-kube-api-access-frvl5\") pod \"45818f79-2835-4d52-85a0-57217bcb8258\" (UID: \"45818f79-2835-4d52-85a0-57217bcb8258\") "
Jul 28 17:31:03.489433 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:03.489404    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45818f79-2835-4d52-85a0-57217bcb8258-bundle" (OuterVolumeSpecName: "bundle") pod "45818f79-2835-4d52-85a0-57217bcb8258" (UID: "45818f79-2835-4d52-85a0-57217bcb8258"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:31:03.491078 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:03.491049    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45818f79-2835-4d52-85a0-57217bcb8258-kube-api-access-frvl5" (OuterVolumeSpecName: "kube-api-access-frvl5") pod "45818f79-2835-4d52-85a0-57217bcb8258" (UID: "45818f79-2835-4d52-85a0-57217bcb8258"). InnerVolumeSpecName "kube-api-access-frvl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:31:03.494207 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:03.494187    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45818f79-2835-4d52-85a0-57217bcb8258-util" (OuterVolumeSpecName: "util") pod "45818f79-2835-4d52-85a0-57217bcb8258" (UID: "45818f79-2835-4d52-85a0-57217bcb8258"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:31:03.590467 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:03.590438    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frvl5\" (UniqueName: \"kubernetes.io/projected/45818f79-2835-4d52-85a0-57217bcb8258-kube-api-access-frvl5\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:31:03.590467 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:03.590465    2543 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45818f79-2835-4d52-85a0-57217bcb8258-util\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:31:03.590639 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:03.590481    2543 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45818f79-2835-4d52-85a0-57217bcb8258-bundle\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:31:04.202373 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:04.202345    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9wsxt"
Jul 28 17:31:04.202533 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:04.202334    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9wsxt" event={"ID":"45818f79-2835-4d52-85a0-57217bcb8258","Type":"ContainerDied","Data":"b414a4067fc79e71545c434a1a9b7363dbc31a25d61a14b4bf7bb11ad3e3ef44"}
Jul 28 17:31:04.202533 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:04.202469    2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b414a4067fc79e71545c434a1a9b7363dbc31a25d61a14b4bf7bb11ad3e3ef44"
Jul 28 17:31:10.065595 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:10.065562    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z4vbw"]
Jul 28 17:31:10.065988 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:10.065862    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45818f79-2835-4d52-85a0-57217bcb8258" containerName="util"
Jul 28 17:31:10.065988 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:10.065872    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="45818f79-2835-4d52-85a0-57217bcb8258" containerName="util"
Jul 28 17:31:10.065988 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:10.065879    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45818f79-2835-4d52-85a0-57217bcb8258" containerName="pull"
Jul 28 17:31:10.065988 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:10.065884    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="45818f79-2835-4d52-85a0-57217bcb8258" containerName="pull"
Jul 28 17:31:10.065988 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:10.065895    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45818f79-2835-4d52-85a0-57217bcb8258" containerName="extract"
Jul 28 17:31:10.065988 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:10.065900    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="45818f79-2835-4d52-85a0-57217bcb8258" containerName="extract"
Jul 28 17:31:10.065988 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:10.065955    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="45818f79-2835-4d52-85a0-57217bcb8258" containerName="extract"
Jul 28 17:31:10.068872 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:10.068856    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z4vbw"
Jul 28 17:31:10.071810 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:10.071792    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 28 17:31:10.071890 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:10.071821    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:31:10.072931 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:10.072915    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-s8k8d\""
Jul 28 17:31:10.088112 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:10.088085    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z4vbw"]
Jul 28 17:31:10.245596 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:10.245565    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pl7x\" (UniqueName: \"kubernetes.io/projected/05119f32-e9b9-4557-806d-74dd2c9e2ad0-kube-api-access-2pl7x\") pod \"openshift-lws-operator-fd8ccff4c-z4vbw\" (UID: \"05119f32-e9b9-4557-806d-74dd2c9e2ad0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z4vbw"
Jul 28 17:31:10.245771 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:10.245635    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05119f32-e9b9-4557-806d-74dd2c9e2ad0-tmp\") pod \"openshift-lws-operator-fd8ccff4c-z4vbw\" (UID: \"05119f32-e9b9-4557-806d-74dd2c9e2ad0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z4vbw"
Jul 28 17:31:10.346248 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:10.346167    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pl7x\" (UniqueName: \"kubernetes.io/projected/05119f32-e9b9-4557-806d-74dd2c9e2ad0-kube-api-access-2pl7x\") pod \"openshift-lws-operator-fd8ccff4c-z4vbw\" (UID: \"05119f32-e9b9-4557-806d-74dd2c9e2ad0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z4vbw"
Jul 28 17:31:10.346400 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:10.346261    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05119f32-e9b9-4557-806d-74dd2c9e2ad0-tmp\") pod \"openshift-lws-operator-fd8ccff4c-z4vbw\" (UID: \"05119f32-e9b9-4557-806d-74dd2c9e2ad0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z4vbw"
Jul 28 17:31:10.346577 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:10.346554    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05119f32-e9b9-4557-806d-74dd2c9e2ad0-tmp\") pod \"openshift-lws-operator-fd8ccff4c-z4vbw\" (UID: \"05119f32-e9b9-4557-806d-74dd2c9e2ad0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z4vbw"
Jul 28 17:31:10.361026 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:10.360991    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pl7x\" (UniqueName: \"kubernetes.io/projected/05119f32-e9b9-4557-806d-74dd2c9e2ad0-kube-api-access-2pl7x\") pod \"openshift-lws-operator-fd8ccff4c-z4vbw\" (UID: \"05119f32-e9b9-4557-806d-74dd2c9e2ad0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z4vbw"
Jul 28 17:31:10.391884 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:10.391835    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z4vbw"
Jul 28 17:31:10.523943 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:10.523918    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z4vbw"]
Jul 28 17:31:10.526241 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:31:10.526211    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05119f32_e9b9_4557_806d_74dd2c9e2ad0.slice/crio-89f381df0bbe13f2691dbf42280e587b6d17bf13a73f8c543c393f42858c01f1 WatchSource:0}: Error finding container 89f381df0bbe13f2691dbf42280e587b6d17bf13a73f8c543c393f42858c01f1: Status 404 returned error can't find the container with id 89f381df0bbe13f2691dbf42280e587b6d17bf13a73f8c543c393f42858c01f1
Jul 28 17:31:11.233719 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:11.233681    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z4vbw" event={"ID":"05119f32-e9b9-4557-806d-74dd2c9e2ad0","Type":"ContainerStarted","Data":"89f381df0bbe13f2691dbf42280e587b6d17bf13a73f8c543c393f42858c01f1"}
Jul 28 17:31:12.239877 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:12.239845    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z4vbw" event={"ID":"05119f32-e9b9-4557-806d-74dd2c9e2ad0","Type":"ContainerStarted","Data":"030051241c95f2df076f21ebcea7c35664a2334493a488beb1823efc9bcd7aa2"}
Jul 28 17:31:12.258248 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:12.258203    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z4vbw" podStartSLOduration=0.681795383 podStartE2EDuration="2.258185042s" totalImagesPullingTime="1.576389659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:31:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:31:12.25633295 +0000 UTC m=+475.662606690" watchObservedRunningTime="2026-07-28 17:31:12.258185042 +0000 UTC m=+475.664458790"
Jul 28 17:31:24.313898 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:24.313864    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-fd6565d47-w5jsc"]
Jul 28 17:31:24.317298 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:24.317281    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-fd6565d47-w5jsc"
Jul 28 17:31:24.321212 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:24.321190    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 28 17:31:24.321399 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:24.321382    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 28 17:31:24.321443 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:24.321424    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-7lw22\""
Jul 28 17:31:24.321514 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:24.321497    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 28 17:31:24.334491 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:24.334466    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-fd6565d47-w5jsc"]
Jul 28 17:31:24.372317 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:24.372288    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/44f15402-5b8d-4073-a77e-c32c4ac33d23-metrics-cert\") pod \"lws-controller-manager-fd6565d47-w5jsc\" (UID: \"44f15402-5b8d-4073-a77e-c32c4ac33d23\") " pod="openshift-lws-operator/lws-controller-manager-fd6565d47-w5jsc"
Jul 28 17:31:24.372447 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:24.372341    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44f15402-5b8d-4073-a77e-c32c4ac33d23-cert\") pod \"lws-controller-manager-fd6565d47-w5jsc\" (UID: \"44f15402-5b8d-4073-a77e-c32c4ac33d23\") " pod="openshift-lws-operator/lws-controller-manager-fd6565d47-w5jsc"
Jul 28 17:31:24.372447 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:24.372372    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/44f15402-5b8d-4073-a77e-c32c4ac33d23-manager-config\") pod \"lws-controller-manager-fd6565d47-w5jsc\" (UID: \"44f15402-5b8d-4073-a77e-c32c4ac33d23\") " pod="openshift-lws-operator/lws-controller-manager-fd6565d47-w5jsc"
Jul 28 17:31:24.372447 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:24.372403    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj9fx\" (UniqueName: \"kubernetes.io/projected/44f15402-5b8d-4073-a77e-c32c4ac33d23-kube-api-access-tj9fx\") pod \"lws-controller-manager-fd6565d47-w5jsc\" (UID: \"44f15402-5b8d-4073-a77e-c32c4ac33d23\") " pod="openshift-lws-operator/lws-controller-manager-fd6565d47-w5jsc"
Jul 28 17:31:24.473112 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:24.473086    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj9fx\" (UniqueName: \"kubernetes.io/projected/44f15402-5b8d-4073-a77e-c32c4ac33d23-kube-api-access-tj9fx\") pod \"lws-controller-manager-fd6565d47-w5jsc\" (UID: \"44f15402-5b8d-4073-a77e-c32c4ac33d23\") " pod="openshift-lws-operator/lws-controller-manager-fd6565d47-w5jsc"
Jul 28 17:31:24.473256 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:24.473147    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/44f15402-5b8d-4073-a77e-c32c4ac33d23-metrics-cert\") pod \"lws-controller-manager-fd6565d47-w5jsc\" (UID: \"44f15402-5b8d-4073-a77e-c32c4ac33d23\") " pod="openshift-lws-operator/lws-controller-manager-fd6565d47-w5jsc"
Jul 28 17:31:24.473256 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:24.473193    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44f15402-5b8d-4073-a77e-c32c4ac33d23-cert\") pod \"lws-controller-manager-fd6565d47-w5jsc\" (UID: \"44f15402-5b8d-4073-a77e-c32c4ac33d23\") " pod="openshift-lws-operator/lws-controller-manager-fd6565d47-w5jsc"
Jul 28 17:31:24.473256 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:24.473215    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/44f15402-5b8d-4073-a77e-c32c4ac33d23-manager-config\") pod \"lws-controller-manager-fd6565d47-w5jsc\" (UID: \"44f15402-5b8d-4073-a77e-c32c4ac33d23\") " pod="openshift-lws-operator/lws-controller-manager-fd6565d47-w5jsc"
Jul 28 17:31:24.473777 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:24.473753    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/44f15402-5b8d-4073-a77e-c32c4ac33d23-manager-config\") pod \"lws-controller-manager-fd6565d47-w5jsc\" (UID: \"44f15402-5b8d-4073-a77e-c32c4ac33d23\") " pod="openshift-lws-operator/lws-controller-manager-fd6565d47-w5jsc"
Jul 28 17:31:24.475680 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:24.475656    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/44f15402-5b8d-4073-a77e-c32c4ac33d23-metrics-cert\") pod \"lws-controller-manager-fd6565d47-w5jsc\" (UID: \"44f15402-5b8d-4073-a77e-c32c4ac33d23\") " pod="openshift-lws-operator/lws-controller-manager-fd6565d47-w5jsc"
Jul 28 17:31:24.475767 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:24.475723    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44f15402-5b8d-4073-a77e-c32c4ac33d23-cert\") pod \"lws-controller-manager-fd6565d47-w5jsc\" (UID: \"44f15402-5b8d-4073-a77e-c32c4ac33d23\") " pod="openshift-lws-operator/lws-controller-manager-fd6565d47-w5jsc"
Jul 28 17:31:24.482652 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:24.482631    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj9fx\" (UniqueName: \"kubernetes.io/projected/44f15402-5b8d-4073-a77e-c32c4ac33d23-kube-api-access-tj9fx\") pod \"lws-controller-manager-fd6565d47-w5jsc\" (UID: \"44f15402-5b8d-4073-a77e-c32c4ac33d23\") " pod="openshift-lws-operator/lws-controller-manager-fd6565d47-w5jsc"
Jul 28 17:31:24.627222 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:24.627147    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-fd6565d47-w5jsc"
Jul 28 17:31:24.796552 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:24.796522    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-fd6565d47-w5jsc"]
Jul 28 17:31:24.799367 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:31:24.799337    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f15402_5b8d_4073_a77e_c32c4ac33d23.slice/crio-2626d705b5ca3fa959cac8e94e5fab1a914be5e38384325804589ae0d4494bac WatchSource:0}: Error finding container 2626d705b5ca3fa959cac8e94e5fab1a914be5e38384325804589ae0d4494bac: Status 404 returned error can't find the container with id 2626d705b5ca3fa959cac8e94e5fab1a914be5e38384325804589ae0d4494bac
Jul 28 17:31:25.298710 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:25.298678    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-fd6565d47-w5jsc" event={"ID":"44f15402-5b8d-4073-a77e-c32c4ac33d23","Type":"ContainerStarted","Data":"2626d705b5ca3fa959cac8e94e5fab1a914be5e38384325804589ae0d4494bac"}
Jul 28 17:31:26.305534 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:26.305498    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-fd6565d47-w5jsc" event={"ID":"44f15402-5b8d-4073-a77e-c32c4ac33d23","Type":"ContainerStarted","Data":"1cf175a57396c91aeb66ced01690e96259b8d9af19f4d9cee3a68452ea1e4159"}
Jul 28 17:31:26.305939 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:26.305660    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-fd6565d47-w5jsc"
Jul 28 17:31:26.323715 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:26.323668    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-fd6565d47-w5jsc" podStartSLOduration=0.925563844 podStartE2EDuration="2.323653473s" totalImagesPullingTime="1.398089629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:31:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:31:26.321827124 +0000 UTC m=+489.728100920" watchObservedRunningTime="2026-07-28 17:31:26.323653473 +0000 UTC m=+489.729927214"
Jul 28 17:31:37.312017 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:37.311939    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-fd6565d47-w5jsc"
Jul 28 17:31:41.936223 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:41.936186    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4rmrw"]
Jul 28 17:31:41.938522 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:41.938507    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4rmrw"
Jul 28 17:31:41.942555 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:41.942526    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 28 17:31:41.943510 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:41.943490    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 28 17:31:41.943591 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:41.943518    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vmfb2\""
Jul 28 17:31:41.960782 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:41.960756    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4rmrw"]
Jul 28 17:31:42.022923 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.022891    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e73880e-64eb-4e56-8642-d132ee1eaf96-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4rmrw\" (UID: \"2e73880e-64eb-4e56-8642-d132ee1eaf96\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4rmrw"
Jul 28 17:31:42.023095 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.022932    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e73880e-64eb-4e56-8642-d132ee1eaf96-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4rmrw\" (UID: \"2e73880e-64eb-4e56-8642-d132ee1eaf96\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4rmrw"
Jul 28 17:31:42.023095 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.022963    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbbfh\" (UniqueName: \"kubernetes.io/projected/2e73880e-64eb-4e56-8642-d132ee1eaf96-kube-api-access-rbbfh\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4rmrw\" (UID: \"2e73880e-64eb-4e56-8642-d132ee1eaf96\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4rmrw"
Jul 28 17:31:42.062416 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.062383    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jzzvr"]
Jul 28 17:31:42.064920 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.064904    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jzzvr"
Jul 28 17:31:42.080247 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.080216    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jzzvr"]
Jul 28 17:31:42.123961 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.123931    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbbfh\" (UniqueName: \"kubernetes.io/projected/2e73880e-64eb-4e56-8642-d132ee1eaf96-kube-api-access-rbbfh\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4rmrw\" (UID: \"2e73880e-64eb-4e56-8642-d132ee1eaf96\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4rmrw"
Jul 28 17:31:42.124214 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.124015    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e73880e-64eb-4e56-8642-d132ee1eaf96-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4rmrw\" (UID: \"2e73880e-64eb-4e56-8642-d132ee1eaf96\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4rmrw"
Jul 28 17:31:42.124214 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.124035    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e73880e-64eb-4e56-8642-d132ee1eaf96-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4rmrw\" (UID: \"2e73880e-64eb-4e56-8642-d132ee1eaf96\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4rmrw"
Jul 28 17:31:42.124377 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.124360    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e73880e-64eb-4e56-8642-d132ee1eaf96-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4rmrw\" (UID: \"2e73880e-64eb-4e56-8642-d132ee1eaf96\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4rmrw"
Jul 28 17:31:42.124469 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.124450    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e73880e-64eb-4e56-8642-d132ee1eaf96-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4rmrw\" (UID: \"2e73880e-64eb-4e56-8642-d132ee1eaf96\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4rmrw"
Jul 28 17:31:42.138031 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.137999    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbbfh\" (UniqueName: \"kubernetes.io/projected/2e73880e-64eb-4e56-8642-d132ee1eaf96-kube-api-access-rbbfh\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4rmrw\" (UID: \"2e73880e-64eb-4e56-8642-d132ee1eaf96\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4rmrw"
Jul 28 17:31:42.156665 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.156640    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4l77x"]
Jul 28 17:31:42.158951 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.158936    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4l77x"
Jul 28 17:31:42.171484 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.171457    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4l77x"]
Jul 28 17:31:42.224878 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.224724    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf801-bde5-47aa-b8e6-45b31a925ad9-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4l77x\" (UID: \"5fdcf801-bde5-47aa-b8e6-45b31a925ad9\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4l77x"
Jul 28 17:31:42.224878 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.224783    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgfb4\" (UniqueName: \"kubernetes.io/projected/e227d129-ef64-4711-a61a-3d2c20d37d8c-kube-api-access-sgfb4\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jzzvr\" (UID: \"e227d129-ef64-4711-a61a-3d2c20d37d8c\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jzzvr"
Jul 28 17:31:42.224878 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.224858    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e227d129-ef64-4711-a61a-3d2c20d37d8c-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jzzvr\" (UID: \"e227d129-ef64-4711-a61a-3d2c20d37d8c\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jzzvr"
Jul 28 17:31:42.225174 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.224905    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf801-bde5-47aa-b8e6-45b31a925ad9-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4l77x\" (UID: \"5fdcf801-bde5-47aa-b8e6-45b31a925ad9\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4l77x"
Jul 28 17:31:42.225174 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.225006    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e227d129-ef64-4711-a61a-3d2c20d37d8c-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jzzvr\" (UID: \"e227d129-ef64-4711-a61a-3d2c20d37d8c\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jzzvr"
Jul 28 17:31:42.225174 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.225041    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgdts\" (UniqueName: \"kubernetes.io/projected/5fdcf801-bde5-47aa-b8e6-45b31a925ad9-kube-api-access-tgdts\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4l77x\" (UID: \"5fdcf801-bde5-47aa-b8e6-45b31a925ad9\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4l77x"
Jul 28 17:31:42.234952 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.234925    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qt7lt"]
Jul 28 17:31:42.237287 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.237273    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qt7lt"
Jul 28 17:31:42.247446 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.247389    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4rmrw"
Jul 28 17:31:42.249583 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.249344    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qt7lt"]
Jul 28 17:31:42.325885 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.325845    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e227d129-ef64-4711-a61a-3d2c20d37d8c-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jzzvr\" (UID: \"e227d129-ef64-4711-a61a-3d2c20d37d8c\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jzzvr"
Jul 28 17:31:42.326037 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.325903    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksjwp\" (UniqueName: \"kubernetes.io/projected/99402a99-2dc8-4e79-85a4-7f41a4bdc078-kube-api-access-ksjwp\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qt7lt\" (UID: \"99402a99-2dc8-4e79-85a4-7f41a4bdc078\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qt7lt"
Jul 28 17:31:42.326037 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.325936    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf801-bde5-47aa-b8e6-45b31a925ad9-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4l77x\" (UID: \"5fdcf801-bde5-47aa-b8e6-45b31a925ad9\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4l77x"
Jul 28 17:31:42.326037 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.325964    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99402a99-2dc8-4e79-85a4-7f41a4bdc078-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qt7lt\" (UID: \"99402a99-2dc8-4e79-85a4-7f41a4bdc078\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qt7lt"
Jul 28 17:31:42.326195 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.326065    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e227d129-ef64-4711-a61a-3d2c20d37d8c-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jzzvr\" (UID: \"e227d129-ef64-4711-a61a-3d2c20d37d8c\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jzzvr"
Jul 28 17:31:42.326195 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.326110    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgdts\" (UniqueName: \"kubernetes.io/projected/5fdcf801-bde5-47aa-b8e6-45b31a925ad9-kube-api-access-tgdts\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4l77x\" (UID: \"5fdcf801-bde5-47aa-b8e6-45b31a925ad9\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4l77x"
Jul 28 17:31:42.326291 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.326211    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf801-bde5-47aa-b8e6-45b31a925ad9-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4l77x\" (UID: \"5fdcf801-bde5-47aa-b8e6-45b31a925ad9\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4l77x"
Jul 28 17:31:42.326291 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.326244    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99402a99-2dc8-4e79-85a4-7f41a4bdc078-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qt7lt\" (UID: \"99402a99-2dc8-4e79-85a4-7f41a4bdc078\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qt7lt"
Jul 28 17:31:42.326291 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.326284    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgfb4\" (UniqueName: \"kubernetes.io/projected/e227d129-ef64-4711-a61a-3d2c20d37d8c-kube-api-access-sgfb4\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jzzvr\" (UID: \"e227d129-ef64-4711-a61a-3d2c20d37d8c\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jzzvr"
Jul 28 17:31:42.326482 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.326286    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e227d129-ef64-4711-a61a-3d2c20d37d8c-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jzzvr\" (UID: \"e227d129-ef64-4711-a61a-3d2c20d37d8c\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jzzvr"
Jul 28 17:31:42.326482 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.326346    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e227d129-ef64-4711-a61a-3d2c20d37d8c-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jzzvr\" (UID: \"e227d129-ef64-4711-a61a-3d2c20d37d8c\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jzzvr"
Jul 28 17:31:42.326482 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.326401    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf801-bde5-47aa-b8e6-45b31a925ad9-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4l77x\" (UID: \"5fdcf801-bde5-47aa-b8e6-45b31a925ad9\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4l77x"
Jul 28 17:31:42.326482 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.326465    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf801-bde5-47aa-b8e6-45b31a925ad9-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4l77x\" (UID: \"5fdcf801-bde5-47aa-b8e6-45b31a925ad9\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4l77x"
Jul 28 17:31:42.334703 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.334664    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgfb4\" (UniqueName: \"kubernetes.io/projected/e227d129-ef64-4711-a61a-3d2c20d37d8c-kube-api-access-sgfb4\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jzzvr\" (UID: \"e227d129-ef64-4711-a61a-3d2c20d37d8c\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jzzvr"
Jul 28 17:31:42.335634 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.335610    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgdts\" (UniqueName: \"kubernetes.io/projected/5fdcf801-bde5-47aa-b8e6-45b31a925ad9-kube-api-access-tgdts\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4l77x\" (UID: \"5fdcf801-bde5-47aa-b8e6-45b31a925ad9\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4l77x"
Jul 28 17:31:42.373566 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.373538    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jzzvr"
Jul 28 17:31:42.381309 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.381285    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4rmrw"]
Jul 28 17:31:42.382836 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:31:42.382812    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e73880e_64eb_4e56_8642_d132ee1eaf96.slice/crio-819aeae23c53e8c87be31b2a31601c5c3ad15efe857c8252c03f9e92df977a6c WatchSource:0}: Error finding container 819aeae23c53e8c87be31b2a31601c5c3ad15efe857c8252c03f9e92df977a6c: Status 404 returned error can't find the container with id 819aeae23c53e8c87be31b2a31601c5c3ad15efe857c8252c03f9e92df977a6c
Jul 28 17:31:42.427371 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.427341    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99402a99-2dc8-4e79-85a4-7f41a4bdc078-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qt7lt\" (UID: \"99402a99-2dc8-4e79-85a4-7f41a4bdc078\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qt7lt"
Jul 28 17:31:42.427620 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.427428    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksjwp\" (UniqueName: \"kubernetes.io/projected/99402a99-2dc8-4e79-85a4-7f41a4bdc078-kube-api-access-ksjwp\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qt7lt\" (UID: \"99402a99-2dc8-4e79-85a4-7f41a4bdc078\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qt7lt"
Jul 28 17:31:42.427620 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.427458    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99402a99-2dc8-4e79-85a4-7f41a4bdc078-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qt7lt\" (UID: \"99402a99-2dc8-4e79-85a4-7f41a4bdc078\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qt7lt"
Jul 28 17:31:42.428122 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.428078    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99402a99-2dc8-4e79-85a4-7f41a4bdc078-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qt7lt\" (UID: \"99402a99-2dc8-4e79-85a4-7f41a4bdc078\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qt7lt"
Jul 28 17:31:42.428122 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.428104    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99402a99-2dc8-4e79-85a4-7f41a4bdc078-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qt7lt\" (UID: \"99402a99-2dc8-4e79-85a4-7f41a4bdc078\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qt7lt"
Jul 28 17:31:42.436814 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.436789    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksjwp\" (UniqueName: \"kubernetes.io/projected/99402a99-2dc8-4e79-85a4-7f41a4bdc078-kube-api-access-ksjwp\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qt7lt\" (UID: \"99402a99-2dc8-4e79-85a4-7f41a4bdc078\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qt7lt"
Jul 28 17:31:42.467922 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.467887    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4l77x"
Jul 28 17:31:42.522879 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.522849    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jzzvr"]
Jul 28 17:31:42.545742 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.545710    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qt7lt"
Jul 28 17:31:42.546053 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:31:42.546026    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode227d129_ef64_4711_a61a_3d2c20d37d8c.slice/crio-aa2d5bb8e7c163776eaaa8d26e053f33dd6c4fa05d7a8cb65c3d5055c8d8bd87 WatchSource:0}: Error finding container aa2d5bb8e7c163776eaaa8d26e053f33dd6c4fa05d7a8cb65c3d5055c8d8bd87: Status 404 returned error can't find the container with id aa2d5bb8e7c163776eaaa8d26e053f33dd6c4fa05d7a8cb65c3d5055c8d8bd87
Jul 28 17:31:42.618749 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.618588    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4l77x"]
Jul 28 17:31:42.622351 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:31:42.622310    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fdcf801_bde5_47aa_b8e6_45b31a925ad9.slice/crio-e7632629aeae1da4c50cf804f42330c920be77c74c10973be24a29423fbd85a5 WatchSource:0}: Error finding container e7632629aeae1da4c50cf804f42330c920be77c74c10973be24a29423fbd85a5: Status 404 returned error can't find the container with id e7632629aeae1da4c50cf804f42330c920be77c74c10973be24a29423fbd85a5
Jul 28 17:31:42.700262 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:42.700198    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qt7lt"]
Jul 28 17:31:42.743622 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:31:42.743599    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99402a99_2dc8_4e79_85a4_7f41a4bdc078.slice/crio-cee2628b54b4831376d1d0629220053567ef90da4b5b195a6715eb5a150421fc WatchSource:0}: Error finding container cee2628b54b4831376d1d0629220053567ef90da4b5b195a6715eb5a150421fc: Status 404 returned error can't find the container with id cee2628b54b4831376d1d0629220053567ef90da4b5b195a6715eb5a150421fc
Jul 28 17:31:43.390621 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:43.390586    2543 generic.go:356] "Generic (PLEG): container finished" podID="2e73880e-64eb-4e56-8642-d132ee1eaf96" containerID="51cd710b5e6bfbc0c5cbf2db8f9edd169a7511a5572ae938aca1d4530eb69bce" exitCode=0
Jul 28 17:31:43.391051 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:43.390676    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4rmrw" event={"ID":"2e73880e-64eb-4e56-8642-d132ee1eaf96","Type":"ContainerDied","Data":"51cd710b5e6bfbc0c5cbf2db8f9edd169a7511a5572ae938aca1d4530eb69bce"}
Jul 28 17:31:43.391051 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:43.390735    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4rmrw" event={"ID":"2e73880e-64eb-4e56-8642-d132ee1eaf96","Type":"ContainerStarted","Data":"819aeae23c53e8c87be31b2a31601c5c3ad15efe857c8252c03f9e92df977a6c"}
Jul 28 17:31:43.392258 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:43.392238    2543 generic.go:356] "Generic (PLEG): container finished" podID="5fdcf801-bde5-47aa-b8e6-45b31a925ad9" containerID="1c6a3c2ecffd9dd9fbe08ecadde038405c9afebf29b870c4580544bda199199f" exitCode=0
Jul 28 17:31:43.392316 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:43.392296    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4l77x" event={"ID":"5fdcf801-bde5-47aa-b8e6-45b31a925ad9","Type":"ContainerDied","Data":"1c6a3c2ecffd9dd9fbe08ecadde038405c9afebf29b870c4580544bda199199f"}
Jul 28 17:31:43.392369 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:43.392343    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4l77x" event={"ID":"5fdcf801-bde5-47aa-b8e6-45b31a925ad9","Type":"ContainerStarted","Data":"e7632629aeae1da4c50cf804f42330c920be77c74c10973be24a29423fbd85a5"}
Jul 28 17:31:43.393983 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:43.393956    2543 generic.go:356] "Generic (PLEG): container finished" podID="99402a99-2dc8-4e79-85a4-7f41a4bdc078" containerID="f5816c66db682c0ad15c23e47422dfb37cba966a6ca8bd0ba86b16f22b42b310" exitCode=0
Jul 28 17:31:43.394102 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:43.393998    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qt7lt" event={"ID":"99402a99-2dc8-4e79-85a4-7f41a4bdc078","Type":"ContainerDied","Data":"f5816c66db682c0ad15c23e47422dfb37cba966a6ca8bd0ba86b16f22b42b310"}
Jul 28 17:31:43.394102 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:43.394041    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qt7lt" event={"ID":"99402a99-2dc8-4e79-85a4-7f41a4bdc078","Type":"ContainerStarted","Data":"cee2628b54b4831376d1d0629220053567ef90da4b5b195a6715eb5a150421fc"}
Jul 28 17:31:43.395999 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:43.395901    2543 generic.go:356] "Generic (PLEG): container finished" podID="e227d129-ef64-4711-a61a-3d2c20d37d8c" containerID="6e393cfab6cf5026d8bcac67ffcdc1e7998979f855071415a1d988cabadc1eb5" exitCode=0
Jul 28 17:31:43.395999 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:43.395954    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jzzvr" event={"ID":"e227d129-ef64-4711-a61a-3d2c20d37d8c","Type":"ContainerDied","Data":"6e393cfab6cf5026d8bcac67ffcdc1e7998979f855071415a1d988cabadc1eb5"}
Jul 28 17:31:43.396154 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:43.396004    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jzzvr" event={"ID":"e227d129-ef64-4711-a61a-3d2c20d37d8c","Type":"ContainerStarted","Data":"aa2d5bb8e7c163776eaaa8d26e053f33dd6c4fa05d7a8cb65c3d5055c8d8bd87"}
Jul 28 17:31:44.402498 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:44.402449    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4rmrw" event={"ID":"2e73880e-64eb-4e56-8642-d132ee1eaf96","Type":"ContainerStarted","Data":"d085ea88431c3e8a0be8b3d4f29ce40bba832d7c184f9d6dc030278dd091d0e0"}
Jul 28 17:31:44.404703 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:44.404676    2543 generic.go:356] "Generic (PLEG): container finished" podID="5fdcf801-bde5-47aa-b8e6-45b31a925ad9" containerID="020f3feac23e957b26dc356fb6f1b6dce85a8b4ec997ae052c2400f5a3379845" exitCode=0
Jul 28 17:31:44.404832 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:44.404760    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4l77x" event={"ID":"5fdcf801-bde5-47aa-b8e6-45b31a925ad9","Type":"ContainerDied","Data":"020f3feac23e957b26dc356fb6f1b6dce85a8b4ec997ae052c2400f5a3379845"}
Jul 28 17:31:44.407368 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:44.407341    2543 generic.go:356] "Generic (PLEG): container finished" podID="99402a99-2dc8-4e79-85a4-7f41a4bdc078" containerID="a0dbdef6b6fe3b2fa74d472cbaaf5f499e9806ee16b250c1dd74b673d165079f" exitCode=0
Jul 28 17:31:44.407637 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:44.407424    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qt7lt" event={"ID":"99402a99-2dc8-4e79-85a4-7f41a4bdc078","Type":"ContainerDied","Data":"a0dbdef6b6fe3b2fa74d472cbaaf5f499e9806ee16b250c1dd74b673d165079f"}
Jul 28 17:31:44.409421 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:44.409401    2543 generic.go:356] "Generic (PLEG): container finished" podID="e227d129-ef64-4711-a61a-3d2c20d37d8c" containerID="0be671d202547eba492bc75ac557249d1dbb05b83016bfb087a6afa1e0de8568" exitCode=0
Jul 28 17:31:44.409507 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:44.409433    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jzzvr" event={"ID":"e227d129-ef64-4711-a61a-3d2c20d37d8c","Type":"ContainerDied","Data":"0be671d202547eba492bc75ac557249d1dbb05b83016bfb087a6afa1e0de8568"}
Jul 28 17:31:45.415752 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:45.415719    2543 generic.go:356] "Generic (PLEG): container finished" podID="5fdcf801-bde5-47aa-b8e6-45b31a925ad9" containerID="5d946632a57aa17844b83bcf878c96a2997e8471c92f10dcf04c12868d525828" exitCode=0
Jul 28 17:31:45.416184 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:45.415798    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4l77x" event={"ID":"5fdcf801-bde5-47aa-b8e6-45b31a925ad9","Type":"ContainerDied","Data":"5d946632a57aa17844b83bcf878c96a2997e8471c92f10dcf04c12868d525828"}
Jul 28 17:31:45.417821 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:45.417797    2543 generic.go:356] "Generic (PLEG): container finished" podID="99402a99-2dc8-4e79-85a4-7f41a4bdc078" containerID="de7107216d89e431d20ed400fa644f1a4a78e0496594d476fad30776a37690bb" exitCode=0
Jul 28 17:31:45.417925 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:45.417833    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qt7lt" event={"ID":"99402a99-2dc8-4e79-85a4-7f41a4bdc078","Type":"ContainerDied","Data":"de7107216d89e431d20ed400fa644f1a4a78e0496594d476fad30776a37690bb"}
Jul 28 17:31:45.419919 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:45.419898    2543 generic.go:356] "Generic (PLEG): container finished" podID="e227d129-ef64-4711-a61a-3d2c20d37d8c" containerID="9dc7fd52d02774cddc25ef2204f9fe88ede190fbe37b5db6c88d9bb90b8ba6e1" exitCode=0
Jul 28 17:31:45.420005 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:45.419928    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jzzvr" event={"ID":"e227d129-ef64-4711-a61a-3d2c20d37d8c","Type":"ContainerDied","Data":"9dc7fd52d02774cddc25ef2204f9fe88ede190fbe37b5db6c88d9bb90b8ba6e1"}
Jul 28 17:31:45.421560 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:45.421536    2543 generic.go:356] "Generic (PLEG): container finished" podID="2e73880e-64eb-4e56-8642-d132ee1eaf96" containerID="d085ea88431c3e8a0be8b3d4f29ce40bba832d7c184f9d6dc030278dd091d0e0" exitCode=0
Jul 28 17:31:45.421660 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:45.421560    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4rmrw" event={"ID":"2e73880e-64eb-4e56-8642-d132ee1eaf96","Type":"ContainerDied","Data":"d085ea88431c3e8a0be8b3d4f29ce40bba832d7c184f9d6dc030278dd091d0e0"}
Jul 28 17:31:46.428580 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:46.428551    2543 generic.go:356] "Generic (PLEG): container finished" podID="2e73880e-64eb-4e56-8642-d132ee1eaf96" containerID="c102ceb80a2c452f10cd6c28f4811931a7b487efde827186dcbfee17272905cb" exitCode=0
Jul 28 17:31:46.428961 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:46.428624    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4rmrw" event={"ID":"2e73880e-64eb-4e56-8642-d132ee1eaf96","Type":"ContainerDied","Data":"c102ceb80a2c452f10cd6c28f4811931a7b487efde827186dcbfee17272905cb"}
Jul 28 17:31:46.579515 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:46.579492    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jzzvr"
Jul 28 17:31:46.617948 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:46.617922    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4l77x"
Jul 28 17:31:46.621002 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:46.620982    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qt7lt"
Jul 28 17:31:46.665054 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:46.665024    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e227d129-ef64-4711-a61a-3d2c20d37d8c-util\") pod \"e227d129-ef64-4711-a61a-3d2c20d37d8c\" (UID: \"e227d129-ef64-4711-a61a-3d2c20d37d8c\") "
Jul 28 17:31:46.665225 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:46.665089    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e227d129-ef64-4711-a61a-3d2c20d37d8c-bundle\") pod \"e227d129-ef64-4711-a61a-3d2c20d37d8c\" (UID: \"e227d129-ef64-4711-a61a-3d2c20d37d8c\") "
Jul 28 17:31:46.665225 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:46.665197    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgfb4\" (UniqueName: \"kubernetes.io/projected/e227d129-ef64-4711-a61a-3d2c20d37d8c-kube-api-access-sgfb4\") pod \"e227d129-ef64-4711-a61a-3d2c20d37d8c\" (UID: \"e227d129-ef64-4711-a61a-3d2c20d37d8c\") "
Jul 28 17:31:46.665785 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:46.665756    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e227d129-ef64-4711-a61a-3d2c20d37d8c-bundle" (OuterVolumeSpecName: "bundle") pod "e227d129-ef64-4711-a61a-3d2c20d37d8c" (UID: "e227d129-ef64-4711-a61a-3d2c20d37d8c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:31:46.667258 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:46.667233    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e227d129-ef64-4711-a61a-3d2c20d37d8c-kube-api-access-sgfb4" (OuterVolumeSpecName: "kube-api-access-sgfb4") pod "e227d129-ef64-4711-a61a-3d2c20d37d8c" (UID: "e227d129-ef64-4711-a61a-3d2c20d37d8c"). InnerVolumeSpecName "kube-api-access-sgfb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:31:46.672404 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:46.672364    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e227d129-ef64-4711-a61a-3d2c20d37d8c-util" (OuterVolumeSpecName: "util") pod "e227d129-ef64-4711-a61a-3d2c20d37d8c" (UID: "e227d129-ef64-4711-a61a-3d2c20d37d8c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:31:46.766193 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:46.766163    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksjwp\" (UniqueName: \"kubernetes.io/projected/99402a99-2dc8-4e79-85a4-7f41a4bdc078-kube-api-access-ksjwp\") pod \"99402a99-2dc8-4e79-85a4-7f41a4bdc078\" (UID: \"99402a99-2dc8-4e79-85a4-7f41a4bdc078\") "
Jul 28 17:31:46.766337 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:46.766227    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf801-bde5-47aa-b8e6-45b31a925ad9-bundle\") pod \"5fdcf801-bde5-47aa-b8e6-45b31a925ad9\" (UID: \"5fdcf801-bde5-47aa-b8e6-45b31a925ad9\") "
Jul 28 17:31:46.766337 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:46.766256    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgdts\" (UniqueName: \"kubernetes.io/projected/5fdcf801-bde5-47aa-b8e6-45b31a925ad9-kube-api-access-tgdts\") pod \"5fdcf801-bde5-47aa-b8e6-45b31a925ad9\" (UID: \"5fdcf801-bde5-47aa-b8e6-45b31a925ad9\") "
Jul 28 17:31:46.766337 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:46.766281    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99402a99-2dc8-4e79-85a4-7f41a4bdc078-bundle\") pod \"99402a99-2dc8-4e79-85a4-7f41a4bdc078\" (UID: \"99402a99-2dc8-4e79-85a4-7f41a4bdc078\") "
Jul 28 17:31:46.766337 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:46.766312    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf801-bde5-47aa-b8e6-45b31a925ad9-util\") pod \"5fdcf801-bde5-47aa-b8e6-45b31a925ad9\" (UID: \"5fdcf801-bde5-47aa-b8e6-45b31a925ad9\") "
Jul 28 17:31:46.766542 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:46.766358    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99402a99-2dc8-4e79-85a4-7f41a4bdc078-util\") pod \"99402a99-2dc8-4e79-85a4-7f41a4bdc078\" (UID: \"99402a99-2dc8-4e79-85a4-7f41a4bdc078\") "
Jul 28 17:31:46.766642 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:46.766626    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgfb4\" (UniqueName: \"kubernetes.io/projected/e227d129-ef64-4711-a61a-3d2c20d37d8c-kube-api-access-sgfb4\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:31:46.766701 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:46.766647    2543 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e227d129-ef64-4711-a61a-3d2c20d37d8c-util\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:31:46.766701 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:46.766661    2543 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e227d129-ef64-4711-a61a-3d2c20d37d8c-bundle\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:31:46.766849 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:46.766822    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99402a99-2dc8-4e79-85a4-7f41a4bdc078-bundle" (OuterVolumeSpecName: "bundle") pod "99402a99-2dc8-4e79-85a4-7f41a4bdc078" (UID: "99402a99-2dc8-4e79-85a4-7f41a4bdc078"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:31:46.767187 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:46.767162    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fdcf801-bde5-47aa-b8e6-45b31a925ad9-bundle" (OuterVolumeSpecName: "bundle") pod "5fdcf801-bde5-47aa-b8e6-45b31a925ad9" (UID: "5fdcf801-bde5-47aa-b8e6-45b31a925ad9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:31:46.768422 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:46.768403    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fdcf801-bde5-47aa-b8e6-45b31a925ad9-kube-api-access-tgdts" (OuterVolumeSpecName: "kube-api-access-tgdts") pod "5fdcf801-bde5-47aa-b8e6-45b31a925ad9" (UID: "5fdcf801-bde5-47aa-b8e6-45b31a925ad9"). InnerVolumeSpecName "kube-api-access-tgdts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:31:46.768482 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:46.768456    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99402a99-2dc8-4e79-85a4-7f41a4bdc078-kube-api-access-ksjwp" (OuterVolumeSpecName: "kube-api-access-ksjwp") pod "99402a99-2dc8-4e79-85a4-7f41a4bdc078" (UID: "99402a99-2dc8-4e79-85a4-7f41a4bdc078"). InnerVolumeSpecName "kube-api-access-ksjwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:31:46.775092 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:46.775066    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99402a99-2dc8-4e79-85a4-7f41a4bdc078-util" (OuterVolumeSpecName: "util") pod "99402a99-2dc8-4e79-85a4-7f41a4bdc078" (UID: "99402a99-2dc8-4e79-85a4-7f41a4bdc078"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:31:46.775628 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:46.775610    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fdcf801-bde5-47aa-b8e6-45b31a925ad9-util" (OuterVolumeSpecName: "util") pod "5fdcf801-bde5-47aa-b8e6-45b31a925ad9" (UID: "5fdcf801-bde5-47aa-b8e6-45b31a925ad9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:31:46.867272 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:46.867237    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksjwp\" (UniqueName: \"kubernetes.io/projected/99402a99-2dc8-4e79-85a4-7f41a4bdc078-kube-api-access-ksjwp\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:31:46.867272 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:46.867266    2543 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf801-bde5-47aa-b8e6-45b31a925ad9-bundle\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:31:46.867272 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:46.867277    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgdts\" (UniqueName: \"kubernetes.io/projected/5fdcf801-bde5-47aa-b8e6-45b31a925ad9-kube-api-access-tgdts\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:31:46.867481 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:46.867286    2543 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99402a99-2dc8-4e79-85a4-7f41a4bdc078-bundle\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:31:46.867481 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:46.867294    2543 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5fdcf801-bde5-47aa-b8e6-45b31a925ad9-util\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:31:46.867481 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:46.867303    2543 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99402a99-2dc8-4e79-85a4-7f41a4bdc078-util\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:31:47.435760 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:47.435686    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jzzvr"
Jul 28 17:31:47.435760 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:47.435701    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jzzvr" event={"ID":"e227d129-ef64-4711-a61a-3d2c20d37d8c","Type":"ContainerDied","Data":"aa2d5bb8e7c163776eaaa8d26e053f33dd6c4fa05d7a8cb65c3d5055c8d8bd87"}
Jul 28 17:31:47.435760 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:47.435750    2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa2d5bb8e7c163776eaaa8d26e053f33dd6c4fa05d7a8cb65c3d5055c8d8bd87"
Jul 28 17:31:47.437853 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:47.437832    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4l77x"
Jul 28 17:31:47.437978 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:47.437843    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c4l77x" event={"ID":"5fdcf801-bde5-47aa-b8e6-45b31a925ad9","Type":"ContainerDied","Data":"e7632629aeae1da4c50cf804f42330c920be77c74c10973be24a29423fbd85a5"}
Jul 28 17:31:47.437978 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:47.437898    2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7632629aeae1da4c50cf804f42330c920be77c74c10973be24a29423fbd85a5"
Jul 28 17:31:47.444070 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:47.443709    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qt7lt" event={"ID":"99402a99-2dc8-4e79-85a4-7f41a4bdc078","Type":"ContainerDied","Data":"cee2628b54b4831376d1d0629220053567ef90da4b5b195a6715eb5a150421fc"}
Jul 28 17:31:47.444070 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:47.443825    2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cee2628b54b4831376d1d0629220053567ef90da4b5b195a6715eb5a150421fc"
Jul 28 17:31:47.444070 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:47.443986    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0qt7lt"
Jul 28 17:31:47.581358 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:47.581335    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4rmrw"
Jul 28 17:31:47.674153 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:47.674109    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbbfh\" (UniqueName: \"kubernetes.io/projected/2e73880e-64eb-4e56-8642-d132ee1eaf96-kube-api-access-rbbfh\") pod \"2e73880e-64eb-4e56-8642-d132ee1eaf96\" (UID: \"2e73880e-64eb-4e56-8642-d132ee1eaf96\") "
Jul 28 17:31:47.674338 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:47.674192    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e73880e-64eb-4e56-8642-d132ee1eaf96-util\") pod \"2e73880e-64eb-4e56-8642-d132ee1eaf96\" (UID: \"2e73880e-64eb-4e56-8642-d132ee1eaf96\") "
Jul 28 17:31:47.674338 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:47.674223    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e73880e-64eb-4e56-8642-d132ee1eaf96-bundle\") pod \"2e73880e-64eb-4e56-8642-d132ee1eaf96\" (UID: \"2e73880e-64eb-4e56-8642-d132ee1eaf96\") "
Jul 28 17:31:47.674784 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:47.674758    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e73880e-64eb-4e56-8642-d132ee1eaf96-bundle" (OuterVolumeSpecName: "bundle") pod "2e73880e-64eb-4e56-8642-d132ee1eaf96" (UID: "2e73880e-64eb-4e56-8642-d132ee1eaf96"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:31:47.676254 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:47.676219    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e73880e-64eb-4e56-8642-d132ee1eaf96-kube-api-access-rbbfh" (OuterVolumeSpecName: "kube-api-access-rbbfh") pod "2e73880e-64eb-4e56-8642-d132ee1eaf96" (UID: "2e73880e-64eb-4e56-8642-d132ee1eaf96"). InnerVolumeSpecName "kube-api-access-rbbfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:31:47.680005 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:47.679971    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e73880e-64eb-4e56-8642-d132ee1eaf96-util" (OuterVolumeSpecName: "util") pod "2e73880e-64eb-4e56-8642-d132ee1eaf96" (UID: "2e73880e-64eb-4e56-8642-d132ee1eaf96"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:31:47.775540 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:47.775515    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbbfh\" (UniqueName: \"kubernetes.io/projected/2e73880e-64eb-4e56-8642-d132ee1eaf96-kube-api-access-rbbfh\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:31:47.775540 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:47.775539    2543 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e73880e-64eb-4e56-8642-d132ee1eaf96-util\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:31:47.775710 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:47.775549    2543 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e73880e-64eb-4e56-8642-d132ee1eaf96-bundle\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:31:48.450293 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:48.450260    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4rmrw"
Jul 28 17:31:48.450293 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:48.450281    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb4rmrw" event={"ID":"2e73880e-64eb-4e56-8642-d132ee1eaf96","Type":"ContainerDied","Data":"819aeae23c53e8c87be31b2a31601c5c3ad15efe857c8252c03f9e92df977a6c"}
Jul 28 17:31:48.450738 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:31:48.450336    2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="819aeae23c53e8c87be31b2a31601c5c3ad15efe857c8252c03f9e92df977a6c"
Jul 28 17:32:47.273707 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.273667    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-4bfr9"]
Jul 28 17:32:47.274343 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.274120    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99402a99-2dc8-4e79-85a4-7f41a4bdc078" containerName="extract"
Jul 28 17:32:47.274343 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.274156    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="99402a99-2dc8-4e79-85a4-7f41a4bdc078" containerName="extract"
Jul 28 17:32:47.274343 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.274172    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e227d129-ef64-4711-a61a-3d2c20d37d8c" containerName="extract"
Jul 28 17:32:47.274343 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.274180    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="e227d129-ef64-4711-a61a-3d2c20d37d8c" containerName="extract"
Jul 28 17:32:47.274343 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.274193    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e227d129-ef64-4711-a61a-3d2c20d37d8c" containerName="util"
Jul 28 17:32:47.274343 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.274201    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="e227d129-ef64-4711-a61a-3d2c20d37d8c" containerName="util"
Jul 28 17:32:47.274343 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.274210    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e227d129-ef64-4711-a61a-3d2c20d37d8c" containerName="pull"
Jul 28 17:32:47.274343 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.274218    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="e227d129-ef64-4711-a61a-3d2c20d37d8c" containerName="pull"
Jul 28 17:32:47.274343 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.274233    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99402a99-2dc8-4e79-85a4-7f41a4bdc078" containerName="util"
Jul 28 17:32:47.274343 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.274242    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="99402a99-2dc8-4e79-85a4-7f41a4bdc078" containerName="util"
Jul 28 17:32:47.274343 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.274252    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e73880e-64eb-4e56-8642-d132ee1eaf96" containerName="extract"
Jul 28 17:32:47.274343 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.274261    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e73880e-64eb-4e56-8642-d132ee1eaf96" containerName="extract"
Jul 28 17:32:47.274343 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.274275    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99402a99-2dc8-4e79-85a4-7f41a4bdc078" containerName="pull"
Jul 28 17:32:47.274343 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.274284    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="99402a99-2dc8-4e79-85a4-7f41a4bdc078" containerName="pull"
Jul 28 17:32:47.274343 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.274295    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fdcf801-bde5-47aa-b8e6-45b31a925ad9" containerName="extract"
Jul 28 17:32:47.274343 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.274303    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fdcf801-bde5-47aa-b8e6-45b31a925ad9" containerName="extract"
Jul 28 17:32:47.274343 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.274316    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e73880e-64eb-4e56-8642-d132ee1eaf96" containerName="pull"
Jul 28 17:32:47.274343 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.274324    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e73880e-64eb-4e56-8642-d132ee1eaf96" containerName="pull"
Jul 28 17:32:47.274343 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.274334    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e73880e-64eb-4e56-8642-d132ee1eaf96" containerName="util"
Jul 28 17:32:47.274343 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.274341    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e73880e-64eb-4e56-8642-d132ee1eaf96" containerName="util"
Jul 28 17:32:47.274343 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.274352    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fdcf801-bde5-47aa-b8e6-45b31a925ad9" containerName="util"
Jul 28 17:32:47.275350 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.274360    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fdcf801-bde5-47aa-b8e6-45b31a925ad9" containerName="util"
Jul 28 17:32:47.275350 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.274369    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fdcf801-bde5-47aa-b8e6-45b31a925ad9" containerName="pull"
Jul 28 17:32:47.275350 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.274376    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fdcf801-bde5-47aa-b8e6-45b31a925ad9" containerName="pull"
Jul 28 17:32:47.275350 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.274464    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="99402a99-2dc8-4e79-85a4-7f41a4bdc078" containerName="extract"
Jul 28 17:32:47.275350 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.274477    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="e227d129-ef64-4711-a61a-3d2c20d37d8c" containerName="extract"
Jul 28 17:32:47.275350 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.274488    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fdcf801-bde5-47aa-b8e6-45b31a925ad9" containerName="extract"
Jul 28 17:32:47.275350 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.274500    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e73880e-64eb-4e56-8642-d132ee1eaf96" containerName="extract"
Jul 28 17:32:47.277613 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.277592    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-4bfr9"
Jul 28 17:32:47.280174 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.280157    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 17:32:47.280694 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.280677    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 17:32:47.280805 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.280680    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6z7f5\""
Jul 28 17:32:47.287637 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.287615    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-4bfr9"]
Jul 28 17:32:47.291020 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.290999    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5g78\" (UniqueName: \"kubernetes.io/projected/cdbd462b-b499-4de5-a213-6a52cf5cb3d4-kube-api-access-t5g78\") pod \"authorino-b54df94b4-4bfr9\" (UID: \"cdbd462b-b499-4de5-a213-6a52cf5cb3d4\") " pod="kuadrant-system/authorino-b54df94b4-4bfr9"
Jul 28 17:32:47.392447 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.392421    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5g78\" (UniqueName: \"kubernetes.io/projected/cdbd462b-b499-4de5-a213-6a52cf5cb3d4-kube-api-access-t5g78\") pod \"authorino-b54df94b4-4bfr9\" (UID: \"cdbd462b-b499-4de5-a213-6a52cf5cb3d4\") " pod="kuadrant-system/authorino-b54df94b4-4bfr9"
Jul 28 17:32:47.404676 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.404650    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5g78\" (UniqueName: \"kubernetes.io/projected/cdbd462b-b499-4de5-a213-6a52cf5cb3d4-kube-api-access-t5g78\") pod \"authorino-b54df94b4-4bfr9\" (UID: \"cdbd462b-b499-4de5-a213-6a52cf5cb3d4\") " pod="kuadrant-system/authorino-b54df94b4-4bfr9"
Jul 28 17:32:47.428289 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.428243    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-4tlpg"]
Jul 28 17:32:47.431591 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.431574    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-4tlpg"
Jul 28 17:32:47.437709 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.437688    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-4tlpg"]
Jul 28 17:32:47.492833 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.492800    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n65mv\" (UniqueName: \"kubernetes.io/projected/97f2ada9-15b6-46ea-95e4-1d8a7394235f-kube-api-access-n65mv\") pod \"authorino-c548b5d7d-4tlpg\" (UID: \"97f2ada9-15b6-46ea-95e4-1d8a7394235f\") " pod="kuadrant-system/authorino-c548b5d7d-4tlpg"
Jul 28 17:32:47.587837 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.587760    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-4bfr9"
Jul 28 17:32:47.593977 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.593956    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n65mv\" (UniqueName: \"kubernetes.io/projected/97f2ada9-15b6-46ea-95e4-1d8a7394235f-kube-api-access-n65mv\") pod \"authorino-c548b5d7d-4tlpg\" (UID: \"97f2ada9-15b6-46ea-95e4-1d8a7394235f\") " pod="kuadrant-system/authorino-c548b5d7d-4tlpg"
Jul 28 17:32:47.609282 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.609258    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n65mv\" (UniqueName: \"kubernetes.io/projected/97f2ada9-15b6-46ea-95e4-1d8a7394235f-kube-api-access-n65mv\") pod \"authorino-c548b5d7d-4tlpg\" (UID: \"97f2ada9-15b6-46ea-95e4-1d8a7394235f\") " pod="kuadrant-system/authorino-c548b5d7d-4tlpg"
Jul 28 17:32:47.715651 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.715536    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-4bfr9"]
Jul 28 17:32:47.718289 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:32:47.718263    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdbd462b_b499_4de5_a213_6a52cf5cb3d4.slice/crio-745d87d3fde63d1f3f37fdd16c85bf40d60345ceb3be40c3df5be9e41891b22f WatchSource:0}: Error finding container 745d87d3fde63d1f3f37fdd16c85bf40d60345ceb3be40c3df5be9e41891b22f: Status 404 returned error can't find the container with id 745d87d3fde63d1f3f37fdd16c85bf40d60345ceb3be40c3df5be9e41891b22f
Jul 28 17:32:47.736923 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.736898    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-4bfr9" event={"ID":"cdbd462b-b499-4de5-a213-6a52cf5cb3d4","Type":"ContainerStarted","Data":"745d87d3fde63d1f3f37fdd16c85bf40d60345ceb3be40c3df5be9e41891b22f"}
Jul 28 17:32:47.742054 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.742038    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-4tlpg"
Jul 28 17:32:47.870843 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:47.870817    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-4tlpg"]
Jul 28 17:32:47.872789 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:32:47.872766    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97f2ada9_15b6_46ea_95e4_1d8a7394235f.slice/crio-6ee8133ca87823074f2a697719f6dac74f8634463ae41adb23c9367f7c70a808 WatchSource:0}: Error finding container 6ee8133ca87823074f2a697719f6dac74f8634463ae41adb23c9367f7c70a808: Status 404 returned error can't find the container with id 6ee8133ca87823074f2a697719f6dac74f8634463ae41adb23c9367f7c70a808
Jul 28 17:32:48.745377 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:48.745311    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-4tlpg" event={"ID":"97f2ada9-15b6-46ea-95e4-1d8a7394235f","Type":"ContainerStarted","Data":"6ee8133ca87823074f2a697719f6dac74f8634463ae41adb23c9367f7c70a808"}
Jul 28 17:32:50.756727 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:50.756690    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-4bfr9" event={"ID":"cdbd462b-b499-4de5-a213-6a52cf5cb3d4","Type":"ContainerStarted","Data":"2542e76b703fd73343294cc3ae1b365c45962296f2d6694a18b6899e2503931f"}
Jul 28 17:32:50.758302 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:50.758273    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-4tlpg" event={"ID":"97f2ada9-15b6-46ea-95e4-1d8a7394235f","Type":"ContainerStarted","Data":"59f31a65bc8e8b3fdfb01b1ab0cf670cfde363dff9963e8d71574bdb07b9142d"}
Jul 28 17:32:50.773172 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:50.773109    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-4bfr9" podStartSLOduration=1.24797417 podStartE2EDuration="3.773094846s" totalImagesPullingTime="2.525120676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:32:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:32:50.771932514 +0000 UTC m=+574.178206254" watchObservedRunningTime="2026-07-28 17:32:50.773094846 +0000 UTC m=+574.179368588"
Jul 28 17:32:50.786795 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:50.786744    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-4tlpg" podStartSLOduration=1.415649734 podStartE2EDuration="3.786726181s" totalImagesPullingTime="2.371076447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:32:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:32:50.786182521 +0000 UTC m=+574.192456263" watchObservedRunningTime="2026-07-28 17:32:50.786726181 +0000 UTC m=+574.192999928"
Jul 28 17:32:50.811627 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:50.811598    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-4bfr9"]
Jul 28 17:32:52.768247 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:52.768210    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-4bfr9" podUID="cdbd462b-b499-4de5-a213-6a52cf5cb3d4" containerName="authorino" containerID="cri-o://2542e76b703fd73343294cc3ae1b365c45962296f2d6694a18b6899e2503931f" gracePeriod=30
Jul 28 17:32:53.006716 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:53.006691    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-4bfr9"
Jul 28 17:32:53.031149 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:53.031071    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5g78\" (UniqueName: \"kubernetes.io/projected/cdbd462b-b499-4de5-a213-6a52cf5cb3d4-kube-api-access-t5g78\") pod \"cdbd462b-b499-4de5-a213-6a52cf5cb3d4\" (UID: \"cdbd462b-b499-4de5-a213-6a52cf5cb3d4\") "
Jul 28 17:32:53.033067 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:53.033042    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdbd462b-b499-4de5-a213-6a52cf5cb3d4-kube-api-access-t5g78" (OuterVolumeSpecName: "kube-api-access-t5g78") pod "cdbd462b-b499-4de5-a213-6a52cf5cb3d4" (UID: "cdbd462b-b499-4de5-a213-6a52cf5cb3d4"). InnerVolumeSpecName "kube-api-access-t5g78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:32:53.132371 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:53.132330    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5g78\" (UniqueName: \"kubernetes.io/projected/cdbd462b-b499-4de5-a213-6a52cf5cb3d4-kube-api-access-t5g78\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:32:53.773766 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:53.773734    2543 generic.go:356] "Generic (PLEG): container finished" podID="cdbd462b-b499-4de5-a213-6a52cf5cb3d4" containerID="2542e76b703fd73343294cc3ae1b365c45962296f2d6694a18b6899e2503931f" exitCode=0
Jul 28 17:32:53.774193 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:53.773778    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-4bfr9"
Jul 28 17:32:53.774193 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:53.773821    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-4bfr9" event={"ID":"cdbd462b-b499-4de5-a213-6a52cf5cb3d4","Type":"ContainerDied","Data":"2542e76b703fd73343294cc3ae1b365c45962296f2d6694a18b6899e2503931f"}
Jul 28 17:32:53.774193 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:53.773863    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-4bfr9" event={"ID":"cdbd462b-b499-4de5-a213-6a52cf5cb3d4","Type":"ContainerDied","Data":"745d87d3fde63d1f3f37fdd16c85bf40d60345ceb3be40c3df5be9e41891b22f"}
Jul 28 17:32:53.774193 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:53.773884    2543 scope.go:117] "RemoveContainer" containerID="2542e76b703fd73343294cc3ae1b365c45962296f2d6694a18b6899e2503931f"
Jul 28 17:32:53.782406 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:53.782382    2543 scope.go:117] "RemoveContainer" containerID="2542e76b703fd73343294cc3ae1b365c45962296f2d6694a18b6899e2503931f"
Jul 28 17:32:53.782658 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:32:53.782627    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2542e76b703fd73343294cc3ae1b365c45962296f2d6694a18b6899e2503931f\": container with ID starting with 2542e76b703fd73343294cc3ae1b365c45962296f2d6694a18b6899e2503931f not found: ID does not exist" containerID="2542e76b703fd73343294cc3ae1b365c45962296f2d6694a18b6899e2503931f"
Jul 28 17:32:53.782735 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:53.782669    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2542e76b703fd73343294cc3ae1b365c45962296f2d6694a18b6899e2503931f"} err="failed to get container status \"2542e76b703fd73343294cc3ae1b365c45962296f2d6694a18b6899e2503931f\": rpc error: code = NotFound desc = could not find container \"2542e76b703fd73343294cc3ae1b365c45962296f2d6694a18b6899e2503931f\": container with ID starting with 2542e76b703fd73343294cc3ae1b365c45962296f2d6694a18b6899e2503931f not found: ID does not exist"
Jul 28 17:32:53.792071 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:53.792049    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-4bfr9"]
Jul 28 17:32:53.796404 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:53.796384    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-4bfr9"]
Jul 28 17:32:55.157860 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:32:55.157818    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdbd462b-b499-4de5-a213-6a52cf5cb3d4" path="/var/lib/kubelet/pods/cdbd462b-b499-4de5-a213-6a52cf5cb3d4/volumes"
Jul 28 17:33:10.781066 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:10.781023    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cf4765b85-5shw9"]
Jul 28 17:33:10.781558 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:10.781523    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdbd462b-b499-4de5-a213-6a52cf5cb3d4" containerName="authorino"
Jul 28 17:33:10.781558 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:10.781540    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdbd462b-b499-4de5-a213-6a52cf5cb3d4" containerName="authorino"
Jul 28 17:33:10.781667 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:10.781621    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdbd462b-b499-4de5-a213-6a52cf5cb3d4" containerName="authorino"
Jul 28 17:33:10.784522 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:10.784501    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf4765b85-5shw9"
Jul 28 17:33:10.787055 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:10.787033    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 28 17:33:10.794083 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:10.794051    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cf4765b85-5shw9"]
Jul 28 17:33:10.882202 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:10.882173    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npt82\" (UniqueName: \"kubernetes.io/projected/ae97ec5d-a62d-4674-bf00-578d8afe1cad-kube-api-access-npt82\") pod \"authorino-7cf4765b85-5shw9\" (UID: \"ae97ec5d-a62d-4674-bf00-578d8afe1cad\") " pod="kuadrant-system/authorino-7cf4765b85-5shw9"
Jul 28 17:33:10.882387 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:10.882227    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ae97ec5d-a62d-4674-bf00-578d8afe1cad-tls-cert\") pod \"authorino-7cf4765b85-5shw9\" (UID: \"ae97ec5d-a62d-4674-bf00-578d8afe1cad\") " pod="kuadrant-system/authorino-7cf4765b85-5shw9"
Jul 28 17:33:10.983070 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:10.983039    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ae97ec5d-a62d-4674-bf00-578d8afe1cad-tls-cert\") pod \"authorino-7cf4765b85-5shw9\" (UID: \"ae97ec5d-a62d-4674-bf00-578d8afe1cad\") " pod="kuadrant-system/authorino-7cf4765b85-5shw9"
Jul 28 17:33:10.983255 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:10.983220    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npt82\" (UniqueName: \"kubernetes.io/projected/ae97ec5d-a62d-4674-bf00-578d8afe1cad-kube-api-access-npt82\") pod \"authorino-7cf4765b85-5shw9\" (UID: \"ae97ec5d-a62d-4674-bf00-578d8afe1cad\") " pod="kuadrant-system/authorino-7cf4765b85-5shw9"
Jul 28 17:33:10.985463 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:10.985434    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ae97ec5d-a62d-4674-bf00-578d8afe1cad-tls-cert\") pod \"authorino-7cf4765b85-5shw9\" (UID: \"ae97ec5d-a62d-4674-bf00-578d8afe1cad\") " pod="kuadrant-system/authorino-7cf4765b85-5shw9"
Jul 28 17:33:10.990964 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:10.990946    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npt82\" (UniqueName: \"kubernetes.io/projected/ae97ec5d-a62d-4674-bf00-578d8afe1cad-kube-api-access-npt82\") pod \"authorino-7cf4765b85-5shw9\" (UID: \"ae97ec5d-a62d-4674-bf00-578d8afe1cad\") " pod="kuadrant-system/authorino-7cf4765b85-5shw9"
Jul 28 17:33:11.093766 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:11.093682    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf4765b85-5shw9"
Jul 28 17:33:11.228086 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:11.228057    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cf4765b85-5shw9"]
Jul 28 17:33:11.230846 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:33:11.230816    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae97ec5d_a62d_4674_bf00_578d8afe1cad.slice/crio-a1d4d5daf877497fbafb351a7be4c76ff833268256a0020539e95c80c30b5c43 WatchSource:0}: Error finding container a1d4d5daf877497fbafb351a7be4c76ff833268256a0020539e95c80c30b5c43: Status 404 returned error can't find the container with id a1d4d5daf877497fbafb351a7be4c76ff833268256a0020539e95c80c30b5c43
Jul 28 17:33:11.861032 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:11.861001    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cf4765b85-5shw9" event={"ID":"ae97ec5d-a62d-4674-bf00-578d8afe1cad","Type":"ContainerStarted","Data":"a1d4d5daf877497fbafb351a7be4c76ff833268256a0020539e95c80c30b5c43"}
Jul 28 17:33:12.866977 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:12.866944    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cf4765b85-5shw9" event={"ID":"ae97ec5d-a62d-4674-bf00-578d8afe1cad","Type":"ContainerStarted","Data":"fb22fa5b2dd0c0817d68019188226f853335fb5e510831a27327eaa3e501755a"}
Jul 28 17:33:12.884043 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:12.883993    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7cf4765b85-5shw9" podStartSLOduration=1.850314802 podStartE2EDuration="2.883976183s" totalImagesPullingTime="1.033661381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:33:12.882015633 +0000 UTC m=+596.288289375" watchObservedRunningTime="2026-07-28 17:33:12.883976183 +0000 UTC m=+596.290249971"
Jul 28 17:33:12.910567 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:12.910531    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-4tlpg"]
Jul 28 17:33:12.910803 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:12.910764    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-4tlpg" podUID="97f2ada9-15b6-46ea-95e4-1d8a7394235f" containerName="authorino" containerID="cri-o://59f31a65bc8e8b3fdfb01b1ab0cf670cfde363dff9963e8d71574bdb07b9142d" gracePeriod=30
Jul 28 17:33:13.156524 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:13.156501    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-4tlpg"
Jul 28 17:33:13.204153 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:13.201968    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n65mv\" (UniqueName: \"kubernetes.io/projected/97f2ada9-15b6-46ea-95e4-1d8a7394235f-kube-api-access-n65mv\") pod \"97f2ada9-15b6-46ea-95e4-1d8a7394235f\" (UID: \"97f2ada9-15b6-46ea-95e4-1d8a7394235f\") "
Jul 28 17:33:13.207977 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:13.207948    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f2ada9-15b6-46ea-95e4-1d8a7394235f-kube-api-access-n65mv" (OuterVolumeSpecName: "kube-api-access-n65mv") pod "97f2ada9-15b6-46ea-95e4-1d8a7394235f" (UID: "97f2ada9-15b6-46ea-95e4-1d8a7394235f"). InnerVolumeSpecName "kube-api-access-n65mv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:33:13.303492 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:13.303464    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n65mv\" (UniqueName: \"kubernetes.io/projected/97f2ada9-15b6-46ea-95e4-1d8a7394235f-kube-api-access-n65mv\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:33:13.872827 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:13.872790    2543 generic.go:356] "Generic (PLEG): container finished" podID="97f2ada9-15b6-46ea-95e4-1d8a7394235f" containerID="59f31a65bc8e8b3fdfb01b1ab0cf670cfde363dff9963e8d71574bdb07b9142d" exitCode=0
Jul 28 17:33:13.873246 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:13.872867    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-4tlpg"
Jul 28 17:33:13.873246 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:13.872876    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-4tlpg" event={"ID":"97f2ada9-15b6-46ea-95e4-1d8a7394235f","Type":"ContainerDied","Data":"59f31a65bc8e8b3fdfb01b1ab0cf670cfde363dff9963e8d71574bdb07b9142d"}
Jul 28 17:33:13.873246 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:13.872918    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-4tlpg" event={"ID":"97f2ada9-15b6-46ea-95e4-1d8a7394235f","Type":"ContainerDied","Data":"6ee8133ca87823074f2a697719f6dac74f8634463ae41adb23c9367f7c70a808"}
Jul 28 17:33:13.873246 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:13.872936    2543 scope.go:117] "RemoveContainer" containerID="59f31a65bc8e8b3fdfb01b1ab0cf670cfde363dff9963e8d71574bdb07b9142d"
Jul 28 17:33:13.882548 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:13.882530    2543 scope.go:117] "RemoveContainer" containerID="59f31a65bc8e8b3fdfb01b1ab0cf670cfde363dff9963e8d71574bdb07b9142d"
Jul 28 17:33:13.882864 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:33:13.882840    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59f31a65bc8e8b3fdfb01b1ab0cf670cfde363dff9963e8d71574bdb07b9142d\": container with ID starting with 59f31a65bc8e8b3fdfb01b1ab0cf670cfde363dff9963e8d71574bdb07b9142d not found: ID does not exist" containerID="59f31a65bc8e8b3fdfb01b1ab0cf670cfde363dff9963e8d71574bdb07b9142d"
Jul 28 17:33:13.882947 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:13.882868    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59f31a65bc8e8b3fdfb01b1ab0cf670cfde363dff9963e8d71574bdb07b9142d"} err="failed to get container status \"59f31a65bc8e8b3fdfb01b1ab0cf670cfde363dff9963e8d71574bdb07b9142d\": rpc error: code = NotFound desc = could not find container \"59f31a65bc8e8b3fdfb01b1ab0cf670cfde363dff9963e8d71574bdb07b9142d\": container with ID starting with 59f31a65bc8e8b3fdfb01b1ab0cf670cfde363dff9963e8d71574bdb07b9142d not found: ID does not exist"
Jul 28 17:33:13.894703 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:13.894682    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-4tlpg"]
Jul 28 17:33:13.900092 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:13.900064    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-4tlpg"]
Jul 28 17:33:15.152548 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:15.152521    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97f2ada9-15b6-46ea-95e4-1d8a7394235f" path="/var/lib/kubelet/pods/97f2ada9-15b6-46ea-95e4-1d8a7394235f/volumes"
Jul 28 17:33:17.627401 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:17.627368    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["observability/jaeger-6bbd7d4c9d-z8vbf"]
Jul 28 17:33:17.627781 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:17.627714    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97f2ada9-15b6-46ea-95e4-1d8a7394235f" containerName="authorino"
Jul 28 17:33:17.627781 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:17.627724    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="97f2ada9-15b6-46ea-95e4-1d8a7394235f" containerName="authorino"
Jul 28 17:33:17.627851 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:17.627793    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="97f2ada9-15b6-46ea-95e4-1d8a7394235f" containerName="authorino"
Jul 28 17:33:17.632106 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:17.632089    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="observability/jaeger-6bbd7d4c9d-z8vbf"
Jul 28 17:33:17.635221 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:17.635195    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"observability\"/\"kube-root-ca.crt\""
Jul 28 17:33:17.635221 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:17.635212    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"observability\"/\"jaeger-dockercfg-cjrcn\""
Jul 28 17:33:17.635934 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:17.635921    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"observability\"/\"openshift-service-ca.crt\""
Jul 28 17:33:17.639618 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:17.639602    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["observability/jaeger-6bbd7d4c9d-z8vbf"]
Jul 28 17:33:17.742701 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:17.742668    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqmr6\" (UniqueName: \"kubernetes.io/projected/7eeffbf4-719d-48c2-ae29-7c0d6853d8ea-kube-api-access-kqmr6\") pod \"jaeger-6bbd7d4c9d-z8vbf\" (UID: \"7eeffbf4-719d-48c2-ae29-7c0d6853d8ea\") " pod="observability/jaeger-6bbd7d4c9d-z8vbf"
Jul 28 17:33:17.843362 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:17.843332    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqmr6\" (UniqueName: \"kubernetes.io/projected/7eeffbf4-719d-48c2-ae29-7c0d6853d8ea-kube-api-access-kqmr6\") pod \"jaeger-6bbd7d4c9d-z8vbf\" (UID: \"7eeffbf4-719d-48c2-ae29-7c0d6853d8ea\") " pod="observability/jaeger-6bbd7d4c9d-z8vbf"
Jul 28 17:33:17.854627 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:17.854604    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqmr6\" (UniqueName: \"kubernetes.io/projected/7eeffbf4-719d-48c2-ae29-7c0d6853d8ea-kube-api-access-kqmr6\") pod \"jaeger-6bbd7d4c9d-z8vbf\" (UID: \"7eeffbf4-719d-48c2-ae29-7c0d6853d8ea\") " pod="observability/jaeger-6bbd7d4c9d-z8vbf"
Jul 28 17:33:17.941183 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:17.941094    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="observability/jaeger-6bbd7d4c9d-z8vbf"
Jul 28 17:33:18.066796 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:18.066771    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["observability/jaeger-6bbd7d4c9d-z8vbf"]
Jul 28 17:33:18.069209 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:33:18.069186    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7eeffbf4_719d_48c2_ae29_7c0d6853d8ea.slice/crio-44358a25141b79a78598ebef2d0d316ddc1ada536215ca03cd065cc7cce4d187 WatchSource:0}: Error finding container 44358a25141b79a78598ebef2d0d316ddc1ada536215ca03cd065cc7cce4d187: Status 404 returned error can't find the container with id 44358a25141b79a78598ebef2d0d316ddc1ada536215ca03cd065cc7cce4d187
Jul 28 17:33:18.899945 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:18.899906    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="observability/jaeger-6bbd7d4c9d-z8vbf" event={"ID":"7eeffbf4-719d-48c2-ae29-7c0d6853d8ea","Type":"ContainerStarted","Data":"44358a25141b79a78598ebef2d0d316ddc1ada536215ca03cd065cc7cce4d187"}
Jul 28 17:33:19.906612 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:19.906577    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="observability/jaeger-6bbd7d4c9d-z8vbf" event={"ID":"7eeffbf4-719d-48c2-ae29-7c0d6853d8ea","Type":"ContainerStarted","Data":"6ad64ff1cd77f01752f95730a12b7bcca9ea73df6fd5b97b13cfabb73a4273aa"}
Jul 28 17:33:19.907083 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:19.906697    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="observability/jaeger-6bbd7d4c9d-z8vbf"
Jul 28 17:33:19.924333 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:19.924262    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="observability/jaeger-6bbd7d4c9d-z8vbf" podStartSLOduration=1.2003347500000001 podStartE2EDuration="2.924248959s" totalImagesPullingTime="1.723914209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:33:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:33:19.923174222 +0000 UTC m=+603.329447964" watchObservedRunningTime="2026-07-28 17:33:19.924248959 +0000 UTC m=+603.330522699"
Jul 28 17:33:20.914266 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:20.914232    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="observability/jaeger-6bbd7d4c9d-z8vbf"
Jul 28 17:33:22.740749 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:22.740719    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-4n9rn"]
Jul 28 17:33:22.741214 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:22.740911    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-cd4d44c67-4n9rn" podUID="720b74c0-3bc1-42a8-9ba7-63d612b5044d" containerName="manager" containerID="cri-o://11240730b56936a23668f2939af4d338ad027dfe9fe5d0441959d578c059f601" gracePeriod=10
Jul 28 17:33:22.766633 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:22.766606    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-tftvf"]
Jul 28 17:33:22.769892 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:22.769876    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cd4d44c67-tftvf"
Jul 28 17:33:22.784357 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:22.784330    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-tftvf"]
Jul 28 17:33:22.887775 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:22.887742    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2eeab2fe-1bad-4c77-a992-3d4db33d5387-cert\") pod \"kserve-controller-manager-cd4d44c67-tftvf\" (UID: \"2eeab2fe-1bad-4c77-a992-3d4db33d5387\") " pod="kserve/kserve-controller-manager-cd4d44c67-tftvf"
Jul 28 17:33:22.887923 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:22.887800    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5bd6\" (UniqueName: \"kubernetes.io/projected/2eeab2fe-1bad-4c77-a992-3d4db33d5387-kube-api-access-g5bd6\") pod \"kserve-controller-manager-cd4d44c67-tftvf\" (UID: \"2eeab2fe-1bad-4c77-a992-3d4db33d5387\") " pod="kserve/kserve-controller-manager-cd4d44c67-tftvf"
Jul 28 17:33:22.923847 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:22.923820    2543 generic.go:356] "Generic (PLEG): container finished" podID="720b74c0-3bc1-42a8-9ba7-63d612b5044d" containerID="11240730b56936a23668f2939af4d338ad027dfe9fe5d0441959d578c059f601" exitCode=0
Jul 28 17:33:22.923972 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:22.923888    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cd4d44c67-4n9rn" event={"ID":"720b74c0-3bc1-42a8-9ba7-63d612b5044d","Type":"ContainerDied","Data":"11240730b56936a23668f2939af4d338ad027dfe9fe5d0441959d578c059f601"}
Jul 28 17:33:22.988987 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:22.988963    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2eeab2fe-1bad-4c77-a992-3d4db33d5387-cert\") pod \"kserve-controller-manager-cd4d44c67-tftvf\" (UID: \"2eeab2fe-1bad-4c77-a992-3d4db33d5387\") " pod="kserve/kserve-controller-manager-cd4d44c67-tftvf"
Jul 28 17:33:22.989095 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:22.989002    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5bd6\" (UniqueName: \"kubernetes.io/projected/2eeab2fe-1bad-4c77-a992-3d4db33d5387-kube-api-access-g5bd6\") pod \"kserve-controller-manager-cd4d44c67-tftvf\" (UID: \"2eeab2fe-1bad-4c77-a992-3d4db33d5387\") " pod="kserve/kserve-controller-manager-cd4d44c67-tftvf"
Jul 28 17:33:22.991482 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:22.991423    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2eeab2fe-1bad-4c77-a992-3d4db33d5387-cert\") pod \"kserve-controller-manager-cd4d44c67-tftvf\" (UID: \"2eeab2fe-1bad-4c77-a992-3d4db33d5387\") " pod="kserve/kserve-controller-manager-cd4d44c67-tftvf"
Jul 28 17:33:22.997390 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:22.997367    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5bd6\" (UniqueName: \"kubernetes.io/projected/2eeab2fe-1bad-4c77-a992-3d4db33d5387-kube-api-access-g5bd6\") pod \"kserve-controller-manager-cd4d44c67-tftvf\" (UID: \"2eeab2fe-1bad-4c77-a992-3d4db33d5387\") " pod="kserve/kserve-controller-manager-cd4d44c67-tftvf"
Jul 28 17:33:23.003683 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:23.003662    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cd4d44c67-4n9rn"
Jul 28 17:33:23.090384 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:23.090352    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/720b74c0-3bc1-42a8-9ba7-63d612b5044d-cert\") pod \"720b74c0-3bc1-42a8-9ba7-63d612b5044d\" (UID: \"720b74c0-3bc1-42a8-9ba7-63d612b5044d\") "
Jul 28 17:33:23.090543 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:23.090408    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwj94\" (UniqueName: \"kubernetes.io/projected/720b74c0-3bc1-42a8-9ba7-63d612b5044d-kube-api-access-gwj94\") pod \"720b74c0-3bc1-42a8-9ba7-63d612b5044d\" (UID: \"720b74c0-3bc1-42a8-9ba7-63d612b5044d\") "
Jul 28 17:33:23.092430 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:23.092398    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/720b74c0-3bc1-42a8-9ba7-63d612b5044d-cert" (OuterVolumeSpecName: "cert") pod "720b74c0-3bc1-42a8-9ba7-63d612b5044d" (UID: "720b74c0-3bc1-42a8-9ba7-63d612b5044d"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:33:23.092535 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:23.092483    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/720b74c0-3bc1-42a8-9ba7-63d612b5044d-kube-api-access-gwj94" (OuterVolumeSpecName: "kube-api-access-gwj94") pod "720b74c0-3bc1-42a8-9ba7-63d612b5044d" (UID: "720b74c0-3bc1-42a8-9ba7-63d612b5044d"). InnerVolumeSpecName "kube-api-access-gwj94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:33:23.125010 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:23.124989    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cd4d44c67-tftvf"
Jul 28 17:33:23.191368 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:23.191334    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwj94\" (UniqueName: \"kubernetes.io/projected/720b74c0-3bc1-42a8-9ba7-63d612b5044d-kube-api-access-gwj94\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:33:23.191368 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:23.191368    2543 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/720b74c0-3bc1-42a8-9ba7-63d612b5044d-cert\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:33:23.260307 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:23.260282    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-tftvf"]
Jul 28 17:33:23.262244 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:33:23.262217    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eeab2fe_1bad_4c77_a992_3d4db33d5387.slice/crio-16f1c543d2a779f42e3d878186cf784f9eb9c85fb3d03a4dfd69d243ea3dee03 WatchSource:0}: Error finding container 16f1c543d2a779f42e3d878186cf784f9eb9c85fb3d03a4dfd69d243ea3dee03: Status 404 returned error can't find the container with id 16f1c543d2a779f42e3d878186cf784f9eb9c85fb3d03a4dfd69d243ea3dee03
Jul 28 17:33:23.930369 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:23.930294    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-cd4d44c67-4n9rn"
Jul 28 17:33:23.930369 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:23.930318    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cd4d44c67-4n9rn" event={"ID":"720b74c0-3bc1-42a8-9ba7-63d612b5044d","Type":"ContainerDied","Data":"5d3efdab74e83a03f8578ff919aca392080a82b8f132e61607670a88f3f75b53"}
Jul 28 17:33:23.930867 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:23.930411    2543 scope.go:117] "RemoveContainer" containerID="11240730b56936a23668f2939af4d338ad027dfe9fe5d0441959d578c059f601"
Jul 28 17:33:23.932061 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:23.932038    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cd4d44c67-tftvf" event={"ID":"2eeab2fe-1bad-4c77-a992-3d4db33d5387","Type":"ContainerStarted","Data":"760d3faff9b81c938c365b7715817c5f679b7d7a225875a5d6278661a8482981"}
Jul 28 17:33:23.932212 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:23.932075    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-cd4d44c67-tftvf" event={"ID":"2eeab2fe-1bad-4c77-a992-3d4db33d5387","Type":"ContainerStarted","Data":"16f1c543d2a779f42e3d878186cf784f9eb9c85fb3d03a4dfd69d243ea3dee03"}
Jul 28 17:33:23.932212 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:23.932172    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-cd4d44c67-tftvf"
Jul 28 17:33:23.947675 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:23.947654    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-4n9rn"]
Jul 28 17:33:23.951962 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:23.951941    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-cd4d44c67-4n9rn"]
Jul 28 17:33:23.973925 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:23.973875    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-cd4d44c67-tftvf" podStartSLOduration=1.6440377659999998 podStartE2EDuration="1.973862824s" totalImagesPullingTime="329.825058ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:33:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:33:23.97170028 +0000 UTC m=+607.377974044" watchObservedRunningTime="2026-07-28 17:33:23.973862824 +0000 UTC m=+607.380136593"
Jul 28 17:33:25.152620 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:25.152586    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="720b74c0-3bc1-42a8-9ba7-63d612b5044d" path="/var/lib/kubelet/pods/720b74c0-3bc1-42a8-9ba7-63d612b5044d/volumes"
Jul 28 17:33:54.941483 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:33:54.941407    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-cd4d44c67-tftvf"
Jul 28 17:34:13.307172 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:34:13.307129    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-jmftq"]
Jul 28 17:34:13.307553 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:34:13.307471    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="720b74c0-3bc1-42a8-9ba7-63d612b5044d" containerName="manager"
Jul 28 17:34:13.307553 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:34:13.307481    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="720b74c0-3bc1-42a8-9ba7-63d612b5044d" containerName="manager"
Jul 28 17:34:13.307553 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:34:13.307541    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="720b74c0-3bc1-42a8-9ba7-63d612b5044d" containerName="manager"
Jul 28 17:34:13.310575 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:34:13.310560    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-jmftq"
Jul 28 17:34:13.313344 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:34:13.313320    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-b84k6\""
Jul 28 17:34:13.313460 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:34:13.313368    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 17:34:13.320667 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:34:13.320645    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-jmftq"]
Jul 28 17:34:13.418450 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:34:13.418421    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5258d5f9-6236-4bd2-80d8-8397d993c189-model-cache\") pod \"s3-init-jmftq\" (UID: \"5258d5f9-6236-4bd2-80d8-8397d993c189\") " pod="kserve/s3-init-jmftq"
Jul 28 17:34:13.418609 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:34:13.418463    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r5dl\" (UniqueName: \"kubernetes.io/projected/5258d5f9-6236-4bd2-80d8-8397d993c189-kube-api-access-8r5dl\") pod \"s3-init-jmftq\" (UID: \"5258d5f9-6236-4bd2-80d8-8397d993c189\") " pod="kserve/s3-init-jmftq"
Jul 28 17:34:13.519702 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:34:13.519671    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5258d5f9-6236-4bd2-80d8-8397d993c189-model-cache\") pod \"s3-init-jmftq\" (UID: \"5258d5f9-6236-4bd2-80d8-8397d993c189\") " pod="kserve/s3-init-jmftq"
Jul 28 17:34:13.519852 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:34:13.519714    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r5dl\" (UniqueName: \"kubernetes.io/projected/5258d5f9-6236-4bd2-80d8-8397d993c189-kube-api-access-8r5dl\") pod \"s3-init-jmftq\" (UID: \"5258d5f9-6236-4bd2-80d8-8397d993c189\") " pod="kserve/s3-init-jmftq"
Jul 28 17:34:13.520058 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:34:13.520039    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5258d5f9-6236-4bd2-80d8-8397d993c189-model-cache\") pod \"s3-init-jmftq\" (UID: \"5258d5f9-6236-4bd2-80d8-8397d993c189\") " pod="kserve/s3-init-jmftq"
Jul 28 17:34:13.529400 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:34:13.529375    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r5dl\" (UniqueName: \"kubernetes.io/projected/5258d5f9-6236-4bd2-80d8-8397d993c189-kube-api-access-8r5dl\") pod \"s3-init-jmftq\" (UID: \"5258d5f9-6236-4bd2-80d8-8397d993c189\") " pod="kserve/s3-init-jmftq"
Jul 28 17:34:13.620385 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:34:13.620311    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-jmftq"
Jul 28 17:34:13.751421 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:34:13.751398    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-jmftq"]
Jul 28 17:34:13.753349 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:34:13.753321    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5258d5f9_6236_4bd2_80d8_8397d993c189.slice/crio-d740a656a968c9dfdf24a439ca8021756e231b1282deeccac6dd754b55c05f5a WatchSource:0}: Error finding container d740a656a968c9dfdf24a439ca8021756e231b1282deeccac6dd754b55c05f5a: Status 404 returned error can't find the container with id d740a656a968c9dfdf24a439ca8021756e231b1282deeccac6dd754b55c05f5a
Jul 28 17:34:14.176366 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:34:14.176333    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jmftq" event={"ID":"5258d5f9-6236-4bd2-80d8-8397d993c189","Type":"ContainerStarted","Data":"d740a656a968c9dfdf24a439ca8021756e231b1282deeccac6dd754b55c05f5a"}
Jul 28 17:34:17.193742 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:34:17.193704    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jmftq" event={"ID":"5258d5f9-6236-4bd2-80d8-8397d993c189","Type":"ContainerStarted","Data":"6ec38e228a2d53795f8ac4f8d9ca08b04d60ea56ed015f3ca5793373b1ecc32c"}
Jul 28 17:34:19.208451 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:34:19.208411    2543 generic.go:356] "Generic (PLEG): container finished" podID="5258d5f9-6236-4bd2-80d8-8397d993c189" containerID="6ec38e228a2d53795f8ac4f8d9ca08b04d60ea56ed015f3ca5793373b1ecc32c" exitCode=0
Jul 28 17:34:19.208810 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:34:19.208487    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jmftq" event={"ID":"5258d5f9-6236-4bd2-80d8-8397d993c189","Type":"ContainerDied","Data":"6ec38e228a2d53795f8ac4f8d9ca08b04d60ea56ed015f3ca5793373b1ecc32c"}
Jul 28 17:34:23.658098 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:34:23.658076    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 17:34:24.241555 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:34:24.241520    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jmftq" event={"ID":"5258d5f9-6236-4bd2-80d8-8397d993c189","Type":"ContainerStarted","Data":"532e78250112ac714cad8363cf045756bb51c939e2ecc6d10e9e87dd8244d138"}
Jul 28 17:34:24.258788 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:34:24.258745    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-jmftq" podStartSLOduration=1.505440853 podStartE2EDuration="11.258729395s" totalImagesPullingTime="7.676374606s" totalInitContainerRuntime="2.076913936s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:34:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:34:24.257947031 +0000 UTC m=+667.664220773" watchObservedRunningTime="2026-07-28 17:34:24.258729395 +0000 UTC m=+667.665003135"
Jul 28 17:35:29.567655 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:35:29.567615    2543 generic.go:356] "Generic (PLEG): container finished" podID="5258d5f9-6236-4bd2-80d8-8397d993c189" containerID="532e78250112ac714cad8363cf045756bb51c939e2ecc6d10e9e87dd8244d138" exitCode=0
Jul 28 17:35:29.568100 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:35:29.567688    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jmftq" event={"ID":"5258d5f9-6236-4bd2-80d8-8397d993c189","Type":"ContainerDied","Data":"532e78250112ac714cad8363cf045756bb51c939e2ecc6d10e9e87dd8244d138"}
Jul 28 17:35:30.709549 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:35:30.709523    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-jmftq"
Jul 28 17:35:30.782293 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:35:30.782263    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5258d5f9-6236-4bd2-80d8-8397d993c189-model-cache\") pod \"5258d5f9-6236-4bd2-80d8-8397d993c189\" (UID: \"5258d5f9-6236-4bd2-80d8-8397d993c189\") "
Jul 28 17:35:30.782473 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:35:30.782358    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8r5dl\" (UniqueName: \"kubernetes.io/projected/5258d5f9-6236-4bd2-80d8-8397d993c189-kube-api-access-8r5dl\") pod \"5258d5f9-6236-4bd2-80d8-8397d993c189\" (UID: \"5258d5f9-6236-4bd2-80d8-8397d993c189\") "
Jul 28 17:35:30.784500 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:35:30.784471    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5258d5f9-6236-4bd2-80d8-8397d993c189-kube-api-access-8r5dl" (OuterVolumeSpecName: "kube-api-access-8r5dl") pod "5258d5f9-6236-4bd2-80d8-8397d993c189" (UID: "5258d5f9-6236-4bd2-80d8-8397d993c189"). InnerVolumeSpecName "kube-api-access-8r5dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:35:30.801277 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:35:30.801249    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5258d5f9-6236-4bd2-80d8-8397d993c189-model-cache" (OuterVolumeSpecName: "model-cache") pod "5258d5f9-6236-4bd2-80d8-8397d993c189" (UID: "5258d5f9-6236-4bd2-80d8-8397d993c189"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:35:30.883507 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:35:30.883438    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r5dl\" (UniqueName: \"kubernetes.io/projected/5258d5f9-6236-4bd2-80d8-8397d993c189-kube-api-access-8r5dl\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:35:30.883507 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:35:30.883463    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5258d5f9-6236-4bd2-80d8-8397d993c189-model-cache\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:35:31.580456 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:35:31.580426    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-jmftq"
Jul 28 17:35:31.580456 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:35:31.580448    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jmftq" event={"ID":"5258d5f9-6236-4bd2-80d8-8397d993c189","Type":"ContainerDied","Data":"d740a656a968c9dfdf24a439ca8021756e231b1282deeccac6dd754b55c05f5a"}
Jul 28 17:35:31.580665 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:35:31.580479    2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d740a656a968c9dfdf24a439ca8021756e231b1282deeccac6dd754b55c05f5a"
Jul 28 17:35:41.066981 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:35:41.066946    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bb5d8fd44-hr7k8"]
Jul 28 17:36:06.086315 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:06.086247    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-bb5d8fd44-hr7k8" podUID="7a0bd377-c6b3-4361-bcb2-ed02ea99659a" containerName="console" containerID="cri-o://b98fa01be5e450f2f4601ed86e4cdb187bf82976bc6b4b5357600fa8f525d1cf" gracePeriod=15
Jul 28 17:36:06.333533 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:06.333504    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb5d8fd44-hr7k8"
Jul 28 17:36:06.481049 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:06.480956    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-console-config\") pod \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\" (UID: \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\") "
Jul 28 17:36:06.481049 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:06.481041    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-oauth-serving-cert\") pod \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\" (UID: \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\") "
Jul 28 17:36:06.481310 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:06.481083    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-console-oauth-config\") pod \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\" (UID: \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\") "
Jul 28 17:36:06.481310 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:06.481109    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-service-ca\") pod \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\" (UID: \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\") "
Jul 28 17:36:06.481310 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:06.481178    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-trusted-ca-bundle\") pod \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\" (UID: \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\") "
Jul 28 17:36:06.481310 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:06.481207    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-console-serving-cert\") pod \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\" (UID: \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\") "
Jul 28 17:36:06.481310 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:06.481234    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nrtr\" (UniqueName: \"kubernetes.io/projected/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-kube-api-access-6nrtr\") pod \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\" (UID: \"7a0bd377-c6b3-4361-bcb2-ed02ea99659a\") "
Jul 28 17:36:06.481551 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:06.481434    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-console-config" (OuterVolumeSpecName: "console-config") pod "7a0bd377-c6b3-4361-bcb2-ed02ea99659a" (UID: "7a0bd377-c6b3-4361-bcb2-ed02ea99659a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:36:06.481551 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:06.481511    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7a0bd377-c6b3-4361-bcb2-ed02ea99659a" (UID: "7a0bd377-c6b3-4361-bcb2-ed02ea99659a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:36:06.481551 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:06.481532    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-service-ca" (OuterVolumeSpecName: "service-ca") pod "7a0bd377-c6b3-4361-bcb2-ed02ea99659a" (UID: "7a0bd377-c6b3-4361-bcb2-ed02ea99659a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:36:06.481712 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:06.481622    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7a0bd377-c6b3-4361-bcb2-ed02ea99659a" (UID: "7a0bd377-c6b3-4361-bcb2-ed02ea99659a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:36:06.483269 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:06.483246    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7a0bd377-c6b3-4361-bcb2-ed02ea99659a" (UID: "7a0bd377-c6b3-4361-bcb2-ed02ea99659a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:36:06.483654 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:06.483629    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-kube-api-access-6nrtr" (OuterVolumeSpecName: "kube-api-access-6nrtr") pod "7a0bd377-c6b3-4361-bcb2-ed02ea99659a" (UID: "7a0bd377-c6b3-4361-bcb2-ed02ea99659a"). InnerVolumeSpecName "kube-api-access-6nrtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:36:06.483727 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:06.483696    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7a0bd377-c6b3-4361-bcb2-ed02ea99659a" (UID: "7a0bd377-c6b3-4361-bcb2-ed02ea99659a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:36:06.581844 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:06.581805    2543 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-oauth-serving-cert\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:36:06.581844 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:06.581840    2543 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-console-oauth-config\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:36:06.581844 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:06.581849    2543 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-service-ca\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:36:06.581844 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:06.581858    2543 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-trusted-ca-bundle\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:36:06.582084 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:06.581867    2543 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-console-serving-cert\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:36:06.582084 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:06.581876    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nrtr\" (UniqueName: \"kubernetes.io/projected/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-kube-api-access-6nrtr\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:36:06.582084 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:06.581885    2543 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a0bd377-c6b3-4361-bcb2-ed02ea99659a-console-config\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:36:06.758628 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:06.758598    2543 generic.go:356] "Generic (PLEG): container finished" podID="7a0bd377-c6b3-4361-bcb2-ed02ea99659a" containerID="b98fa01be5e450f2f4601ed86e4cdb187bf82976bc6b4b5357600fa8f525d1cf" exitCode=0
Jul 28 17:36:06.758784 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:06.758656    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb5d8fd44-hr7k8"
Jul 28 17:36:06.758784 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:06.758687    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb5d8fd44-hr7k8" event={"ID":"7a0bd377-c6b3-4361-bcb2-ed02ea99659a","Type":"ContainerDied","Data":"b98fa01be5e450f2f4601ed86e4cdb187bf82976bc6b4b5357600fa8f525d1cf"}
Jul 28 17:36:06.758784 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:06.758744    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb5d8fd44-hr7k8" event={"ID":"7a0bd377-c6b3-4361-bcb2-ed02ea99659a","Type":"ContainerDied","Data":"4370a787bff300af91948b25c765241fd5b378f6c681b0e7b670bea49303622f"}
Jul 28 17:36:06.758784 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:06.758772    2543 scope.go:117] "RemoveContainer" containerID="b98fa01be5e450f2f4601ed86e4cdb187bf82976bc6b4b5357600fa8f525d1cf"
Jul 28 17:36:06.768027 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:06.768009    2543 scope.go:117] "RemoveContainer" containerID="b98fa01be5e450f2f4601ed86e4cdb187bf82976bc6b4b5357600fa8f525d1cf"
Jul 28 17:36:06.768314 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:36:06.768294    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b98fa01be5e450f2f4601ed86e4cdb187bf82976bc6b4b5357600fa8f525d1cf\": container with ID starting with b98fa01be5e450f2f4601ed86e4cdb187bf82976bc6b4b5357600fa8f525d1cf not found: ID does not exist" containerID="b98fa01be5e450f2f4601ed86e4cdb187bf82976bc6b4b5357600fa8f525d1cf"
Jul 28 17:36:06.768361 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:06.768323    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b98fa01be5e450f2f4601ed86e4cdb187bf82976bc6b4b5357600fa8f525d1cf"} err="failed to get container status \"b98fa01be5e450f2f4601ed86e4cdb187bf82976bc6b4b5357600fa8f525d1cf\": rpc error: code = NotFound desc = could not find container \"b98fa01be5e450f2f4601ed86e4cdb187bf82976bc6b4b5357600fa8f525d1cf\": container with ID starting with b98fa01be5e450f2f4601ed86e4cdb187bf82976bc6b4b5357600fa8f525d1cf not found: ID does not exist"
Jul 28 17:36:06.781396 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:06.781359    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bb5d8fd44-hr7k8"]
Jul 28 17:36:06.785928 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:06.785906    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bb5d8fd44-hr7k8"]
Jul 28 17:36:07.152462 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:07.152384    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a0bd377-c6b3-4361-bcb2-ed02ea99659a" path="/var/lib/kubelet/pods/7a0bd377-c6b3-4361-bcb2-ed02ea99659a/volumes"
Jul 28 17:36:07.911963 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:07.911932    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"]
Jul 28 17:36:07.912344 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:07.912327    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a0bd377-c6b3-4361-bcb2-ed02ea99659a" containerName="console"
Jul 28 17:36:07.912440 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:07.912347    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a0bd377-c6b3-4361-bcb2-ed02ea99659a" containerName="console"
Jul 28 17:36:07.912440 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:07.912362    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5258d5f9-6236-4bd2-80d8-8397d993c189" containerName="download-hf-model"
Jul 28 17:36:07.912440 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:07.912371    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="5258d5f9-6236-4bd2-80d8-8397d993c189" containerName="download-hf-model"
Jul 28 17:36:07.912440 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:07.912390    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5258d5f9-6236-4bd2-80d8-8397d993c189" containerName="s3-init"
Jul 28 17:36:07.912440 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:07.912399    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="5258d5f9-6236-4bd2-80d8-8397d993c189" containerName="s3-init"
Jul 28 17:36:07.912693 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:07.912488    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="5258d5f9-6236-4bd2-80d8-8397d993c189" containerName="s3-init"
Jul 28 17:36:07.912693 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:07.912500    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a0bd377-c6b3-4361-bcb2-ed02ea99659a" containerName="console"
Jul 28 17:36:07.916609 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:07.916588    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"
Jul 28 17:36:07.919266 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:07.919242    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Jul 28 17:36:07.919266 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:07.919257    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Jul 28 17:36:07.919450 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:07.919304    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-pkw65\""
Jul 28 17:36:07.919450 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:07.919242    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Jul 28 17:36:07.919450 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:07.919253    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Jul 28 17:36:07.927194 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:07.927172    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"]
Jul 28 17:36:08.093467 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:08.093433    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fe6edc3-2f58-4bcb-912d-594051776ce7-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw\" (UID: \"2fe6edc3-2f58-4bcb-912d-594051776ce7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"
Jul 28 17:36:08.093625 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:08.093490    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fe6edc3-2f58-4bcb-912d-594051776ce7-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw\" (UID: \"2fe6edc3-2f58-4bcb-912d-594051776ce7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"
Jul 28 17:36:08.093625 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:08.093555    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fe6edc3-2f58-4bcb-912d-594051776ce7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw\" (UID: \"2fe6edc3-2f58-4bcb-912d-594051776ce7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"
Jul 28 17:36:08.093731 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:08.093617    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fe6edc3-2f58-4bcb-912d-594051776ce7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw\" (UID: \"2fe6edc3-2f58-4bcb-912d-594051776ce7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"
Jul 28 17:36:08.093731 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:08.093666    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fe6edc3-2f58-4bcb-912d-594051776ce7-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw\" (UID: \"2fe6edc3-2f58-4bcb-912d-594051776ce7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"
Jul 28 17:36:08.093731 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:08.093690    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7z9d\" (UniqueName: \"kubernetes.io/projected/2fe6edc3-2f58-4bcb-912d-594051776ce7-kube-api-access-k7z9d\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw\" (UID: \"2fe6edc3-2f58-4bcb-912d-594051776ce7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"
Jul 28 17:36:08.093877 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:08.093754    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fe6edc3-2f58-4bcb-912d-594051776ce7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw\" (UID: \"2fe6edc3-2f58-4bcb-912d-594051776ce7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"
Jul 28 17:36:08.194596 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:08.194508    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fe6edc3-2f58-4bcb-912d-594051776ce7-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw\" (UID: \"2fe6edc3-2f58-4bcb-912d-594051776ce7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"
Jul 28 17:36:08.194596 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:08.194563    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fe6edc3-2f58-4bcb-912d-594051776ce7-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw\" (UID: \"2fe6edc3-2f58-4bcb-912d-594051776ce7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"
Jul 28 17:36:08.194596 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:08.194594    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fe6edc3-2f58-4bcb-912d-594051776ce7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw\" (UID: \"2fe6edc3-2f58-4bcb-912d-594051776ce7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"
Jul 28 17:36:08.195234 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:08.194622    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fe6edc3-2f58-4bcb-912d-594051776ce7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw\" (UID: \"2fe6edc3-2f58-4bcb-912d-594051776ce7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"
Jul 28 17:36:08.195234 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:08.194677    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fe6edc3-2f58-4bcb-912d-594051776ce7-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw\" (UID: \"2fe6edc3-2f58-4bcb-912d-594051776ce7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"
Jul 28 17:36:08.195234 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:08.194716    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7z9d\" (UniqueName: \"kubernetes.io/projected/2fe6edc3-2f58-4bcb-912d-594051776ce7-kube-api-access-k7z9d\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw\" (UID: \"2fe6edc3-2f58-4bcb-912d-594051776ce7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"
Jul 28 17:36:08.195234 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:08.194765    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fe6edc3-2f58-4bcb-912d-594051776ce7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw\" (UID: \"2fe6edc3-2f58-4bcb-912d-594051776ce7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"
Jul 28 17:36:08.195234 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:08.195003    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fe6edc3-2f58-4bcb-912d-594051776ce7-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw\" (UID: \"2fe6edc3-2f58-4bcb-912d-594051776ce7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"
Jul 28 17:36:08.195234 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:08.195031    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fe6edc3-2f58-4bcb-912d-594051776ce7-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw\" (UID: \"2fe6edc3-2f58-4bcb-912d-594051776ce7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"
Jul 28 17:36:08.195234 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:08.195054    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fe6edc3-2f58-4bcb-912d-594051776ce7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw\" (UID: \"2fe6edc3-2f58-4bcb-912d-594051776ce7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"
Jul 28 17:36:08.195505 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:08.195269    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fe6edc3-2f58-4bcb-912d-594051776ce7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw\" (UID: \"2fe6edc3-2f58-4bcb-912d-594051776ce7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"
Jul 28 17:36:08.197146 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:08.197109    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fe6edc3-2f58-4bcb-912d-594051776ce7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw\" (UID: \"2fe6edc3-2f58-4bcb-912d-594051776ce7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"
Jul 28 17:36:08.197302 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:08.197264    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fe6edc3-2f58-4bcb-912d-594051776ce7-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw\" (UID: \"2fe6edc3-2f58-4bcb-912d-594051776ce7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"
Jul 28 17:36:08.205517 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:08.205396    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7z9d\" (UniqueName: \"kubernetes.io/projected/2fe6edc3-2f58-4bcb-912d-594051776ce7-kube-api-access-k7z9d\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw\" (UID: \"2fe6edc3-2f58-4bcb-912d-594051776ce7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"
Jul 28 17:36:08.226904 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:08.226878    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"
Jul 28 17:36:08.393431 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:08.393394    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"]
Jul 28 17:36:08.397170 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:36:08.397123    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe6edc3_2f58_4bcb_912d_594051776ce7.slice/crio-608b4874142a8c84a6a4edc5cfa724b395b46d80ac63e4aac13557dc79138926 WatchSource:0}: Error finding container 608b4874142a8c84a6a4edc5cfa724b395b46d80ac63e4aac13557dc79138926: Status 404 returned error can't find the container with id 608b4874142a8c84a6a4edc5cfa724b395b46d80ac63e4aac13557dc79138926
Jul 28 17:36:08.779796 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:08.779756    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw" event={"ID":"2fe6edc3-2f58-4bcb-912d-594051776ce7","Type":"ContainerStarted","Data":"3dfdddf1dae82a3febb3b2d4f466ccee5eb8a98e913b840211f89c89d24a3532"}
Jul 28 17:36:08.779983 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:08.779845    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw" event={"ID":"2fe6edc3-2f58-4bcb-912d-594051776ce7","Type":"ContainerStarted","Data":"608b4874142a8c84a6a4edc5cfa724b395b46d80ac63e4aac13557dc79138926"}
Jul 28 17:36:14.819170 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:14.819056    2543 generic.go:356] "Generic (PLEG): container finished" podID="2fe6edc3-2f58-4bcb-912d-594051776ce7" containerID="3dfdddf1dae82a3febb3b2d4f466ccee5eb8a98e913b840211f89c89d24a3532" exitCode=0
Jul 28 17:36:14.819170 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:14.819111    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw" event={"ID":"2fe6edc3-2f58-4bcb-912d-594051776ce7","Type":"ContainerDied","Data":"3dfdddf1dae82a3febb3b2d4f466ccee5eb8a98e913b840211f89c89d24a3532"}
Jul 28 17:36:14.820535 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:14.820518    2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:36:44.011649 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:44.011611    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw" event={"ID":"2fe6edc3-2f58-4bcb-912d-594051776ce7","Type":"ContainerStarted","Data":"74a8ec09a34eb1730dc155c3e82cde12069517ab0bb6be79b26c431c0c5b5217"}
Jul 28 17:36:44.050873 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:44.050814    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw" podStartSLOduration=2.84457231 podStartE2EDuration="37.050799125s" totalImagesPullingTime="28.16243748s" totalInitContainerRuntime="6.043789335s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:36:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:36:44.050610354 +0000 UTC m=+807.456884106" watchObservedRunningTime="2026-07-28 17:36:44.050799125 +0000 UTC m=+807.457072900"
Jul 28 17:36:44.227536 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:44.227493    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"
Jul 28 17:36:48.227916 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:48.227882    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"
Jul 28 17:36:48.229493 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:48.229451    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw" podUID="2fe6edc3-2f58-4bcb-912d-594051776ce7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 28 17:36:58.227833 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:36:58.227790    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw" podUID="2fe6edc3-2f58-4bcb-912d-594051776ce7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 28 17:37:08.227913 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:37:08.227847    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw" podUID="2fe6edc3-2f58-4bcb-912d-594051776ce7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 28 17:37:18.228394 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:37:18.228342    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw" podUID="2fe6edc3-2f58-4bcb-912d-594051776ce7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 28 17:37:28.228096 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:37:28.228056    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw" podUID="2fe6edc3-2f58-4bcb-912d-594051776ce7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 28 17:37:38.228417 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:37:38.228363    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw" podUID="2fe6edc3-2f58-4bcb-912d-594051776ce7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 28 17:37:48.228237 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:37:48.228189    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw" podUID="2fe6edc3-2f58-4bcb-912d-594051776ce7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 28 17:37:58.228110 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:37:58.228053    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw" podUID="2fe6edc3-2f58-4bcb-912d-594051776ce7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 28 17:38:08.227487 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:38:08.227432    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw" podUID="2fe6edc3-2f58-4bcb-912d-594051776ce7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 28 17:38:18.227967 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:38:18.227920    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw" podUID="2fe6edc3-2f58-4bcb-912d-594051776ce7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 28 17:38:28.228307 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:38:28.228256    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw" podUID="2fe6edc3-2f58-4bcb-912d-594051776ce7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 28 17:38:38.237221 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:38:38.237182    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"
Jul 28 17:38:38.245039 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:38:38.245003    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"
Jul 28 17:39:04.886958 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:04.886924    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"]
Jul 28 17:39:19.908874 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:19.908811    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw" podUID="2fe6edc3-2f58-4bcb-912d-594051776ce7" containerName="main" containerID="cri-o://74a8ec09a34eb1730dc155c3e82cde12069517ab0bb6be79b26c431c0c5b5217" gracePeriod=45
Jul 28 17:39:20.238182 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.238145    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw" podUID="2fe6edc3-2f58-4bcb-912d-594051776ce7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 28 17:39:20.666380 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.666355    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"
Jul 28 17:39:20.730147 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.730100    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fe6edc3-2f58-4bcb-912d-594051776ce7-tls-certs\") pod \"2fe6edc3-2f58-4bcb-912d-594051776ce7\" (UID: \"2fe6edc3-2f58-4bcb-912d-594051776ce7\") "
Jul 28 17:39:20.730336 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.730155    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fe6edc3-2f58-4bcb-912d-594051776ce7-model-cache\") pod \"2fe6edc3-2f58-4bcb-912d-594051776ce7\" (UID: \"2fe6edc3-2f58-4bcb-912d-594051776ce7\") "
Jul 28 17:39:20.730336 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.730182    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fe6edc3-2f58-4bcb-912d-594051776ce7-kserve-provision-location\") pod \"2fe6edc3-2f58-4bcb-912d-594051776ce7\" (UID: \"2fe6edc3-2f58-4bcb-912d-594051776ce7\") "
Jul 28 17:39:20.730336 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.730205    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fe6edc3-2f58-4bcb-912d-594051776ce7-dshm\") pod \"2fe6edc3-2f58-4bcb-912d-594051776ce7\" (UID: \"2fe6edc3-2f58-4bcb-912d-594051776ce7\") "
Jul 28 17:39:20.730336 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.730233    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fe6edc3-2f58-4bcb-912d-594051776ce7-home\") pod \"2fe6edc3-2f58-4bcb-912d-594051776ce7\" (UID: \"2fe6edc3-2f58-4bcb-912d-594051776ce7\") "
Jul 28 17:39:20.730336 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.730252    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fe6edc3-2f58-4bcb-912d-594051776ce7-tmp-dir\") pod \"2fe6edc3-2f58-4bcb-912d-594051776ce7\" (UID: \"2fe6edc3-2f58-4bcb-912d-594051776ce7\") "
Jul 28 17:39:20.730336 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.730309    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7z9d\" (UniqueName: \"kubernetes.io/projected/2fe6edc3-2f58-4bcb-912d-594051776ce7-kube-api-access-k7z9d\") pod \"2fe6edc3-2f58-4bcb-912d-594051776ce7\" (UID: \"2fe6edc3-2f58-4bcb-912d-594051776ce7\") "
Jul 28 17:39:20.730662 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.730408    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe6edc3-2f58-4bcb-912d-594051776ce7-model-cache" (OuterVolumeSpecName: "model-cache") pod "2fe6edc3-2f58-4bcb-912d-594051776ce7" (UID: "2fe6edc3-2f58-4bcb-912d-594051776ce7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:39:20.730662 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.730612    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fe6edc3-2f58-4bcb-912d-594051776ce7-model-cache\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:39:20.730931 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.730905    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe6edc3-2f58-4bcb-912d-594051776ce7-home" (OuterVolumeSpecName: "home") pod "2fe6edc3-2f58-4bcb-912d-594051776ce7" (UID: "2fe6edc3-2f58-4bcb-912d-594051776ce7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:39:20.732559 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.732537    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fe6edc3-2f58-4bcb-912d-594051776ce7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2fe6edc3-2f58-4bcb-912d-594051776ce7" (UID: "2fe6edc3-2f58-4bcb-912d-594051776ce7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:39:20.732658 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.732574    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fe6edc3-2f58-4bcb-912d-594051776ce7-kube-api-access-k7z9d" (OuterVolumeSpecName: "kube-api-access-k7z9d") pod "2fe6edc3-2f58-4bcb-912d-594051776ce7" (UID: "2fe6edc3-2f58-4bcb-912d-594051776ce7"). InnerVolumeSpecName "kube-api-access-k7z9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:39:20.732699 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.732662    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe6edc3-2f58-4bcb-912d-594051776ce7-dshm" (OuterVolumeSpecName: "dshm") pod "2fe6edc3-2f58-4bcb-912d-594051776ce7" (UID: "2fe6edc3-2f58-4bcb-912d-594051776ce7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:39:20.749729 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.749699    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe6edc3-2f58-4bcb-912d-594051776ce7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2fe6edc3-2f58-4bcb-912d-594051776ce7" (UID: "2fe6edc3-2f58-4bcb-912d-594051776ce7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:39:20.795195 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.795090    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe6edc3-2f58-4bcb-912d-594051776ce7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2fe6edc3-2f58-4bcb-912d-594051776ce7" (UID: "2fe6edc3-2f58-4bcb-912d-594051776ce7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:39:20.831091 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.831063    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fe6edc3-2f58-4bcb-912d-594051776ce7-tmp-dir\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:39:20.831091 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.831088    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7z9d\" (UniqueName: \"kubernetes.io/projected/2fe6edc3-2f58-4bcb-912d-594051776ce7-kube-api-access-k7z9d\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:39:20.831091 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.831098    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fe6edc3-2f58-4bcb-912d-594051776ce7-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:39:20.831091 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.831107    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fe6edc3-2f58-4bcb-912d-594051776ce7-kserve-provision-location\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:39:20.831091 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.831115    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fe6edc3-2f58-4bcb-912d-594051776ce7-dshm\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:39:20.831091 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.831122    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fe6edc3-2f58-4bcb-912d-594051776ce7-home\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:39:20.833601 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.833572    2543 generic.go:356] "Generic (PLEG): container finished" podID="2fe6edc3-2f58-4bcb-912d-594051776ce7" containerID="74a8ec09a34eb1730dc155c3e82cde12069517ab0bb6be79b26c431c0c5b5217" exitCode=0
Jul 28 17:39:20.833698 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.833631    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"
Jul 28 17:39:20.833698 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.833657    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw" event={"ID":"2fe6edc3-2f58-4bcb-912d-594051776ce7","Type":"ContainerDied","Data":"74a8ec09a34eb1730dc155c3e82cde12069517ab0bb6be79b26c431c0c5b5217"}
Jul 28 17:39:20.833771 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.833718    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw" event={"ID":"2fe6edc3-2f58-4bcb-912d-594051776ce7","Type":"ContainerDied","Data":"608b4874142a8c84a6a4edc5cfa724b395b46d80ac63e4aac13557dc79138926"}
Jul 28 17:39:20.833771 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.833746    2543 scope.go:117] "RemoveContainer" containerID="74a8ec09a34eb1730dc155c3e82cde12069517ab0bb6be79b26c431c0c5b5217"
Jul 28 17:39:20.848053 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.848036    2543 scope.go:117] "RemoveContainer" containerID="3dfdddf1dae82a3febb3b2d4f466ccee5eb8a98e913b840211f89c89d24a3532"
Jul 28 17:39:20.856112 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.856094    2543 scope.go:117] "RemoveContainer" containerID="74a8ec09a34eb1730dc155c3e82cde12069517ab0bb6be79b26c431c0c5b5217"
Jul 28 17:39:20.856396 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:39:20.856376    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74a8ec09a34eb1730dc155c3e82cde12069517ab0bb6be79b26c431c0c5b5217\": container with ID starting with 74a8ec09a34eb1730dc155c3e82cde12069517ab0bb6be79b26c431c0c5b5217 not found: ID does not exist" containerID="74a8ec09a34eb1730dc155c3e82cde12069517ab0bb6be79b26c431c0c5b5217"
Jul 28 17:39:20.856469 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.856403    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74a8ec09a34eb1730dc155c3e82cde12069517ab0bb6be79b26c431c0c5b5217"} err="failed to get container status \"74a8ec09a34eb1730dc155c3e82cde12069517ab0bb6be79b26c431c0c5b5217\": rpc error: code = NotFound desc = could not find container \"74a8ec09a34eb1730dc155c3e82cde12069517ab0bb6be79b26c431c0c5b5217\": container with ID starting with 74a8ec09a34eb1730dc155c3e82cde12069517ab0bb6be79b26c431c0c5b5217 not found: ID does not exist"
Jul 28 17:39:20.856469 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.856419    2543 scope.go:117] "RemoveContainer" containerID="3dfdddf1dae82a3febb3b2d4f466ccee5eb8a98e913b840211f89c89d24a3532"
Jul 28 17:39:20.856663 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:39:20.856646    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dfdddf1dae82a3febb3b2d4f466ccee5eb8a98e913b840211f89c89d24a3532\": container with ID starting with 3dfdddf1dae82a3febb3b2d4f466ccee5eb8a98e913b840211f89c89d24a3532 not found: ID does not exist" containerID="3dfdddf1dae82a3febb3b2d4f466ccee5eb8a98e913b840211f89c89d24a3532"
Jul 28 17:39:20.856702 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.856669    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dfdddf1dae82a3febb3b2d4f466ccee5eb8a98e913b840211f89c89d24a3532"} err="failed to get container status \"3dfdddf1dae82a3febb3b2d4f466ccee5eb8a98e913b840211f89c89d24a3532\": rpc error: code = NotFound desc = could not find container \"3dfdddf1dae82a3febb3b2d4f466ccee5eb8a98e913b840211f89c89d24a3532\": container with ID starting with 3dfdddf1dae82a3febb3b2d4f466ccee5eb8a98e913b840211f89c89d24a3532 not found: ID does not exist"
Jul 28 17:39:20.863401 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.863380    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"]
Jul 28 17:39:20.866915 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:20.866894    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5b7665b9b8nppnw"]
Jul 28 17:39:21.152659 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:39:21.152579    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fe6edc3-2f58-4bcb-912d-594051776ce7" path="/var/lib/kubelet/pods/2fe6edc3-2f58-4bcb-912d-594051776ce7/volumes"
Jul 28 17:40:01.123519 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.123486    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 28 17:40:01.123953 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.123831    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fe6edc3-2f58-4bcb-912d-594051776ce7" containerName="storage-initializer"
Jul 28 17:40:01.123953 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.123842    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fe6edc3-2f58-4bcb-912d-594051776ce7" containerName="storage-initializer"
Jul 28 17:40:01.123953 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.123862    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fe6edc3-2f58-4bcb-912d-594051776ce7" containerName="main"
Jul 28 17:40:01.123953 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.123868    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fe6edc3-2f58-4bcb-912d-594051776ce7" containerName="main"
Jul 28 17:40:01.123953 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.123922    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fe6edc3-2f58-4bcb-912d-594051776ce7" containerName="main"
Jul 28 17:40:01.127046 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.127029    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:40:01.129537 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.129519    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Jul 28 17:40:01.129684 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.129612    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 28 17:40:01.130325 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.130307    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-m4fbq\""
Jul 28 17:40:01.130426 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.130322    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Jul 28 17:40:01.130426 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.130335    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Jul 28 17:40:01.130530 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.130430    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-rgfwt\""
Jul 28 17:40:01.159277 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.159249    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 28 17:40:01.251708 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.251674    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:40:01.251871 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.251716    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:40:01.251871 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.251741    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:40:01.251871 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.251795    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:40:01.251871 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.251842    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:40:01.252063 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.252042    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8rq9\" (UniqueName: \"kubernetes.io/projected/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-kube-api-access-h8rq9\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:40:01.252119 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.252100    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:40:01.352990 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.352957    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:40:01.352990 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.352991    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:40:01.353253 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.353019    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:40:01.353253 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.353057    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:40:01.353253 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.353086    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8rq9\" (UniqueName: \"kubernetes.io/projected/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-kube-api-access-h8rq9\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:40:01.353253 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.353121    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:40:01.353253 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.353238    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:40:01.353540 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.353331    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:40:01.353540 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.353492    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:40:01.353641 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.353568    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:40:01.353641 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.353616    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:40:01.355350 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.355332    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:40:01.355538 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.355521    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:40:01.361252 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.361234    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8rq9\" (UniqueName: \"kubernetes.io/projected/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-kube-api-access-h8rq9\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:40:01.437524 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.437459    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:40:01.570272 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:01.570243    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 28 17:40:01.572802 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:40:01.572774    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b7c1fd1_111d_48c2_b52f_92aaba1a7c14.slice/crio-373ecd6b7bee027b8ab56537cac0e77607090debc7e53bfb42f717cd032ace32 WatchSource:0}: Error finding container 373ecd6b7bee027b8ab56537cac0e77607090debc7e53bfb42f717cd032ace32: Status 404 returned error can't find the container with id 373ecd6b7bee027b8ab56537cac0e77607090debc7e53bfb42f717cd032ace32
Jul 28 17:40:02.047159 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:02.047103    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14","Type":"ContainerStarted","Data":"c8d0294f7c95a5d5205fc0c451cfa4daca5abb216f8469895a2478071329ab60"}
Jul 28 17:40:02.047339 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:02.047181    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14","Type":"ContainerStarted","Data":"373ecd6b7bee027b8ab56537cac0e77607090debc7e53bfb42f717cd032ace32"}
Jul 28 17:40:11.093073 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:11.093041    2543 generic.go:356] "Generic (PLEG): container finished" podID="2b7c1fd1-111d-48c2-b52f-92aaba1a7c14" containerID="c8d0294f7c95a5d5205fc0c451cfa4daca5abb216f8469895a2478071329ab60" exitCode=0
Jul 28 17:40:11.093477 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:11.093099    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14","Type":"ContainerDied","Data":"c8d0294f7c95a5d5205fc0c451cfa4daca5abb216f8469895a2478071329ab60"}
Jul 28 17:40:12.100531 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:12.100493    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14","Type":"ContainerStarted","Data":"8f49d645c46126ef3491fc452cc8d0cb867eaf206544e0bded9b37d4592083dd"}
Jul 28 17:40:12.124024 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:12.123963    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.186824753 podStartE2EDuration="11.123940905s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.937116152s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:40:12.122553935 +0000 UTC m=+1015.528827667" watchObservedRunningTime="2026-07-28 17:40:12.123940905 +0000 UTC m=+1015.530214647"
Jul 28 17:40:12.438179 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:12.438076    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:40:21.437618 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:21.437576    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:40:21.439140 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:21.439102    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="2b7c1fd1-111d-48c2-b52f-92aaba1a7c14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 28 17:40:31.438124 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:31.438071    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="2b7c1fd1-111d-48c2-b52f-92aaba1a7c14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 28 17:40:41.438632 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:41.438591    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="2b7c1fd1-111d-48c2-b52f-92aaba1a7c14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 28 17:40:51.438858 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:40:51.438817    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="2b7c1fd1-111d-48c2-b52f-92aaba1a7c14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 28 17:41:01.437896 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:41:01.437845    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="2b7c1fd1-111d-48c2-b52f-92aaba1a7c14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 28 17:41:11.438353 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:41:11.438261    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="2b7c1fd1-111d-48c2-b52f-92aaba1a7c14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 28 17:41:21.438416 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:41:21.438374    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="2b7c1fd1-111d-48c2-b52f-92aaba1a7c14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 28 17:41:31.438500 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:41:31.438446    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="2b7c1fd1-111d-48c2-b52f-92aaba1a7c14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 28 17:41:41.438659 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:41:41.438610    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="2b7c1fd1-111d-48c2-b52f-92aaba1a7c14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 28 17:41:51.447859 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:41:51.447831    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:41:51.455404 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:41:51.455375    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:42:43.327841 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:43.327751    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 28 17:42:58.350421 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:58.350377    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="2b7c1fd1-111d-48c2-b52f-92aaba1a7c14" containerName="main" containerID="cri-o://8f49d645c46126ef3491fc452cc8d0cb867eaf206544e0bded9b37d4592083dd" gracePeriod=45
Jul 28 17:42:58.448800 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:58.448751    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="2b7c1fd1-111d-48c2-b52f-92aaba1a7c14" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 28 17:42:59.103917 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.103895    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:42:59.188818 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.188749    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-tmp-dir\") pod \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\" (UID: \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\") "
Jul 28 17:42:59.188818 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.188797    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-dshm\") pod \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\" (UID: \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\") "
Jul 28 17:42:59.189021 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.188828    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-kserve-provision-location\") pod \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\" (UID: \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\") "
Jul 28 17:42:59.189021 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.188849    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-tls-certs\") pod \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\" (UID: \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\") "
Jul 28 17:42:59.189021 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.188866    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-model-cache\") pod \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\" (UID: \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\") "
Jul 28 17:42:59.189021 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.188910    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-home\") pod \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\" (UID: \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\") "
Jul 28 17:42:59.189021 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.188924    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8rq9\" (UniqueName: \"kubernetes.io/projected/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-kube-api-access-h8rq9\") pod \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\" (UID: \"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14\") "
Jul 28 17:42:59.189321 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.189232    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-model-cache" (OuterVolumeSpecName: "model-cache") pod "2b7c1fd1-111d-48c2-b52f-92aaba1a7c14" (UID: "2b7c1fd1-111d-48c2-b52f-92aaba1a7c14"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:42:59.189689 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.189656    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-home" (OuterVolumeSpecName: "home") pod "2b7c1fd1-111d-48c2-b52f-92aaba1a7c14" (UID: "2b7c1fd1-111d-48c2-b52f-92aaba1a7c14"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:42:59.191352 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.191306    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2b7c1fd1-111d-48c2-b52f-92aaba1a7c14" (UID: "2b7c1fd1-111d-48c2-b52f-92aaba1a7c14"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:42:59.191445 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.191380    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-dshm" (OuterVolumeSpecName: "dshm") pod "2b7c1fd1-111d-48c2-b52f-92aaba1a7c14" (UID: "2b7c1fd1-111d-48c2-b52f-92aaba1a7c14"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:42:59.191445 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.191396    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-kube-api-access-h8rq9" (OuterVolumeSpecName: "kube-api-access-h8rq9") pod "2b7c1fd1-111d-48c2-b52f-92aaba1a7c14" (UID: "2b7c1fd1-111d-48c2-b52f-92aaba1a7c14"). InnerVolumeSpecName "kube-api-access-h8rq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:42:59.207267 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.207236    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2b7c1fd1-111d-48c2-b52f-92aaba1a7c14" (UID: "2b7c1fd1-111d-48c2-b52f-92aaba1a7c14"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:42:59.260340 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.260301    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2b7c1fd1-111d-48c2-b52f-92aaba1a7c14" (UID: "2b7c1fd1-111d-48c2-b52f-92aaba1a7c14"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:42:59.290324 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.290302    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-kserve-provision-location\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:42:59.290324 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.290324    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:42:59.290461 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.290334    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-model-cache\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:42:59.290461 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.290342    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-home\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:42:59.290461 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.290350    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8rq9\" (UniqueName: \"kubernetes.io/projected/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-kube-api-access-h8rq9\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:42:59.290461 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.290359    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-tmp-dir\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:42:59.290461 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.290368    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14-dshm\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:42:59.968792 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.968759    2543 generic.go:356] "Generic (PLEG): container finished" podID="2b7c1fd1-111d-48c2-b52f-92aaba1a7c14" containerID="8f49d645c46126ef3491fc452cc8d0cb867eaf206544e0bded9b37d4592083dd" exitCode=0
Jul 28 17:42:59.969257 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.968810    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 28 17:42:59.969257 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.968852    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14","Type":"ContainerDied","Data":"8f49d645c46126ef3491fc452cc8d0cb867eaf206544e0bded9b37d4592083dd"}
Jul 28 17:42:59.969257 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.968911    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"2b7c1fd1-111d-48c2-b52f-92aaba1a7c14","Type":"ContainerDied","Data":"373ecd6b7bee027b8ab56537cac0e77607090debc7e53bfb42f717cd032ace32"}
Jul 28 17:42:59.969257 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.968941    2543 scope.go:117] "RemoveContainer" containerID="8f49d645c46126ef3491fc452cc8d0cb867eaf206544e0bded9b37d4592083dd"
Jul 28 17:42:59.978112 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.978094    2543 scope.go:117] "RemoveContainer" containerID="c8d0294f7c95a5d5205fc0c451cfa4daca5abb216f8469895a2478071329ab60"
Jul 28 17:42:59.986000 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.985986    2543 scope.go:117] "RemoveContainer" containerID="8f49d645c46126ef3491fc452cc8d0cb867eaf206544e0bded9b37d4592083dd"
Jul 28 17:42:59.986245 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:42:59.986229    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f49d645c46126ef3491fc452cc8d0cb867eaf206544e0bded9b37d4592083dd\": container with ID starting with 8f49d645c46126ef3491fc452cc8d0cb867eaf206544e0bded9b37d4592083dd not found: ID does not exist" containerID="8f49d645c46126ef3491fc452cc8d0cb867eaf206544e0bded9b37d4592083dd"
Jul 28 17:42:59.986396 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.986253    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f49d645c46126ef3491fc452cc8d0cb867eaf206544e0bded9b37d4592083dd"} err="failed to get container status \"8f49d645c46126ef3491fc452cc8d0cb867eaf206544e0bded9b37d4592083dd\": rpc error: code = NotFound desc = could not find container \"8f49d645c46126ef3491fc452cc8d0cb867eaf206544e0bded9b37d4592083dd\": container with ID starting with 8f49d645c46126ef3491fc452cc8d0cb867eaf206544e0bded9b37d4592083dd not found: ID does not exist"
Jul 28 17:42:59.986396 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.986274    2543 scope.go:117] "RemoveContainer" containerID="c8d0294f7c95a5d5205fc0c451cfa4daca5abb216f8469895a2478071329ab60"
Jul 28 17:42:59.986488 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:42:59.986472    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8d0294f7c95a5d5205fc0c451cfa4daca5abb216f8469895a2478071329ab60\": container with ID starting with c8d0294f7c95a5d5205fc0c451cfa4daca5abb216f8469895a2478071329ab60 not found: ID does not exist" containerID="c8d0294f7c95a5d5205fc0c451cfa4daca5abb216f8469895a2478071329ab60"
Jul 28 17:42:59.986526 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.986495    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8d0294f7c95a5d5205fc0c451cfa4daca5abb216f8469895a2478071329ab60"} err="failed to get container status \"c8d0294f7c95a5d5205fc0c451cfa4daca5abb216f8469895a2478071329ab60\": rpc error: code = NotFound desc = could not find container \"c8d0294f7c95a5d5205fc0c451cfa4daca5abb216f8469895a2478071329ab60\": container with ID starting with c8d0294f7c95a5d5205fc0c451cfa4daca5abb216f8469895a2478071329ab60 not found: ID does not exist"
Jul 28 17:42:59.993657 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.993638    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 28 17:42:59.996952 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:42:59.996933    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 28 17:43:01.052025 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.051991    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"]
Jul 28 17:43:01.052399 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.052328    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b7c1fd1-111d-48c2-b52f-92aaba1a7c14" containerName="main"
Jul 28 17:43:01.052399 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.052340    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b7c1fd1-111d-48c2-b52f-92aaba1a7c14" containerName="main"
Jul 28 17:43:01.052399 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.052354    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b7c1fd1-111d-48c2-b52f-92aaba1a7c14" containerName="storage-initializer"
Jul 28 17:43:01.052399 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.052360    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b7c1fd1-111d-48c2-b52f-92aaba1a7c14" containerName="storage-initializer"
Jul 28 17:43:01.052540 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.052412    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b7c1fd1-111d-48c2-b52f-92aaba1a7c14" containerName="main"
Jul 28 17:43:01.057000 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.056981    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"
Jul 28 17:43:01.060490 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.060463    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Jul 28 17:43:01.061473 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.061446    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-l5z6k\""
Jul 28 17:43:01.061611 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.061532    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Jul 28 17:43:01.061611 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.061579    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 28 17:43:01.061724 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.061651    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Jul 28 17:43:01.094977 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.094952    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"]
Jul 28 17:43:01.106207 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.106177    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2b74082-3091-44cb-a954-92c3b76111c6-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-97z2j\" (UID: \"e2b74082-3091-44cb-a954-92c3b76111c6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"
Jul 28 17:43:01.106332 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.106218    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2b74082-3091-44cb-a954-92c3b76111c6-home\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-97z2j\" (UID: \"e2b74082-3091-44cb-a954-92c3b76111c6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"
Jul 28 17:43:01.106332 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.106288    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2b74082-3091-44cb-a954-92c3b76111c6-tmp-dir\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-97z2j\" (UID: \"e2b74082-3091-44cb-a954-92c3b76111c6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"
Jul 28 17:43:01.106409 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.106339    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2b74082-3091-44cb-a954-92c3b76111c6-model-cache\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-97z2j\" (UID: \"e2b74082-3091-44cb-a954-92c3b76111c6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"
Jul 28 17:43:01.106409 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.106369    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2b74082-3091-44cb-a954-92c3b76111c6-dshm\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-97z2j\" (UID: \"e2b74082-3091-44cb-a954-92c3b76111c6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"
Jul 28 17:43:01.106409 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.106387    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7pdd\" (UniqueName: \"kubernetes.io/projected/e2b74082-3091-44cb-a954-92c3b76111c6-kube-api-access-d7pdd\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-97z2j\" (UID: \"e2b74082-3091-44cb-a954-92c3b76111c6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"
Jul 28 17:43:01.106505 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.106419    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2b74082-3091-44cb-a954-92c3b76111c6-tls-certs\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-97z2j\" (UID: \"e2b74082-3091-44cb-a954-92c3b76111c6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"
Jul 28 17:43:01.152182 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.152128    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b7c1fd1-111d-48c2-b52f-92aaba1a7c14" path="/var/lib/kubelet/pods/2b7c1fd1-111d-48c2-b52f-92aaba1a7c14/volumes"
Jul 28 17:43:01.207100 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.207069    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2b74082-3091-44cb-a954-92c3b76111c6-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-97z2j\" (UID: \"e2b74082-3091-44cb-a954-92c3b76111c6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"
Jul 28 17:43:01.207100 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.207103    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2b74082-3091-44cb-a954-92c3b76111c6-home\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-97z2j\" (UID: \"e2b74082-3091-44cb-a954-92c3b76111c6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"
Jul 28 17:43:01.207332 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.207250    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2b74082-3091-44cb-a954-92c3b76111c6-tmp-dir\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-97z2j\" (UID: \"e2b74082-3091-44cb-a954-92c3b76111c6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"
Jul 28 17:43:01.207414 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.207383    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2b74082-3091-44cb-a954-92c3b76111c6-model-cache\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-97z2j\" (UID: \"e2b74082-3091-44cb-a954-92c3b76111c6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"
Jul 28 17:43:01.207481 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.207451    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2b74082-3091-44cb-a954-92c3b76111c6-dshm\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-97z2j\" (UID: \"e2b74082-3091-44cb-a954-92c3b76111c6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"
Jul 28 17:43:01.207539 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.207481    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7pdd\" (UniqueName: \"kubernetes.io/projected/e2b74082-3091-44cb-a954-92c3b76111c6-kube-api-access-d7pdd\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-97z2j\" (UID: \"e2b74082-3091-44cb-a954-92c3b76111c6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"
Jul 28 17:43:01.207539 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.207520    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2b74082-3091-44cb-a954-92c3b76111c6-tls-certs\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-97z2j\" (UID: \"e2b74082-3091-44cb-a954-92c3b76111c6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"
Jul 28 17:43:01.207715 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.207692    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2b74082-3091-44cb-a954-92c3b76111c6-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-97z2j\" (UID: \"e2b74082-3091-44cb-a954-92c3b76111c6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"
Jul 28 17:43:01.207793 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.207771    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2b74082-3091-44cb-a954-92c3b76111c6-model-cache\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-97z2j\" (UID: \"e2b74082-3091-44cb-a954-92c3b76111c6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"
Jul 28 17:43:01.207861 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.207839    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2b74082-3091-44cb-a954-92c3b76111c6-tmp-dir\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-97z2j\" (UID: \"e2b74082-3091-44cb-a954-92c3b76111c6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"
Jul 28 17:43:01.207918 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.207888    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2b74082-3091-44cb-a954-92c3b76111c6-home\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-97z2j\" (UID: \"e2b74082-3091-44cb-a954-92c3b76111c6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"
Jul 28 17:43:01.209816 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.209794    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2b74082-3091-44cb-a954-92c3b76111c6-dshm\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-97z2j\" (UID: \"e2b74082-3091-44cb-a954-92c3b76111c6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"
Jul 28 17:43:01.210191 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.210173    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2b74082-3091-44cb-a954-92c3b76111c6-tls-certs\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-97z2j\" (UID: \"e2b74082-3091-44cb-a954-92c3b76111c6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"
Jul 28 17:43:01.217936 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.217914    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7pdd\" (UniqueName: \"kubernetes.io/projected/e2b74082-3091-44cb-a954-92c3b76111c6-kube-api-access-d7pdd\") pod \"auth-invalid-token-test-kserve-57cd4b8d86-97z2j\" (UID: \"e2b74082-3091-44cb-a954-92c3b76111c6\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"
Jul 28 17:43:01.367354 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.367272    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"
Jul 28 17:43:01.520754 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.520721    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"]
Jul 28 17:43:01.523932 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:43:01.523909    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2b74082_3091_44cb_a954_92c3b76111c6.slice/crio-2fc5f07d25fcf8b060f8c759b2a7808293389f9792226ccb98ef7719abefe867 WatchSource:0}: Error finding container 2fc5f07d25fcf8b060f8c759b2a7808293389f9792226ccb98ef7719abefe867: Status 404 returned error can't find the container with id 2fc5f07d25fcf8b060f8c759b2a7808293389f9792226ccb98ef7719abefe867
Jul 28 17:43:01.990658 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.990612    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j" event={"ID":"e2b74082-3091-44cb-a954-92c3b76111c6","Type":"ContainerStarted","Data":"a3d3403d014adc52da368377e2e1dc290cc59cec687e535410da6e65adadf371"}
Jul 28 17:43:01.990877 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:01.990724    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j" event={"ID":"e2b74082-3091-44cb-a954-92c3b76111c6","Type":"ContainerStarted","Data":"2fc5f07d25fcf8b060f8c759b2a7808293389f9792226ccb98ef7719abefe867"}
Jul 28 17:43:15.056746 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:15.056709    2543 generic.go:356] "Generic (PLEG): container finished" podID="e2b74082-3091-44cb-a954-92c3b76111c6" containerID="a3d3403d014adc52da368377e2e1dc290cc59cec687e535410da6e65adadf371" exitCode=0
Jul 28 17:43:15.057209 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:15.056781    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j" event={"ID":"e2b74082-3091-44cb-a954-92c3b76111c6","Type":"ContainerDied","Data":"a3d3403d014adc52da368377e2e1dc290cc59cec687e535410da6e65adadf371"}
Jul 28 17:43:16.064208 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:16.064174    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j" event={"ID":"e2b74082-3091-44cb-a954-92c3b76111c6","Type":"ContainerStarted","Data":"e0797a9b0bd7d4adf269d55fff1cbf6449c22352fe249f0d935c550f2d5e2830"}
Jul 28 17:43:16.091031 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:16.090980    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j" podStartSLOduration=2.175672931 podStartE2EDuration="15.09096455s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.915291619s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:43:16.087617483 +0000 UTC m=+1199.493891237" watchObservedRunningTime="2026-07-28 17:43:16.09096455 +0000 UTC m=+1199.497238292"
Jul 28 17:43:16.367717 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:16.367636    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"
Jul 28 17:43:21.368425 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:21.368379    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"
Jul 28 17:43:21.369902 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:21.369873    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j" podUID="e2b74082-3091-44cb-a954-92c3b76111c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 28 17:43:31.367790 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:31.367737    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j" podUID="e2b74082-3091-44cb-a954-92c3b76111c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 28 17:43:41.368513 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:41.368463    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j" podUID="e2b74082-3091-44cb-a954-92c3b76111c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 28 17:43:51.367893 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:43:51.367850    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j" podUID="e2b74082-3091-44cb-a954-92c3b76111c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 28 17:44:01.368386 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:44:01.368335    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j" podUID="e2b74082-3091-44cb-a954-92c3b76111c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 28 17:44:11.368780 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:44:11.368681    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j" podUID="e2b74082-3091-44cb-a954-92c3b76111c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 28 17:44:21.368350 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:44:21.368301    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j" podUID="e2b74082-3091-44cb-a954-92c3b76111c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 28 17:44:31.368474 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:44:31.368432    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j" podUID="e2b74082-3091-44cb-a954-92c3b76111c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 28 17:44:41.368263 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:44:41.368217    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j" podUID="e2b74082-3091-44cb-a954-92c3b76111c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 28 17:44:51.368658 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:44:51.368622    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j" podUID="e2b74082-3091-44cb-a954-92c3b76111c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 28 17:45:01.368757 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:01.368713    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j" podUID="e2b74082-3091-44cb-a954-92c3b76111c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 28 17:45:11.378151 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:11.378113    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"
Jul 28 17:45:11.385983 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:11.385960    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"
Jul 28 17:45:12.774865 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:12.774832    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"]
Jul 28 17:45:27.797643 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:27.797602    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j" podUID="e2b74082-3091-44cb-a954-92c3b76111c6" containerName="main" containerID="cri-o://e0797a9b0bd7d4adf269d55fff1cbf6449c22352fe249f0d935c550f2d5e2830" gracePeriod=45
Jul 28 17:45:28.379764 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.379721    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j" podUID="e2b74082-3091-44cb-a954-92c3b76111c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Jul 28 17:45:28.555458 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.555397    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"
Jul 28 17:45:28.620694 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.620664    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2b74082-3091-44cb-a954-92c3b76111c6-tls-certs\") pod \"e2b74082-3091-44cb-a954-92c3b76111c6\" (UID: \"e2b74082-3091-44cb-a954-92c3b76111c6\") "
Jul 28 17:45:28.620858 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.620718    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2b74082-3091-44cb-a954-92c3b76111c6-kserve-provision-location\") pod \"e2b74082-3091-44cb-a954-92c3b76111c6\" (UID: \"e2b74082-3091-44cb-a954-92c3b76111c6\") "
Jul 28 17:45:28.620858 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.620751    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2b74082-3091-44cb-a954-92c3b76111c6-home\") pod \"e2b74082-3091-44cb-a954-92c3b76111c6\" (UID: \"e2b74082-3091-44cb-a954-92c3b76111c6\") "
Jul 28 17:45:28.620858 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.620776    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2b74082-3091-44cb-a954-92c3b76111c6-dshm\") pod \"e2b74082-3091-44cb-a954-92c3b76111c6\" (UID: \"e2b74082-3091-44cb-a954-92c3b76111c6\") "
Jul 28 17:45:28.620858 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.620793    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2b74082-3091-44cb-a954-92c3b76111c6-model-cache\") pod \"e2b74082-3091-44cb-a954-92c3b76111c6\" (UID: \"e2b74082-3091-44cb-a954-92c3b76111c6\") "
Jul 28 17:45:28.620858 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.620816    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2b74082-3091-44cb-a954-92c3b76111c6-tmp-dir\") pod \"e2b74082-3091-44cb-a954-92c3b76111c6\" (UID: \"e2b74082-3091-44cb-a954-92c3b76111c6\") "
Jul 28 17:45:28.620858 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.620839    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7pdd\" (UniqueName: \"kubernetes.io/projected/e2b74082-3091-44cb-a954-92c3b76111c6-kube-api-access-d7pdd\") pod \"e2b74082-3091-44cb-a954-92c3b76111c6\" (UID: \"e2b74082-3091-44cb-a954-92c3b76111c6\") "
Jul 28 17:45:28.621220 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.621095    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b74082-3091-44cb-a954-92c3b76111c6-model-cache" (OuterVolumeSpecName: "model-cache") pod "e2b74082-3091-44cb-a954-92c3b76111c6" (UID: "e2b74082-3091-44cb-a954-92c3b76111c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:45:28.621397 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.621373    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b74082-3091-44cb-a954-92c3b76111c6-home" (OuterVolumeSpecName: "home") pod "e2b74082-3091-44cb-a954-92c3b76111c6" (UID: "e2b74082-3091-44cb-a954-92c3b76111c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:45:28.623621 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.623593    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2b74082-3091-44cb-a954-92c3b76111c6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e2b74082-3091-44cb-a954-92c3b76111c6" (UID: "e2b74082-3091-44cb-a954-92c3b76111c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:45:28.623727 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.623593    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b74082-3091-44cb-a954-92c3b76111c6-dshm" (OuterVolumeSpecName: "dshm") pod "e2b74082-3091-44cb-a954-92c3b76111c6" (UID: "e2b74082-3091-44cb-a954-92c3b76111c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:45:28.623727 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.623601    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b74082-3091-44cb-a954-92c3b76111c6-kube-api-access-d7pdd" (OuterVolumeSpecName: "kube-api-access-d7pdd") pod "e2b74082-3091-44cb-a954-92c3b76111c6" (UID: "e2b74082-3091-44cb-a954-92c3b76111c6"). InnerVolumeSpecName "kube-api-access-d7pdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:45:28.632601 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.632574    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b74082-3091-44cb-a954-92c3b76111c6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e2b74082-3091-44cb-a954-92c3b76111c6" (UID: "e2b74082-3091-44cb-a954-92c3b76111c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:45:28.686034 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.685992    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b74082-3091-44cb-a954-92c3b76111c6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e2b74082-3091-44cb-a954-92c3b76111c6" (UID: "e2b74082-3091-44cb-a954-92c3b76111c6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:45:28.722309 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.722280    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2b74082-3091-44cb-a954-92c3b76111c6-kserve-provision-location\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:45:28.722309 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.722308    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2b74082-3091-44cb-a954-92c3b76111c6-home\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:45:28.722465 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.722318    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2b74082-3091-44cb-a954-92c3b76111c6-dshm\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:45:28.722465 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.722326    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2b74082-3091-44cb-a954-92c3b76111c6-model-cache\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:45:28.722465 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.722335    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2b74082-3091-44cb-a954-92c3b76111c6-tmp-dir\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:45:28.722465 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.722343    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7pdd\" (UniqueName: \"kubernetes.io/projected/e2b74082-3091-44cb-a954-92c3b76111c6-kube-api-access-d7pdd\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:45:28.722465 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.722350    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2b74082-3091-44cb-a954-92c3b76111c6-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:45:28.764360 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.764317    2543 generic.go:356] "Generic (PLEG): container finished" podID="e2b74082-3091-44cb-a954-92c3b76111c6" containerID="e0797a9b0bd7d4adf269d55fff1cbf6449c22352fe249f0d935c550f2d5e2830" exitCode=0
Jul 28 17:45:28.764360 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.764360    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j" event={"ID":"e2b74082-3091-44cb-a954-92c3b76111c6","Type":"ContainerDied","Data":"e0797a9b0bd7d4adf269d55fff1cbf6449c22352fe249f0d935c550f2d5e2830"}
Jul 28 17:45:28.764570 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.764373    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"
Jul 28 17:45:28.764570 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.764406    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j" event={"ID":"e2b74082-3091-44cb-a954-92c3b76111c6","Type":"ContainerDied","Data":"2fc5f07d25fcf8b060f8c759b2a7808293389f9792226ccb98ef7719abefe867"}
Jul 28 17:45:28.764570 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.764428    2543 scope.go:117] "RemoveContainer" containerID="e0797a9b0bd7d4adf269d55fff1cbf6449c22352fe249f0d935c550f2d5e2830"
Jul 28 17:45:28.773713 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.773696    2543 scope.go:117] "RemoveContainer" containerID="a3d3403d014adc52da368377e2e1dc290cc59cec687e535410da6e65adadf371"
Jul 28 17:45:28.781849 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.781829    2543 scope.go:117] "RemoveContainer" containerID="e0797a9b0bd7d4adf269d55fff1cbf6449c22352fe249f0d935c550f2d5e2830"
Jul 28 17:45:28.782080 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:45:28.782061    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0797a9b0bd7d4adf269d55fff1cbf6449c22352fe249f0d935c550f2d5e2830\": container with ID starting with e0797a9b0bd7d4adf269d55fff1cbf6449c22352fe249f0d935c550f2d5e2830 not found: ID does not exist" containerID="e0797a9b0bd7d4adf269d55fff1cbf6449c22352fe249f0d935c550f2d5e2830"
Jul 28 17:45:28.782129 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.782089    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0797a9b0bd7d4adf269d55fff1cbf6449c22352fe249f0d935c550f2d5e2830"} err="failed to get container status \"e0797a9b0bd7d4adf269d55fff1cbf6449c22352fe249f0d935c550f2d5e2830\": rpc error: code = NotFound desc = could not find container \"e0797a9b0bd7d4adf269d55fff1cbf6449c22352fe249f0d935c550f2d5e2830\": container with ID starting with e0797a9b0bd7d4adf269d55fff1cbf6449c22352fe249f0d935c550f2d5e2830 not found: ID does not exist"
Jul 28 17:45:28.782129 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.782105    2543 scope.go:117] "RemoveContainer" containerID="a3d3403d014adc52da368377e2e1dc290cc59cec687e535410da6e65adadf371"
Jul 28 17:45:28.782318 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:45:28.782300    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3d3403d014adc52da368377e2e1dc290cc59cec687e535410da6e65adadf371\": container with ID starting with a3d3403d014adc52da368377e2e1dc290cc59cec687e535410da6e65adadf371 not found: ID does not exist" containerID="a3d3403d014adc52da368377e2e1dc290cc59cec687e535410da6e65adadf371"
Jul 28 17:45:28.782359 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.782321    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3d3403d014adc52da368377e2e1dc290cc59cec687e535410da6e65adadf371"} err="failed to get container status \"a3d3403d014adc52da368377e2e1dc290cc59cec687e535410da6e65adadf371\": rpc error: code = NotFound desc = could not find container \"a3d3403d014adc52da368377e2e1dc290cc59cec687e535410da6e65adadf371\": container with ID starting with a3d3403d014adc52da368377e2e1dc290cc59cec687e535410da6e65adadf371 not found: ID does not exist"
Jul 28 17:45:28.788310 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.788289    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"]
Jul 28 17:45:28.795433 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:28.795412    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-57cd4b8d86-97z2j"]
Jul 28 17:45:29.152760 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:29.152724    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2b74082-3091-44cb-a954-92c3b76111c6" path="/var/lib/kubelet/pods/e2b74082-3091-44cb-a954-92c3b76111c6/volumes"
Jul 28 17:45:53.815309 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:53.815229    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr"]
Jul 28 17:45:53.815678 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:53.815560    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2b74082-3091-44cb-a954-92c3b76111c6" containerName="main"
Jul 28 17:45:53.815678 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:53.815571    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2b74082-3091-44cb-a954-92c3b76111c6" containerName="main"
Jul 28 17:45:53.815678 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:53.815579    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2b74082-3091-44cb-a954-92c3b76111c6" containerName="storage-initializer"
Jul 28 17:45:53.815678 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:53.815584    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2b74082-3091-44cb-a954-92c3b76111c6" containerName="storage-initializer"
Jul 28 17:45:53.815678 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:53.815638    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2b74082-3091-44cb-a954-92c3b76111c6" containerName="main"
Jul 28 17:45:53.818283 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:53.818263    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr"
Jul 28 17:45:53.820642 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:53.820620    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Jul 28 17:45:53.821499 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:53.821476    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 28 17:45:53.821566 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:53.821518    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-rgglf\""
Jul 28 17:45:53.821566 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:53.821528    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Jul 28 17:45:53.832558 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:53.832535    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr"]
Jul 28 17:45:53.935310 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:53.935285    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca014c4-9e93-4c73-8f2e-86462143c657-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr\" (UID: \"4ca014c4-9e93-4c73-8f2e-86462143c657\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr"
Jul 28 17:45:53.935462 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:53.935326    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ca014c4-9e93-4c73-8f2e-86462143c657-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr\" (UID: \"4ca014c4-9e93-4c73-8f2e-86462143c657\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr"
Jul 28 17:45:53.935462 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:53.935412    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca014c4-9e93-4c73-8f2e-86462143c657-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr\" (UID: \"4ca014c4-9e93-4c73-8f2e-86462143c657\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr"
Jul 28 17:45:53.935539 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:53.935464    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca014c4-9e93-4c73-8f2e-86462143c657-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr\" (UID: \"4ca014c4-9e93-4c73-8f2e-86462143c657\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr"
Jul 28 17:45:53.935539 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:53.935488    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp72j\" (UniqueName: \"kubernetes.io/projected/4ca014c4-9e93-4c73-8f2e-86462143c657-kube-api-access-gp72j\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr\" (UID: \"4ca014c4-9e93-4c73-8f2e-86462143c657\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr"
Jul 28 17:45:53.935604 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:53.935538    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca014c4-9e93-4c73-8f2e-86462143c657-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr\" (UID: \"4ca014c4-9e93-4c73-8f2e-86462143c657\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr"
Jul 28 17:45:54.037012 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:54.036972    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca014c4-9e93-4c73-8f2e-86462143c657-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr\" (UID: \"4ca014c4-9e93-4c73-8f2e-86462143c657\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr"
Jul 28 17:45:54.037193 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:54.037045    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca014c4-9e93-4c73-8f2e-86462143c657-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr\" (UID: \"4ca014c4-9e93-4c73-8f2e-86462143c657\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr"
Jul 28 17:45:54.037193 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:54.037078    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp72j\" (UniqueName: \"kubernetes.io/projected/4ca014c4-9e93-4c73-8f2e-86462143c657-kube-api-access-gp72j\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr\" (UID: \"4ca014c4-9e93-4c73-8f2e-86462143c657\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr"
Jul 28 17:45:54.037193 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:54.037127    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca014c4-9e93-4c73-8f2e-86462143c657-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr\" (UID: \"4ca014c4-9e93-4c73-8f2e-86462143c657\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr"
Jul 28 17:45:54.037193 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:54.037184    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca014c4-9e93-4c73-8f2e-86462143c657-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr\" (UID: \"4ca014c4-9e93-4c73-8f2e-86462143c657\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr"
Jul 28 17:45:54.037419 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:54.037231    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ca014c4-9e93-4c73-8f2e-86462143c657-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr\" (UID: \"4ca014c4-9e93-4c73-8f2e-86462143c657\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr"
Jul 28 17:45:54.037482 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:54.037442    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca014c4-9e93-4c73-8f2e-86462143c657-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr\" (UID: \"4ca014c4-9e93-4c73-8f2e-86462143c657\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr"
Jul 28 17:45:54.037536 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:54.037486    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca014c4-9e93-4c73-8f2e-86462143c657-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr\" (UID: \"4ca014c4-9e93-4c73-8f2e-86462143c657\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr"
Jul 28 17:45:54.037589 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:54.037556    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ca014c4-9e93-4c73-8f2e-86462143c657-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr\" (UID: \"4ca014c4-9e93-4c73-8f2e-86462143c657\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr"
Jul 28 17:45:54.039572 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:54.039544    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca014c4-9e93-4c73-8f2e-86462143c657-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr\" (UID: \"4ca014c4-9e93-4c73-8f2e-86462143c657\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr"
Jul 28 17:45:54.040000 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:54.039980    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca014c4-9e93-4c73-8f2e-86462143c657-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr\" (UID: \"4ca014c4-9e93-4c73-8f2e-86462143c657\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr"
Jul 28 17:45:54.050448 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:54.050415    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp72j\" (UniqueName: \"kubernetes.io/projected/4ca014c4-9e93-4c73-8f2e-86462143c657-kube-api-access-gp72j\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr\" (UID: \"4ca014c4-9e93-4c73-8f2e-86462143c657\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr"
Jul 28 17:45:54.061623 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:54.061601    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587c2v2z"]
Jul 28 17:45:54.065148 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:54.065114    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587c2v2z"
Jul 28 17:45:54.067475 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:54.067428    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-jmlq4\""
Jul 28 17:45:54.074303 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:54.074274    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587c2v2z"]
Jul 28 17:45:54.128320 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:54.128294    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr"
Jul 28 17:45:54.240735 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:54.240699    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a274d61d-f0a6-4268-ac7a-b16330763df4-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587c2v2z\" (UID: \"a274d61d-f0a6-4268-ac7a-b16330763df4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587c2v2z"
Jul 28 17:45:54.240922 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:54.240749    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcmzf\" (UniqueName: \"kubernetes.io/projected/a274d61d-f0a6-4268-ac7a-b16330763df4-kube-api-access-vcmzf\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587c2v2z\" (UID: \"a274d61d-f0a6-4268-ac7a-b16330763df4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587c2v2z"
Jul 28 17:45:54.262826 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:54.262803    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr"]
Jul 28 17:45:54.264171 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:45:54.264128    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca014c4_9e93_4c73_8f2e_86462143c657.slice/crio-bc45928afdd60c94b86804f1905921f21fe19015070f132fcd925db015e462c5 WatchSource:0}: Error finding container bc45928afdd60c94b86804f1905921f21fe19015070f132fcd925db015e462c5: Status 404 returned error can't find the container with id bc45928afdd60c94b86804f1905921f21fe19015070f132fcd925db015e462c5
Jul 28 17:45:54.265965 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:54.265949    2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:45:54.341755 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:54.341665    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a274d61d-f0a6-4268-ac7a-b16330763df4-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587c2v2z\" (UID: \"a274d61d-f0a6-4268-ac7a-b16330763df4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587c2v2z"
Jul 28 17:45:54.341755 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:54.341706    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcmzf\" (UniqueName: \"kubernetes.io/projected/a274d61d-f0a6-4268-ac7a-b16330763df4-kube-api-access-vcmzf\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587c2v2z\" (UID: \"a274d61d-f0a6-4268-ac7a-b16330763df4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587c2v2z"
Jul 28 17:45:54.344205 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:54.344177    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a274d61d-f0a6-4268-ac7a-b16330763df4-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587c2v2z\" (UID: \"a274d61d-f0a6-4268-ac7a-b16330763df4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587c2v2z"
Jul 28 17:45:54.350250 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:54.350220    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcmzf\" (UniqueName: \"kubernetes.io/projected/a274d61d-f0a6-4268-ac7a-b16330763df4-kube-api-access-vcmzf\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-56587c2v2z\" (UID: \"a274d61d-f0a6-4268-ac7a-b16330763df4\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587c2v2z"
Jul 28 17:45:54.375779 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:54.375754    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587c2v2z"
Jul 28 17:45:54.519650 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:54.519616    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587c2v2z"]
Jul 28 17:45:54.521048 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:45:54.521022    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda274d61d_f0a6_4268_ac7a_b16330763df4.slice/crio-9a6c466f7d9c3d778b99691b6e815362648c2ee425d037d09d5a3d839e0da1e3 WatchSource:0}: Error finding container 9a6c466f7d9c3d778b99691b6e815362648c2ee425d037d09d5a3d839e0da1e3: Status 404 returned error can't find the container with id 9a6c466f7d9c3d778b99691b6e815362648c2ee425d037d09d5a3d839e0da1e3
Jul 28 17:45:54.898790 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:54.898739    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587c2v2z" event={"ID":"a274d61d-f0a6-4268-ac7a-b16330763df4","Type":"ContainerStarted","Data":"9a6c466f7d9c3d778b99691b6e815362648c2ee425d037d09d5a3d839e0da1e3"}
Jul 28 17:45:54.900699 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:54.900668    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr" event={"ID":"4ca014c4-9e93-4c73-8f2e-86462143c657","Type":"ContainerStarted","Data":"bc45928afdd60c94b86804f1905921f21fe19015070f132fcd925db015e462c5"}
Jul 28 17:45:56.914807 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:56.914766    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr" event={"ID":"4ca014c4-9e93-4c73-8f2e-86462143c657","Type":"ContainerStarted","Data":"3ef07afa9a025289612fa705922ae0c6d5ba3bac69e634aefd583810c2eef0fe"}
Jul 28 17:45:56.934861 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:56.934808    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr" podStartSLOduration=2.220404082 podStartE2EDuration="3.934793289s" totalImagesPullingTime="1.714389207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:45:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:45:56.932812965 +0000 UTC m=+1360.339086707" watchObservedRunningTime="2026-07-28 17:45:56.934793289 +0000 UTC m=+1360.341067029"
Jul 28 17:45:57.129158 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:57.129100    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr"
Jul 28 17:45:57.921346 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:57.921310    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587c2v2z" event={"ID":"a274d61d-f0a6-4268-ac7a-b16330763df4","Type":"ContainerStarted","Data":"2917526ce0a027fefa03902b0f149883e99ec371a3b0ed257d296a548f0a6fa4"}
Jul 28 17:45:57.921786 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:57.921519    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587c2v2z"
Jul 28 17:45:57.963794 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:45:57.963736    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587c2v2z" podStartSLOduration=1.351124697 podStartE2EDuration="3.96371775s" totalImagesPullingTime="2.612593053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:45:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:45:57.962632777 +0000 UTC m=+1361.368906522" watchObservedRunningTime="2026-07-28 17:45:57.96371775 +0000 UTC m=+1361.369991492"
Jul 28 17:46:04.128726 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:04.128688    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr"
Jul 28 17:46:04.141428 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:04.141404    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr"
Jul 28 17:46:04.966409 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:04.966382    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr"
Jul 28 17:46:28.927641 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:28.927608    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587c2v2z"
Jul 28 17:46:30.030263 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:30.030227    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587c2v2z"]
Jul 28 17:46:30.042236 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:30.042173    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr"]
Jul 28 17:46:30.054052 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:46:30.054011    2543 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:46:30.054052 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:46:30.054052 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:46:30.054052 ip-10-0-135-177 kubenswrapper[2543]:  > containerID="3ef07afa9a025289612fa705922ae0c6d5ba3bac69e634aefd583810c2eef0fe" cmd=["/bin/sleep","15"]
Jul 28 17:46:30.054398 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:30.054078    2543 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr" message="" err=<
Jul 28 17:46:30.054398 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:46:30.054398 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:46:30.054398 ip-10-0-135-177 kubenswrapper[2543]:  >
Jul 28 17:46:30.054398 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:46:30.054097    2543 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:46:30.054398 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:46:30.054398 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:46:30.054398 ip-10-0-135-177 kubenswrapper[2543]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr" podUID="4ca014c4-9e93-4c73-8f2e-86462143c657" containerName="main" containerID="cri-o://3ef07afa9a025289612fa705922ae0c6d5ba3bac69e634aefd583810c2eef0fe"
Jul 28 17:46:30.054398 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:30.054168    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr" podUID="4ca014c4-9e93-4c73-8f2e-86462143c657" containerName="main" containerID="cri-o://3ef07afa9a025289612fa705922ae0c6d5ba3bac69e634aefd583810c2eef0fe" gracePeriod=60
Jul 28 17:46:30.416277 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:30.416250    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr"
Jul 28 17:46:30.546391 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:30.546355    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca014c4-9e93-4c73-8f2e-86462143c657-model-cache\") pod \"4ca014c4-9e93-4c73-8f2e-86462143c657\" (UID: \"4ca014c4-9e93-4c73-8f2e-86462143c657\") "
Jul 28 17:46:30.546391 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:30.546392    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ca014c4-9e93-4c73-8f2e-86462143c657-home\") pod \"4ca014c4-9e93-4c73-8f2e-86462143c657\" (UID: \"4ca014c4-9e93-4c73-8f2e-86462143c657\") "
Jul 28 17:46:30.546656 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:30.546459    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gp72j\" (UniqueName: \"kubernetes.io/projected/4ca014c4-9e93-4c73-8f2e-86462143c657-kube-api-access-gp72j\") pod \"4ca014c4-9e93-4c73-8f2e-86462143c657\" (UID: \"4ca014c4-9e93-4c73-8f2e-86462143c657\") "
Jul 28 17:46:30.546656 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:30.546478    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca014c4-9e93-4c73-8f2e-86462143c657-tmp-dir\") pod \"4ca014c4-9e93-4c73-8f2e-86462143c657\" (UID: \"4ca014c4-9e93-4c73-8f2e-86462143c657\") "
Jul 28 17:46:30.546656 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:30.546518    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca014c4-9e93-4c73-8f2e-86462143c657-tls-certs\") pod \"4ca014c4-9e93-4c73-8f2e-86462143c657\" (UID: \"4ca014c4-9e93-4c73-8f2e-86462143c657\") "
Jul 28 17:46:30.546656 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:30.546545    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca014c4-9e93-4c73-8f2e-86462143c657-dshm\") pod \"4ca014c4-9e93-4c73-8f2e-86462143c657\" (UID: \"4ca014c4-9e93-4c73-8f2e-86462143c657\") "
Jul 28 17:46:30.546870 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:30.546651    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca014c4-9e93-4c73-8f2e-86462143c657-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ca014c4-9e93-4c73-8f2e-86462143c657" (UID: "4ca014c4-9e93-4c73-8f2e-86462143c657"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:46:30.546870 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:30.546682    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca014c4-9e93-4c73-8f2e-86462143c657-home" (OuterVolumeSpecName: "home") pod "4ca014c4-9e93-4c73-8f2e-86462143c657" (UID: "4ca014c4-9e93-4c73-8f2e-86462143c657"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:46:30.546870 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:30.546820    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca014c4-9e93-4c73-8f2e-86462143c657-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ca014c4-9e93-4c73-8f2e-86462143c657" (UID: "4ca014c4-9e93-4c73-8f2e-86462143c657"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:46:30.546870 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:30.546829    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ca014c4-9e93-4c73-8f2e-86462143c657-model-cache\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:46:30.547013 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:30.546879    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ca014c4-9e93-4c73-8f2e-86462143c657-home\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:46:30.548845 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:30.548821    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ca014c4-9e93-4c73-8f2e-86462143c657-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ca014c4-9e93-4c73-8f2e-86462143c657" (UID: "4ca014c4-9e93-4c73-8f2e-86462143c657"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:46:30.548905 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:30.548823    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca014c4-9e93-4c73-8f2e-86462143c657-dshm" (OuterVolumeSpecName: "dshm") pod "4ca014c4-9e93-4c73-8f2e-86462143c657" (UID: "4ca014c4-9e93-4c73-8f2e-86462143c657"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:46:30.548905 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:30.548851    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca014c4-9e93-4c73-8f2e-86462143c657-kube-api-access-gp72j" (OuterVolumeSpecName: "kube-api-access-gp72j") pod "4ca014c4-9e93-4c73-8f2e-86462143c657" (UID: "4ca014c4-9e93-4c73-8f2e-86462143c657"). InnerVolumeSpecName "kube-api-access-gp72j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:46:30.648088 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:30.648005    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp72j\" (UniqueName: \"kubernetes.io/projected/4ca014c4-9e93-4c73-8f2e-86462143c657-kube-api-access-gp72j\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:46:30.648088 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:30.648033    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ca014c4-9e93-4c73-8f2e-86462143c657-tmp-dir\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:46:30.648088 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:30.648044    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ca014c4-9e93-4c73-8f2e-86462143c657-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:46:30.648088 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:30.648051    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ca014c4-9e93-4c73-8f2e-86462143c657-dshm\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:46:31.090459 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:31.090423    2543 generic.go:356] "Generic (PLEG): container finished" podID="4ca014c4-9e93-4c73-8f2e-86462143c657" containerID="3ef07afa9a025289612fa705922ae0c6d5ba3bac69e634aefd583810c2eef0fe" exitCode=0
Jul 28 17:46:31.090894 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:31.090470    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr"
Jul 28 17:46:31.090894 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:31.090503    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr" event={"ID":"4ca014c4-9e93-4c73-8f2e-86462143c657","Type":"ContainerDied","Data":"3ef07afa9a025289612fa705922ae0c6d5ba3bac69e634aefd583810c2eef0fe"}
Jul 28 17:46:31.090894 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:31.090554    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr" event={"ID":"4ca014c4-9e93-4c73-8f2e-86462143c657","Type":"ContainerDied","Data":"bc45928afdd60c94b86804f1905921f21fe19015070f132fcd925db015e462c5"}
Jul 28 17:46:31.090894 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:31.090577    2543 scope.go:117] "RemoveContainer" containerID="3ef07afa9a025289612fa705922ae0c6d5ba3bac69e634aefd583810c2eef0fe"
Jul 28 17:46:31.100297 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:31.100281    2543 scope.go:117] "RemoveContainer" containerID="3ef07afa9a025289612fa705922ae0c6d5ba3bac69e634aefd583810c2eef0fe"
Jul 28 17:46:31.100542 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:46:31.100523    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ef07afa9a025289612fa705922ae0c6d5ba3bac69e634aefd583810c2eef0fe\": container with ID starting with 3ef07afa9a025289612fa705922ae0c6d5ba3bac69e634aefd583810c2eef0fe not found: ID does not exist" containerID="3ef07afa9a025289612fa705922ae0c6d5ba3bac69e634aefd583810c2eef0fe"
Jul 28 17:46:31.100605 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:31.100550    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ef07afa9a025289612fa705922ae0c6d5ba3bac69e634aefd583810c2eef0fe"} err="failed to get container status \"3ef07afa9a025289612fa705922ae0c6d5ba3bac69e634aefd583810c2eef0fe\": rpc error: code = NotFound desc = could not find container \"3ef07afa9a025289612fa705922ae0c6d5ba3bac69e634aefd583810c2eef0fe\": container with ID starting with 3ef07afa9a025289612fa705922ae0c6d5ba3bac69e634aefd583810c2eef0fe not found: ID does not exist"
Jul 28 17:46:31.112885 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:31.112857    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr"]
Jul 28 17:46:31.118838 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:31.118817    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6htvr"]
Jul 28 17:46:31.152776 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:31.152743    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ca014c4-9e93-4c73-8f2e-86462143c657" path="/var/lib/kubelet/pods/4ca014c4-9e93-4c73-8f2e-86462143c657/volumes"
Jul 28 17:46:38.928025 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:38.927990    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587c2v2z" podUID="a274d61d-f0a6-4268-ac7a-b16330763df4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:46:45.056256 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:45.056214    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587c2v2z" podUID="a274d61d-f0a6-4268-ac7a-b16330763df4" containerName="main" containerID="cri-o://2917526ce0a027fefa03902b0f149883e99ec371a3b0ed257d296a548f0a6fa4" gracePeriod=45
Jul 28 17:46:45.305255 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:45.305231    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587c2v2z"
Jul 28 17:46:45.476102 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:45.476021    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcmzf\" (UniqueName: \"kubernetes.io/projected/a274d61d-f0a6-4268-ac7a-b16330763df4-kube-api-access-vcmzf\") pod \"a274d61d-f0a6-4268-ac7a-b16330763df4\" (UID: \"a274d61d-f0a6-4268-ac7a-b16330763df4\") "
Jul 28 17:46:45.476284 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:45.476172    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a274d61d-f0a6-4268-ac7a-b16330763df4-tls-certs\") pod \"a274d61d-f0a6-4268-ac7a-b16330763df4\" (UID: \"a274d61d-f0a6-4268-ac7a-b16330763df4\") "
Jul 28 17:46:45.477949 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:45.477916    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a274d61d-f0a6-4268-ac7a-b16330763df4-kube-api-access-vcmzf" (OuterVolumeSpecName: "kube-api-access-vcmzf") pod "a274d61d-f0a6-4268-ac7a-b16330763df4" (UID: "a274d61d-f0a6-4268-ac7a-b16330763df4"). InnerVolumeSpecName "kube-api-access-vcmzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:46:45.478076 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:45.477993    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a274d61d-f0a6-4268-ac7a-b16330763df4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a274d61d-f0a6-4268-ac7a-b16330763df4" (UID: "a274d61d-f0a6-4268-ac7a-b16330763df4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:46:45.577557 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:45.577514    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a274d61d-f0a6-4268-ac7a-b16330763df4-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:46:45.577557 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:45.577553    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcmzf\" (UniqueName: \"kubernetes.io/projected/a274d61d-f0a6-4268-ac7a-b16330763df4-kube-api-access-vcmzf\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:46:46.167334 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:46.167296    2543 generic.go:356] "Generic (PLEG): container finished" podID="a274d61d-f0a6-4268-ac7a-b16330763df4" containerID="2917526ce0a027fefa03902b0f149883e99ec371a3b0ed257d296a548f0a6fa4" exitCode=0
Jul 28 17:46:46.167766 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:46.167353    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587c2v2z"
Jul 28 17:46:46.167766 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:46.167378    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587c2v2z" event={"ID":"a274d61d-f0a6-4268-ac7a-b16330763df4","Type":"ContainerDied","Data":"2917526ce0a027fefa03902b0f149883e99ec371a3b0ed257d296a548f0a6fa4"}
Jul 28 17:46:46.167766 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:46.167434    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587c2v2z" event={"ID":"a274d61d-f0a6-4268-ac7a-b16330763df4","Type":"ContainerDied","Data":"9a6c466f7d9c3d778b99691b6e815362648c2ee425d037d09d5a3d839e0da1e3"}
Jul 28 17:46:46.167766 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:46.167456    2543 scope.go:117] "RemoveContainer" containerID="2917526ce0a027fefa03902b0f149883e99ec371a3b0ed257d296a548f0a6fa4"
Jul 28 17:46:46.176878 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:46.176862    2543 scope.go:117] "RemoveContainer" containerID="2917526ce0a027fefa03902b0f149883e99ec371a3b0ed257d296a548f0a6fa4"
Jul 28 17:46:46.177128 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:46:46.177108    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2917526ce0a027fefa03902b0f149883e99ec371a3b0ed257d296a548f0a6fa4\": container with ID starting with 2917526ce0a027fefa03902b0f149883e99ec371a3b0ed257d296a548f0a6fa4 not found: ID does not exist" containerID="2917526ce0a027fefa03902b0f149883e99ec371a3b0ed257d296a548f0a6fa4"
Jul 28 17:46:46.177209 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:46.177155    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2917526ce0a027fefa03902b0f149883e99ec371a3b0ed257d296a548f0a6fa4"} err="failed to get container status \"2917526ce0a027fefa03902b0f149883e99ec371a3b0ed257d296a548f0a6fa4\": rpc error: code = NotFound desc = could not find container \"2917526ce0a027fefa03902b0f149883e99ec371a3b0ed257d296a548f0a6fa4\": container with ID starting with 2917526ce0a027fefa03902b0f149883e99ec371a3b0ed257d296a548f0a6fa4 not found: ID does not exist"
Jul 28 17:46:46.192783 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:46.192758    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587c2v2z"]
Jul 28 17:46:46.203733 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:46.203712    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-56587c2v2z"]
Jul 28 17:46:47.152193 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:47.152160    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a274d61d-f0a6-4268-ac7a-b16330763df4" path="/var/lib/kubelet/pods/a274d61d-f0a6-4268-ac7a-b16330763df4/volumes"
Jul 28 17:46:58.011745 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.011711    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt"]
Jul 28 17:46:58.012111 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.012026    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ca014c4-9e93-4c73-8f2e-86462143c657" containerName="main"
Jul 28 17:46:58.012111 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.012036    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ca014c4-9e93-4c73-8f2e-86462143c657" containerName="main"
Jul 28 17:46:58.012111 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.012063    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a274d61d-f0a6-4268-ac7a-b16330763df4" containerName="main"
Jul 28 17:46:58.012111 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.012069    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="a274d61d-f0a6-4268-ac7a-b16330763df4" containerName="main"
Jul 28 17:46:58.012280 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.012120    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="a274d61d-f0a6-4268-ac7a-b16330763df4" containerName="main"
Jul 28 17:46:58.012280 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.012128    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ca014c4-9e93-4c73-8f2e-86462143c657" containerName="main"
Jul 28 17:46:58.017084 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.016633    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt"
Jul 28 17:46:58.020714 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.020689    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-sfpqc\""
Jul 28 17:46:58.020978 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.020947    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 28 17:46:58.021086 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.020997    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Jul 28 17:46:58.021086 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.021001    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Jul 28 17:46:58.027199 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.027173    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt"]
Jul 28 17:46:58.078475 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.078444    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/558de6b2-e431-4591-b379-b94fe38e91e0-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt\" (UID: \"558de6b2-e431-4591-b379-b94fe38e91e0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt"
Jul 28 17:46:58.078631 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.078489    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/558de6b2-e431-4591-b379-b94fe38e91e0-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt\" (UID: \"558de6b2-e431-4591-b379-b94fe38e91e0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt"
Jul 28 17:46:58.078631 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.078538    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/558de6b2-e431-4591-b379-b94fe38e91e0-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt\" (UID: \"558de6b2-e431-4591-b379-b94fe38e91e0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt"
Jul 28 17:46:58.078631 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.078622    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/558de6b2-e431-4591-b379-b94fe38e91e0-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt\" (UID: \"558de6b2-e431-4591-b379-b94fe38e91e0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt"
Jul 28 17:46:58.078746 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.078641    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhs6d\" (UniqueName: \"kubernetes.io/projected/558de6b2-e431-4591-b379-b94fe38e91e0-kube-api-access-dhs6d\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt\" (UID: \"558de6b2-e431-4591-b379-b94fe38e91e0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt"
Jul 28 17:46:58.078746 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.078674    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/558de6b2-e431-4591-b379-b94fe38e91e0-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt\" (UID: \"558de6b2-e431-4591-b379-b94fe38e91e0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt"
Jul 28 17:46:58.179999 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.179964    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/558de6b2-e431-4591-b379-b94fe38e91e0-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt\" (UID: \"558de6b2-e431-4591-b379-b94fe38e91e0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt"
Jul 28 17:46:58.180183 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.180009    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/558de6b2-e431-4591-b379-b94fe38e91e0-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt\" (UID: \"558de6b2-e431-4591-b379-b94fe38e91e0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt"
Jul 28 17:46:58.180183 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.180047    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/558de6b2-e431-4591-b379-b94fe38e91e0-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt\" (UID: \"558de6b2-e431-4591-b379-b94fe38e91e0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt"
Jul 28 17:46:58.180183 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.180062    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhs6d\" (UniqueName: \"kubernetes.io/projected/558de6b2-e431-4591-b379-b94fe38e91e0-kube-api-access-dhs6d\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt\" (UID: \"558de6b2-e431-4591-b379-b94fe38e91e0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt"
Jul 28 17:46:58.180183 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.180088    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/558de6b2-e431-4591-b379-b94fe38e91e0-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt\" (UID: \"558de6b2-e431-4591-b379-b94fe38e91e0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt"
Jul 28 17:46:58.180183 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.180116    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/558de6b2-e431-4591-b379-b94fe38e91e0-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt\" (UID: \"558de6b2-e431-4591-b379-b94fe38e91e0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt"
Jul 28 17:46:58.180513 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.180491    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/558de6b2-e431-4591-b379-b94fe38e91e0-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt\" (UID: \"558de6b2-e431-4591-b379-b94fe38e91e0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt"
Jul 28 17:46:58.180580 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.180523    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/558de6b2-e431-4591-b379-b94fe38e91e0-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt\" (UID: \"558de6b2-e431-4591-b379-b94fe38e91e0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt"
Jul 28 17:46:58.180580 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.180556    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/558de6b2-e431-4591-b379-b94fe38e91e0-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt\" (UID: \"558de6b2-e431-4591-b379-b94fe38e91e0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt"
Jul 28 17:46:58.182415 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.182391    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/558de6b2-e431-4591-b379-b94fe38e91e0-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt\" (UID: \"558de6b2-e431-4591-b379-b94fe38e91e0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt"
Jul 28 17:46:58.182526 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.182461    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/558de6b2-e431-4591-b379-b94fe38e91e0-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt\" (UID: \"558de6b2-e431-4591-b379-b94fe38e91e0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt"
Jul 28 17:46:58.192449 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.192428    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhs6d\" (UniqueName: \"kubernetes.io/projected/558de6b2-e431-4591-b379-b94fe38e91e0-kube-api-access-dhs6d\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt\" (UID: \"558de6b2-e431-4591-b379-b94fe38e91e0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt"
Jul 28 17:46:58.285460 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.285380    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw"]
Jul 28 17:46:58.288930 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.288914    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw"
Jul 28 17:46:58.291890 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.291866    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-lljr9\""
Jul 28 17:46:58.301549 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.301526    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw"]
Jul 28 17:46:58.329185 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.329157    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt"
Jul 28 17:46:58.382797 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.382717    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tfjm\" (UniqueName: \"kubernetes.io/projected/1804a4bd-b407-4e35-b464-bea77a07b584-kube-api-access-5tfjm\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw\" (UID: \"1804a4bd-b407-4e35-b464-bea77a07b584\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw"
Jul 28 17:46:58.382797 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.382790    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1804a4bd-b407-4e35-b464-bea77a07b584-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw\" (UID: \"1804a4bd-b407-4e35-b464-bea77a07b584\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw"
Jul 28 17:46:58.467349 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.467325    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt"]
Jul 28 17:46:58.468531 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:46:58.468500    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558de6b2_e431_4591_b379_b94fe38e91e0.slice/crio-92ddea883df443ef165420d8cf0b07ab64ebffdcdec948b3212dfd1a51204d4c WatchSource:0}: Error finding container 92ddea883df443ef165420d8cf0b07ab64ebffdcdec948b3212dfd1a51204d4c: Status 404 returned error can't find the container with id 92ddea883df443ef165420d8cf0b07ab64ebffdcdec948b3212dfd1a51204d4c
Jul 28 17:46:58.483548 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.483523    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tfjm\" (UniqueName: \"kubernetes.io/projected/1804a4bd-b407-4e35-b464-bea77a07b584-kube-api-access-5tfjm\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw\" (UID: \"1804a4bd-b407-4e35-b464-bea77a07b584\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw"
Jul 28 17:46:58.483729 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.483573    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1804a4bd-b407-4e35-b464-bea77a07b584-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw\" (UID: \"1804a4bd-b407-4e35-b464-bea77a07b584\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw"
Jul 28 17:46:58.486081 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.486054    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1804a4bd-b407-4e35-b464-bea77a07b584-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw\" (UID: \"1804a4bd-b407-4e35-b464-bea77a07b584\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw"
Jul 28 17:46:58.492195 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.492169    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tfjm\" (UniqueName: \"kubernetes.io/projected/1804a4bd-b407-4e35-b464-bea77a07b584-kube-api-access-5tfjm\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw\" (UID: \"1804a4bd-b407-4e35-b464-bea77a07b584\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw"
Jul 28 17:46:58.599928 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.599895    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw"
Jul 28 17:46:58.731163 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:58.731113    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw"]
Jul 28 17:46:58.733211 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:46:58.733181    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1804a4bd_b407_4e35_b464_bea77a07b584.slice/crio-b5d2cf20567f21087c03dad651499f11d8f980a9090ec903bac469281aaddc7e WatchSource:0}: Error finding container b5d2cf20567f21087c03dad651499f11d8f980a9090ec903bac469281aaddc7e: Status 404 returned error can't find the container with id b5d2cf20567f21087c03dad651499f11d8f980a9090ec903bac469281aaddc7e
Jul 28 17:46:59.233764 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:59.233666    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw" event={"ID":"1804a4bd-b407-4e35-b464-bea77a07b584","Type":"ContainerStarted","Data":"84edd2154ea9282d87f69adea9eae1f5b2bbdde20f7ae6b654fcd74fdb78a858"}
Jul 28 17:46:59.233764 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:59.233732    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw" event={"ID":"1804a4bd-b407-4e35-b464-bea77a07b584","Type":"ContainerStarted","Data":"b5d2cf20567f21087c03dad651499f11d8f980a9090ec903bac469281aaddc7e"}
Jul 28 17:46:59.234347 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:59.233867    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw"
Jul 28 17:46:59.235564 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:59.235541    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt" event={"ID":"558de6b2-e431-4591-b379-b94fe38e91e0","Type":"ContainerStarted","Data":"fa96430090de0ca84818d730c478785d635d31f81141edf96f640c1b93cff6f9"}
Jul 28 17:46:59.235671 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:59.235585    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt" event={"ID":"558de6b2-e431-4591-b379-b94fe38e91e0","Type":"ContainerStarted","Data":"92ddea883df443ef165420d8cf0b07ab64ebffdcdec948b3212dfd1a51204d4c"}
Jul 28 17:46:59.258028 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:59.257982    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw" podStartSLOduration=1.257966826 podStartE2EDuration="1.257966826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:46:59.256587437 +0000 UTC m=+1422.662861179" watchObservedRunningTime="2026-07-28 17:46:59.257966826 +0000 UTC m=+1422.664240561"
Jul 28 17:46:59.283734 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:59.283687    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt" podStartSLOduration=2.2836731009999998 podStartE2EDuration="2.283673101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:46:59.282504051 +0000 UTC m=+1422.688777793" watchObservedRunningTime="2026-07-28 17:46:59.283673101 +0000 UTC m=+1422.689946841"
Jul 28 17:46:59.329243 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:46:59.329213    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt"
Jul 28 17:47:08.329987 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:08.329956    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt"
Jul 28 17:47:08.343415 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:08.343385    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt"
Jul 28 17:47:09.299228 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:09.299202    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt"
Jul 28 17:47:30.243212 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:30.243179    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw"
Jul 28 17:47:31.831185 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:31.831153    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt"]
Jul 28 17:47:31.839322 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:31.839294    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw"]
Jul 28 17:47:31.841307 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:47:31.841255    2543 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:47:31.841307 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:47:31.841307 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:47:31.841307 ip-10-0-135-177 kubenswrapper[2543]:  > containerID="fa96430090de0ca84818d730c478785d635d31f81141edf96f640c1b93cff6f9" cmd=["/bin/sleep","15"]
Jul 28 17:47:31.841579 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:31.841325    2543 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt" message="" err=<
Jul 28 17:47:31.841579 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:47:31.841579 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:47:31.841579 ip-10-0-135-177 kubenswrapper[2543]:  >
Jul 28 17:47:31.841579 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:47:31.841341    2543 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:47:31.841579 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:47:31.841579 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:47:31.841579 ip-10-0-135-177 kubenswrapper[2543]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt" podUID="558de6b2-e431-4591-b379-b94fe38e91e0" containerName="main" containerID="cri-o://fa96430090de0ca84818d730c478785d635d31f81141edf96f640c1b93cff6f9"
Jul 28 17:47:31.841579 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:31.841392    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt" podUID="558de6b2-e431-4591-b379-b94fe38e91e0" containerName="main" containerID="cri-o://fa96430090de0ca84818d730c478785d635d31f81141edf96f640c1b93cff6f9" gracePeriod=60
Jul 28 17:47:32.193824 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:32.193799    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt"
Jul 28 17:47:32.274531 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:32.274506    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/558de6b2-e431-4591-b379-b94fe38e91e0-model-cache\") pod \"558de6b2-e431-4591-b379-b94fe38e91e0\" (UID: \"558de6b2-e431-4591-b379-b94fe38e91e0\") "
Jul 28 17:47:32.274681 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:32.274554    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/558de6b2-e431-4591-b379-b94fe38e91e0-tls-certs\") pod \"558de6b2-e431-4591-b379-b94fe38e91e0\" (UID: \"558de6b2-e431-4591-b379-b94fe38e91e0\") "
Jul 28 17:47:32.274681 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:32.274581    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/558de6b2-e431-4591-b379-b94fe38e91e0-dshm\") pod \"558de6b2-e431-4591-b379-b94fe38e91e0\" (UID: \"558de6b2-e431-4591-b379-b94fe38e91e0\") "
Jul 28 17:47:32.274681 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:32.274626    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhs6d\" (UniqueName: \"kubernetes.io/projected/558de6b2-e431-4591-b379-b94fe38e91e0-kube-api-access-dhs6d\") pod \"558de6b2-e431-4591-b379-b94fe38e91e0\" (UID: \"558de6b2-e431-4591-b379-b94fe38e91e0\") "
Jul 28 17:47:32.274681 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:32.274663    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/558de6b2-e431-4591-b379-b94fe38e91e0-home\") pod \"558de6b2-e431-4591-b379-b94fe38e91e0\" (UID: \"558de6b2-e431-4591-b379-b94fe38e91e0\") "
Jul 28 17:47:32.274885 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:32.274716    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/558de6b2-e431-4591-b379-b94fe38e91e0-tmp-dir\") pod \"558de6b2-e431-4591-b379-b94fe38e91e0\" (UID: \"558de6b2-e431-4591-b379-b94fe38e91e0\") "
Jul 28 17:47:32.274885 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:32.274787    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558de6b2-e431-4591-b379-b94fe38e91e0-model-cache" (OuterVolumeSpecName: "model-cache") pod "558de6b2-e431-4591-b379-b94fe38e91e0" (UID: "558de6b2-e431-4591-b379-b94fe38e91e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:47:32.275029 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:32.275009    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558de6b2-e431-4591-b379-b94fe38e91e0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "558de6b2-e431-4591-b379-b94fe38e91e0" (UID: "558de6b2-e431-4591-b379-b94fe38e91e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:47:32.275082 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:32.275027    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558de6b2-e431-4591-b379-b94fe38e91e0-home" (OuterVolumeSpecName: "home") pod "558de6b2-e431-4591-b379-b94fe38e91e0" (UID: "558de6b2-e431-4591-b379-b94fe38e91e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:47:32.275082 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:32.275070    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/558de6b2-e431-4591-b379-b94fe38e91e0-model-cache\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:47:32.276926 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:32.276906    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/558de6b2-e431-4591-b379-b94fe38e91e0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "558de6b2-e431-4591-b379-b94fe38e91e0" (UID: "558de6b2-e431-4591-b379-b94fe38e91e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:47:32.277076 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:32.277060    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/558de6b2-e431-4591-b379-b94fe38e91e0-dshm" (OuterVolumeSpecName: "dshm") pod "558de6b2-e431-4591-b379-b94fe38e91e0" (UID: "558de6b2-e431-4591-b379-b94fe38e91e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:47:32.277076 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:32.277065    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558de6b2-e431-4591-b379-b94fe38e91e0-kube-api-access-dhs6d" (OuterVolumeSpecName: "kube-api-access-dhs6d") pod "558de6b2-e431-4591-b379-b94fe38e91e0" (UID: "558de6b2-e431-4591-b379-b94fe38e91e0"). InnerVolumeSpecName "kube-api-access-dhs6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:47:32.376455 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:32.376382    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/558de6b2-e431-4591-b379-b94fe38e91e0-tmp-dir\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:47:32.376455 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:32.376408    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/558de6b2-e431-4591-b379-b94fe38e91e0-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:47:32.376455 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:32.376420    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/558de6b2-e431-4591-b379-b94fe38e91e0-dshm\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:47:32.376455 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:32.376431    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhs6d\" (UniqueName: \"kubernetes.io/projected/558de6b2-e431-4591-b379-b94fe38e91e0-kube-api-access-dhs6d\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:47:32.376455 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:32.376443    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/558de6b2-e431-4591-b379-b94fe38e91e0-home\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:47:32.408451 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:32.408420    2543 generic.go:356] "Generic (PLEG): container finished" podID="558de6b2-e431-4591-b379-b94fe38e91e0" containerID="fa96430090de0ca84818d730c478785d635d31f81141edf96f640c1b93cff6f9" exitCode=0
Jul 28 17:47:32.408582 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:32.408464    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt"
Jul 28 17:47:32.408582 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:32.408510    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt" event={"ID":"558de6b2-e431-4591-b379-b94fe38e91e0","Type":"ContainerDied","Data":"fa96430090de0ca84818d730c478785d635d31f81141edf96f640c1b93cff6f9"}
Jul 28 17:47:32.408582 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:32.408563    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt" event={"ID":"558de6b2-e431-4591-b379-b94fe38e91e0","Type":"ContainerDied","Data":"92ddea883df443ef165420d8cf0b07ab64ebffdcdec948b3212dfd1a51204d4c"}
Jul 28 17:47:32.408731 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:32.408586    2543 scope.go:117] "RemoveContainer" containerID="fa96430090de0ca84818d730c478785d635d31f81141edf96f640c1b93cff6f9"
Jul 28 17:47:32.417867 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:32.417823    2543 scope.go:117] "RemoveContainer" containerID="fa96430090de0ca84818d730c478785d635d31f81141edf96f640c1b93cff6f9"
Jul 28 17:47:32.418090 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:47:32.418071    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa96430090de0ca84818d730c478785d635d31f81141edf96f640c1b93cff6f9\": container with ID starting with fa96430090de0ca84818d730c478785d635d31f81141edf96f640c1b93cff6f9 not found: ID does not exist" containerID="fa96430090de0ca84818d730c478785d635d31f81141edf96f640c1b93cff6f9"
Jul 28 17:47:32.418275 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:32.418100    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa96430090de0ca84818d730c478785d635d31f81141edf96f640c1b93cff6f9"} err="failed to get container status \"fa96430090de0ca84818d730c478785d635d31f81141edf96f640c1b93cff6f9\": rpc error: code = NotFound desc = could not find container \"fa96430090de0ca84818d730c478785d635d31f81141edf96f640c1b93cff6f9\": container with ID starting with fa96430090de0ca84818d730c478785d635d31f81141edf96f640c1b93cff6f9 not found: ID does not exist"
Jul 28 17:47:32.437191 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:32.437171    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt"]
Jul 28 17:47:32.443314 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:32.443292    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2fxnt"]
Jul 28 17:47:33.152540 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:33.152511    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="558de6b2-e431-4591-b379-b94fe38e91e0" path="/var/lib/kubelet/pods/558de6b2-e431-4591-b379-b94fe38e91e0/volumes"
Jul 28 17:47:40.243075 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:40.243035    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw" podUID="1804a4bd-b407-4e35-b464-bea77a07b584" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:47:43.061828 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:43.061792    2543 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw" message="" err="command '/bin/sleep 15' exited with 137: "
Jul 28 17:47:43.062181 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:47:43.061832    2543 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw" podUID="1804a4bd-b407-4e35-b464-bea77a07b584" containerName="main" containerID="cri-o://84edd2154ea9282d87f69adea9eae1f5b2bbdde20f7ae6b654fcd74fdb78a858"
Jul 28 17:47:43.062181 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:43.061884    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw" podUID="1804a4bd-b407-4e35-b464-bea77a07b584" containerName="main" containerID="cri-o://84edd2154ea9282d87f69adea9eae1f5b2bbdde20f7ae6b654fcd74fdb78a858" gracePeriod=49
Jul 28 17:47:43.209199 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:43.209176    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw_1804a4bd-b407-4e35-b464-bea77a07b584/main/0.log"
Jul 28 17:47:43.211394 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:43.211377    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw"
Jul 28 17:47:43.363687 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:43.363595    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1804a4bd-b407-4e35-b464-bea77a07b584-tls-certs\") pod \"1804a4bd-b407-4e35-b464-bea77a07b584\" (UID: \"1804a4bd-b407-4e35-b464-bea77a07b584\") "
Jul 28 17:47:43.363687 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:43.363667    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tfjm\" (UniqueName: \"kubernetes.io/projected/1804a4bd-b407-4e35-b464-bea77a07b584-kube-api-access-5tfjm\") pod \"1804a4bd-b407-4e35-b464-bea77a07b584\" (UID: \"1804a4bd-b407-4e35-b464-bea77a07b584\") "
Jul 28 17:47:43.365658 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:43.365627    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1804a4bd-b407-4e35-b464-bea77a07b584-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1804a4bd-b407-4e35-b464-bea77a07b584" (UID: "1804a4bd-b407-4e35-b464-bea77a07b584"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:47:43.365658 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:43.365627    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1804a4bd-b407-4e35-b464-bea77a07b584-kube-api-access-5tfjm" (OuterVolumeSpecName: "kube-api-access-5tfjm") pod "1804a4bd-b407-4e35-b464-bea77a07b584" (UID: "1804a4bd-b407-4e35-b464-bea77a07b584"). InnerVolumeSpecName "kube-api-access-5tfjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:47:43.464563 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:43.464529    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tfjm\" (UniqueName: \"kubernetes.io/projected/1804a4bd-b407-4e35-b464-bea77a07b584-kube-api-access-5tfjm\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:47:43.464563 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:43.464561    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1804a4bd-b407-4e35-b464-bea77a07b584-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:47:43.467970 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:43.467951    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw_1804a4bd-b407-4e35-b464-bea77a07b584/main/0.log"
Jul 28 17:47:43.468059 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:43.467988    2543 generic.go:356] "Generic (PLEG): container finished" podID="1804a4bd-b407-4e35-b464-bea77a07b584" containerID="84edd2154ea9282d87f69adea9eae1f5b2bbdde20f7ae6b654fcd74fdb78a858" exitCode=1
Jul 28 17:47:43.468059 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:43.468020    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw" event={"ID":"1804a4bd-b407-4e35-b464-bea77a07b584","Type":"ContainerDied","Data":"84edd2154ea9282d87f69adea9eae1f5b2bbdde20f7ae6b654fcd74fdb78a858"}
Jul 28 17:47:43.468059 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:43.468039    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw"
Jul 28 17:47:43.468188 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:43.468076    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw" event={"ID":"1804a4bd-b407-4e35-b464-bea77a07b584","Type":"ContainerDied","Data":"b5d2cf20567f21087c03dad651499f11d8f980a9090ec903bac469281aaddc7e"}
Jul 28 17:47:43.468188 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:43.468099    2543 scope.go:117] "RemoveContainer" containerID="84edd2154ea9282d87f69adea9eae1f5b2bbdde20f7ae6b654fcd74fdb78a858"
Jul 28 17:47:43.476903 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:43.476887    2543 scope.go:117] "RemoveContainer" containerID="84edd2154ea9282d87f69adea9eae1f5b2bbdde20f7ae6b654fcd74fdb78a858"
Jul 28 17:47:43.477169 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:47:43.477127    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84edd2154ea9282d87f69adea9eae1f5b2bbdde20f7ae6b654fcd74fdb78a858\": container with ID starting with 84edd2154ea9282d87f69adea9eae1f5b2bbdde20f7ae6b654fcd74fdb78a858 not found: ID does not exist" containerID="84edd2154ea9282d87f69adea9eae1f5b2bbdde20f7ae6b654fcd74fdb78a858"
Jul 28 17:47:43.477215 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:43.477178    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84edd2154ea9282d87f69adea9eae1f5b2bbdde20f7ae6b654fcd74fdb78a858"} err="failed to get container status \"84edd2154ea9282d87f69adea9eae1f5b2bbdde20f7ae6b654fcd74fdb78a858\": rpc error: code = NotFound desc = could not find container \"84edd2154ea9282d87f69adea9eae1f5b2bbdde20f7ae6b654fcd74fdb78a858\": container with ID starting with 84edd2154ea9282d87f69adea9eae1f5b2bbdde20f7ae6b654fcd74fdb78a858 not found: ID does not exist"
Jul 28 17:47:43.489859 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:43.489839    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw"]
Jul 28 17:47:43.494218 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:43.494199    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf5fwjw"]
Jul 28 17:47:45.151883 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:45.151848    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1804a4bd-b407-4e35-b464-bea77a07b584" path="/var/lib/kubelet/pods/1804a4bd-b407-4e35-b464-bea77a07b584/volumes"
Jul 28 17:47:53.916959 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:53.916925    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t"]
Jul 28 17:47:53.917366 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:53.917263    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="558de6b2-e431-4591-b379-b94fe38e91e0" containerName="main"
Jul 28 17:47:53.917366 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:53.917275    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="558de6b2-e431-4591-b379-b94fe38e91e0" containerName="main"
Jul 28 17:47:53.917366 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:53.917289    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1804a4bd-b407-4e35-b464-bea77a07b584" containerName="main"
Jul 28 17:47:53.917366 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:53.917295    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="1804a4bd-b407-4e35-b464-bea77a07b584" containerName="main"
Jul 28 17:47:53.917366 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:53.917346    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="1804a4bd-b407-4e35-b464-bea77a07b584" containerName="main"
Jul 28 17:47:53.917366 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:53.917352    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="558de6b2-e431-4591-b379-b94fe38e91e0" containerName="main"
Jul 28 17:47:53.920267 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:53.920249    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t"
Jul 28 17:47:53.922890 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:53.922868    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-74dq9\""
Jul 28 17:47:53.923019 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:53.922868    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Jul 28 17:47:53.923019 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:53.922871    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Jul 28 17:47:53.923802 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:53.923786    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Jul 28 17:47:53.934056 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:53.934035    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t"]
Jul 28 17:47:53.947115 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:53.947084    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71926e8e-aece-4c27-b48e-20ac6721ecd6-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gt76t\" (UID: \"71926e8e-aece-4c27-b48e-20ac6721ecd6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t"
Jul 28 17:47:53.947115 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:53.947116    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71926e8e-aece-4c27-b48e-20ac6721ecd6-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gt76t\" (UID: \"71926e8e-aece-4c27-b48e-20ac6721ecd6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t"
Jul 28 17:47:53.947316 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:53.947171    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71926e8e-aece-4c27-b48e-20ac6721ecd6-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gt76t\" (UID: \"71926e8e-aece-4c27-b48e-20ac6721ecd6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t"
Jul 28 17:47:53.947316 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:53.947285    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71926e8e-aece-4c27-b48e-20ac6721ecd6-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gt76t\" (UID: \"71926e8e-aece-4c27-b48e-20ac6721ecd6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t"
Jul 28 17:47:53.947439 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:53.947324    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71926e8e-aece-4c27-b48e-20ac6721ecd6-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gt76t\" (UID: \"71926e8e-aece-4c27-b48e-20ac6721ecd6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t"
Jul 28 17:47:53.947439 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:53.947389    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmxsh\" (UniqueName: \"kubernetes.io/projected/71926e8e-aece-4c27-b48e-20ac6721ecd6-kube-api-access-cmxsh\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gt76t\" (UID: \"71926e8e-aece-4c27-b48e-20ac6721ecd6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t"
Jul 28 17:47:54.048444 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.048400    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmxsh\" (UniqueName: \"kubernetes.io/projected/71926e8e-aece-4c27-b48e-20ac6721ecd6-kube-api-access-cmxsh\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gt76t\" (UID: \"71926e8e-aece-4c27-b48e-20ac6721ecd6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t"
Jul 28 17:47:54.048609 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.048465    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71926e8e-aece-4c27-b48e-20ac6721ecd6-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gt76t\" (UID: \"71926e8e-aece-4c27-b48e-20ac6721ecd6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t"
Jul 28 17:47:54.048609 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.048492    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71926e8e-aece-4c27-b48e-20ac6721ecd6-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gt76t\" (UID: \"71926e8e-aece-4c27-b48e-20ac6721ecd6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t"
Jul 28 17:47:54.048609 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.048511    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71926e8e-aece-4c27-b48e-20ac6721ecd6-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gt76t\" (UID: \"71926e8e-aece-4c27-b48e-20ac6721ecd6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t"
Jul 28 17:47:54.048726 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.048648    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71926e8e-aece-4c27-b48e-20ac6721ecd6-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gt76t\" (UID: \"71926e8e-aece-4c27-b48e-20ac6721ecd6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t"
Jul 28 17:47:54.048726 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.048704    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71926e8e-aece-4c27-b48e-20ac6721ecd6-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gt76t\" (UID: \"71926e8e-aece-4c27-b48e-20ac6721ecd6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t"
Jul 28 17:47:54.048936 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.048915    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71926e8e-aece-4c27-b48e-20ac6721ecd6-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gt76t\" (UID: \"71926e8e-aece-4c27-b48e-20ac6721ecd6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t"
Jul 28 17:47:54.048975 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.048944    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71926e8e-aece-4c27-b48e-20ac6721ecd6-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gt76t\" (UID: \"71926e8e-aece-4c27-b48e-20ac6721ecd6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t"
Jul 28 17:47:54.049073 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.049051    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71926e8e-aece-4c27-b48e-20ac6721ecd6-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gt76t\" (UID: \"71926e8e-aece-4c27-b48e-20ac6721ecd6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t"
Jul 28 17:47:54.050841 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.050819    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71926e8e-aece-4c27-b48e-20ac6721ecd6-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gt76t\" (UID: \"71926e8e-aece-4c27-b48e-20ac6721ecd6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t"
Jul 28 17:47:54.051078 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.051061    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71926e8e-aece-4c27-b48e-20ac6721ecd6-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gt76t\" (UID: \"71926e8e-aece-4c27-b48e-20ac6721ecd6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t"
Jul 28 17:47:54.058157 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.057864    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmxsh\" (UniqueName: \"kubernetes.io/projected/71926e8e-aece-4c27-b48e-20ac6721ecd6-kube-api-access-cmxsh\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gt76t\" (UID: \"71926e8e-aece-4c27-b48e-20ac6721ecd6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t"
Jul 28 17:47:54.177075 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.176989    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7957j9jj"]
Jul 28 17:47:54.180566 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.180548    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7957j9jj"
Jul 28 17:47:54.185405 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.185386    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-gr6m4\""
Jul 28 17:47:54.194028 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.194007    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7957j9jj"]
Jul 28 17:47:54.230172 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.230125    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t"
Jul 28 17:47:54.250449 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.250424    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a5bf44-3583-43ac-bd68-8af9b6ec216a-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-7957j9jj\" (UID: \"d1a5bf44-3583-43ac-bd68-8af9b6ec216a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7957j9jj"
Jul 28 17:47:54.250572 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.250464    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffjvk\" (UniqueName: \"kubernetes.io/projected/d1a5bf44-3583-43ac-bd68-8af9b6ec216a-kube-api-access-ffjvk\") pod \"scheduler-custom-template-test-kserve-router-scheduler-7957j9jj\" (UID: \"d1a5bf44-3583-43ac-bd68-8af9b6ec216a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7957j9jj"
Jul 28 17:47:54.250572 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.250491    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a5bf44-3583-43ac-bd68-8af9b6ec216a-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-7957j9jj\" (UID: \"d1a5bf44-3583-43ac-bd68-8af9b6ec216a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7957j9jj"
Jul 28 17:47:54.250656 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.250588    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a5bf44-3583-43ac-bd68-8af9b6ec216a-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-7957j9jj\" (UID: \"d1a5bf44-3583-43ac-bd68-8af9b6ec216a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7957j9jj"
Jul 28 17:47:54.351774 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.351742    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a5bf44-3583-43ac-bd68-8af9b6ec216a-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-7957j9jj\" (UID: \"d1a5bf44-3583-43ac-bd68-8af9b6ec216a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7957j9jj"
Jul 28 17:47:54.351925 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.351790    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a5bf44-3583-43ac-bd68-8af9b6ec216a-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-7957j9jj\" (UID: \"d1a5bf44-3583-43ac-bd68-8af9b6ec216a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7957j9jj"
Jul 28 17:47:54.351925 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.351818    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffjvk\" (UniqueName: \"kubernetes.io/projected/d1a5bf44-3583-43ac-bd68-8af9b6ec216a-kube-api-access-ffjvk\") pod \"scheduler-custom-template-test-kserve-router-scheduler-7957j9jj\" (UID: \"d1a5bf44-3583-43ac-bd68-8af9b6ec216a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7957j9jj"
Jul 28 17:47:54.351925 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.351841    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a5bf44-3583-43ac-bd68-8af9b6ec216a-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-7957j9jj\" (UID: \"d1a5bf44-3583-43ac-bd68-8af9b6ec216a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7957j9jj"
Jul 28 17:47:54.352234 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.352213    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a5bf44-3583-43ac-bd68-8af9b6ec216a-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-7957j9jj\" (UID: \"d1a5bf44-3583-43ac-bd68-8af9b6ec216a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7957j9jj"
Jul 28 17:47:54.352314 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.352298    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a5bf44-3583-43ac-bd68-8af9b6ec216a-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-7957j9jj\" (UID: \"d1a5bf44-3583-43ac-bd68-8af9b6ec216a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7957j9jj"
Jul 28 17:47:54.354084 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.354065    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a5bf44-3583-43ac-bd68-8af9b6ec216a-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-7957j9jj\" (UID: \"d1a5bf44-3583-43ac-bd68-8af9b6ec216a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7957j9jj"
Jul 28 17:47:54.365555 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.365529    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffjvk\" (UniqueName: \"kubernetes.io/projected/d1a5bf44-3583-43ac-bd68-8af9b6ec216a-kube-api-access-ffjvk\") pod \"scheduler-custom-template-test-kserve-router-scheduler-7957j9jj\" (UID: \"d1a5bf44-3583-43ac-bd68-8af9b6ec216a\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7957j9jj"
Jul 28 17:47:54.368745 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.368693    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t"]
Jul 28 17:47:54.370961 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:47:54.370935    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71926e8e_aece_4c27_b48e_20ac6721ecd6.slice/crio-0d84d0ae8cf4c70a920c277074c8a25f26a5d47b83a3dcadb09aa7c2e19c848b WatchSource:0}: Error finding container 0d84d0ae8cf4c70a920c277074c8a25f26a5d47b83a3dcadb09aa7c2e19c848b: Status 404 returned error can't find the container with id 0d84d0ae8cf4c70a920c277074c8a25f26a5d47b83a3dcadb09aa7c2e19c848b
Jul 28 17:47:54.491324 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.490621    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7957j9jj"
Jul 28 17:47:54.524888 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.524845    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t" event={"ID":"71926e8e-aece-4c27-b48e-20ac6721ecd6","Type":"ContainerStarted","Data":"babda9e1a71a3db4673fba742379a8fd8f48e6e1bbf936dfec0279e98b1199dc"}
Jul 28 17:47:54.525058 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.524907    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t" event={"ID":"71926e8e-aece-4c27-b48e-20ac6721ecd6","Type":"ContainerStarted","Data":"0d84d0ae8cf4c70a920c277074c8a25f26a5d47b83a3dcadb09aa7c2e19c848b"}
Jul 28 17:47:54.555629 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.555573    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t" podStartSLOduration=1.5555494300000001 podStartE2EDuration="1.55554943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:47:54.552646676 +0000 UTC m=+1477.958920424" watchObservedRunningTime="2026-07-28 17:47:54.55554943 +0000 UTC m=+1477.961823177"
Jul 28 17:47:54.645435 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:54.645409    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7957j9jj"]
Jul 28 17:47:54.647332 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:47:54.647298    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a5bf44_3583_43ac_bd68_8af9b6ec216a.slice/crio-054c3de01a8d05d4cd261025547b23c9f6297ba25f10ebaf2d2722ba4a3a09ee WatchSource:0}: Error finding container 054c3de01a8d05d4cd261025547b23c9f6297ba25f10ebaf2d2722ba4a3a09ee: Status 404 returned error can't find the container with id 054c3de01a8d05d4cd261025547b23c9f6297ba25f10ebaf2d2722ba4a3a09ee
Jul 28 17:47:55.230604 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:55.230573    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t"
Jul 28 17:47:55.531833 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:55.531801    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7957j9jj" event={"ID":"d1a5bf44-3583-43ac-bd68-8af9b6ec216a","Type":"ContainerStarted","Data":"74d15e944ee107a9a891329cce5b6c5204d3a12d19f010737a978793cf156e83"}
Jul 28 17:47:55.532045 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:55.531854    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7957j9jj" event={"ID":"d1a5bf44-3583-43ac-bd68-8af9b6ec216a","Type":"ContainerStarted","Data":"054c3de01a8d05d4cd261025547b23c9f6297ba25f10ebaf2d2722ba4a3a09ee"}
Jul 28 17:47:55.532045 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:55.532021    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7957j9jj"
Jul 28 17:47:55.551363 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:47:55.551314    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7957j9jj" podStartSLOduration=1.5512965859999999 podStartE2EDuration="1.551296586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:47:55.549450944 +0000 UTC m=+1478.955724686" watchObservedRunningTime="2026-07-28 17:47:55.551296586 +0000 UTC m=+1478.957570326"
Jul 28 17:48:04.230702 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:04.230661    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t"
Jul 28 17:48:04.243108 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:04.243084    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t"
Jul 28 17:48:04.587295 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:04.587270    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t"
Jul 28 17:48:26.537993 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:26.537960    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7957j9jj"
Jul 28 17:48:37.331560 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.331527    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t"]
Jul 28 17:48:37.340440 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:48:37.340405    2543 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:48:37.340440 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:48:37.340440 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:48:37.340440 ip-10-0-135-177 kubenswrapper[2543]:  > containerID="babda9e1a71a3db4673fba742379a8fd8f48e6e1bbf936dfec0279e98b1199dc" cmd=["/bin/sleep","15"]
Jul 28 17:48:37.340742 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.340457    2543 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t" message="" err=<
Jul 28 17:48:37.340742 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:48:37.340742 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:48:37.340742 ip-10-0-135-177 kubenswrapper[2543]:  >
Jul 28 17:48:37.340742 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:48:37.340471    2543 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:48:37.340742 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:48:37.340742 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:48:37.340742 ip-10-0-135-177 kubenswrapper[2543]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t" podUID="71926e8e-aece-4c27-b48e-20ac6721ecd6" containerName="main" containerID="cri-o://babda9e1a71a3db4673fba742379a8fd8f48e6e1bbf936dfec0279e98b1199dc"
Jul 28 17:48:37.340742 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.340508    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t" podUID="71926e8e-aece-4c27-b48e-20ac6721ecd6" containerName="main" containerID="cri-o://babda9e1a71a3db4673fba742379a8fd8f48e6e1bbf936dfec0279e98b1199dc" gracePeriod=60
Jul 28 17:48:37.344700 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.344673    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7957j9jj"]
Jul 28 17:48:37.577300 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.577257    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t" podUID="71926e8e-aece-4c27-b48e-20ac6721ecd6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 28 17:48:37.715913 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.715881    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t"
Jul 28 17:48:37.728035 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.727998    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71926e8e-aece-4c27-b48e-20ac6721ecd6-dshm\") pod \"71926e8e-aece-4c27-b48e-20ac6721ecd6\" (UID: \"71926e8e-aece-4c27-b48e-20ac6721ecd6\") "
Jul 28 17:48:37.728241 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.728047    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmxsh\" (UniqueName: \"kubernetes.io/projected/71926e8e-aece-4c27-b48e-20ac6721ecd6-kube-api-access-cmxsh\") pod \"71926e8e-aece-4c27-b48e-20ac6721ecd6\" (UID: \"71926e8e-aece-4c27-b48e-20ac6721ecd6\") "
Jul 28 17:48:37.728241 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.728097    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71926e8e-aece-4c27-b48e-20ac6721ecd6-tmp-dir\") pod \"71926e8e-aece-4c27-b48e-20ac6721ecd6\" (UID: \"71926e8e-aece-4c27-b48e-20ac6721ecd6\") "
Jul 28 17:48:37.728241 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.728202    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71926e8e-aece-4c27-b48e-20ac6721ecd6-home\") pod \"71926e8e-aece-4c27-b48e-20ac6721ecd6\" (UID: \"71926e8e-aece-4c27-b48e-20ac6721ecd6\") "
Jul 28 17:48:37.728414 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.728252    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71926e8e-aece-4c27-b48e-20ac6721ecd6-model-cache\") pod \"71926e8e-aece-4c27-b48e-20ac6721ecd6\" (UID: \"71926e8e-aece-4c27-b48e-20ac6721ecd6\") "
Jul 28 17:48:37.728414 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.728283    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71926e8e-aece-4c27-b48e-20ac6721ecd6-tls-certs\") pod \"71926e8e-aece-4c27-b48e-20ac6721ecd6\" (UID: \"71926e8e-aece-4c27-b48e-20ac6721ecd6\") "
Jul 28 17:48:37.728513 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.728412    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71926e8e-aece-4c27-b48e-20ac6721ecd6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "71926e8e-aece-4c27-b48e-20ac6721ecd6" (UID: "71926e8e-aece-4c27-b48e-20ac6721ecd6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:48:37.728513 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.728429    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71926e8e-aece-4c27-b48e-20ac6721ecd6-home" (OuterVolumeSpecName: "home") pod "71926e8e-aece-4c27-b48e-20ac6721ecd6" (UID: "71926e8e-aece-4c27-b48e-20ac6721ecd6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:48:37.728513 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.728491    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71926e8e-aece-4c27-b48e-20ac6721ecd6-model-cache" (OuterVolumeSpecName: "model-cache") pod "71926e8e-aece-4c27-b48e-20ac6721ecd6" (UID: "71926e8e-aece-4c27-b48e-20ac6721ecd6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:48:37.728653 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.728592    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71926e8e-aece-4c27-b48e-20ac6721ecd6-tmp-dir\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:48:37.728653 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.728612    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71926e8e-aece-4c27-b48e-20ac6721ecd6-home\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:48:37.728653 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.728625    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71926e8e-aece-4c27-b48e-20ac6721ecd6-model-cache\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:48:37.730325 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.730292    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71926e8e-aece-4c27-b48e-20ac6721ecd6-dshm" (OuterVolumeSpecName: "dshm") pod "71926e8e-aece-4c27-b48e-20ac6721ecd6" (UID: "71926e8e-aece-4c27-b48e-20ac6721ecd6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:48:37.730420 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.730373    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71926e8e-aece-4c27-b48e-20ac6721ecd6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "71926e8e-aece-4c27-b48e-20ac6721ecd6" (UID: "71926e8e-aece-4c27-b48e-20ac6721ecd6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:48:37.730420 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.730382    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71926e8e-aece-4c27-b48e-20ac6721ecd6-kube-api-access-cmxsh" (OuterVolumeSpecName: "kube-api-access-cmxsh") pod "71926e8e-aece-4c27-b48e-20ac6721ecd6" (UID: "71926e8e-aece-4c27-b48e-20ac6721ecd6"). InnerVolumeSpecName "kube-api-access-cmxsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:48:37.750683 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.750658    2543 generic.go:356] "Generic (PLEG): container finished" podID="71926e8e-aece-4c27-b48e-20ac6721ecd6" containerID="babda9e1a71a3db4673fba742379a8fd8f48e6e1bbf936dfec0279e98b1199dc" exitCode=0
Jul 28 17:48:37.750827 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.750721    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t"
Jul 28 17:48:37.750827 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.750742    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t" event={"ID":"71926e8e-aece-4c27-b48e-20ac6721ecd6","Type":"ContainerDied","Data":"babda9e1a71a3db4673fba742379a8fd8f48e6e1bbf936dfec0279e98b1199dc"}
Jul 28 17:48:37.750827 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.750794    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t" event={"ID":"71926e8e-aece-4c27-b48e-20ac6721ecd6","Type":"ContainerDied","Data":"0d84d0ae8cf4c70a920c277074c8a25f26a5d47b83a3dcadb09aa7c2e19c848b"}
Jul 28 17:48:37.750827 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.750817    2543 scope.go:117] "RemoveContainer" containerID="babda9e1a71a3db4673fba742379a8fd8f48e6e1bbf936dfec0279e98b1199dc"
Jul 28 17:48:37.763482 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.761255    2543 scope.go:117] "RemoveContainer" containerID="babda9e1a71a3db4673fba742379a8fd8f48e6e1bbf936dfec0279e98b1199dc"
Jul 28 17:48:37.763482 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:48:37.762037    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"babda9e1a71a3db4673fba742379a8fd8f48e6e1bbf936dfec0279e98b1199dc\": container with ID starting with babda9e1a71a3db4673fba742379a8fd8f48e6e1bbf936dfec0279e98b1199dc not found: ID does not exist" containerID="babda9e1a71a3db4673fba742379a8fd8f48e6e1bbf936dfec0279e98b1199dc"
Jul 28 17:48:37.763482 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.762070    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"babda9e1a71a3db4673fba742379a8fd8f48e6e1bbf936dfec0279e98b1199dc"} err="failed to get container status \"babda9e1a71a3db4673fba742379a8fd8f48e6e1bbf936dfec0279e98b1199dc\": rpc error: code = NotFound desc = could not find container \"babda9e1a71a3db4673fba742379a8fd8f48e6e1bbf936dfec0279e98b1199dc\": container with ID starting with babda9e1a71a3db4673fba742379a8fd8f48e6e1bbf936dfec0279e98b1199dc not found: ID does not exist"
Jul 28 17:48:37.788197 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.788175    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t"]
Jul 28 17:48:37.801589 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.801570    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gt76t"]
Jul 28 17:48:37.829466 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.829440    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71926e8e-aece-4c27-b48e-20ac6721ecd6-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:48:37.829466 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.829460    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71926e8e-aece-4c27-b48e-20ac6721ecd6-dshm\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:48:37.829607 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:37.829472    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmxsh\" (UniqueName: \"kubernetes.io/projected/71926e8e-aece-4c27-b48e-20ac6721ecd6-kube-api-access-cmxsh\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:48:39.152811 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:39.152738    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71926e8e-aece-4c27-b48e-20ac6721ecd6" path="/var/lib/kubelet/pods/71926e8e-aece-4c27-b48e-20ac6721ecd6/volumes"
Jul 28 17:48:46.538391 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:46.538355    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7957j9jj" podUID="d1a5bf44-3583-43ac-bd68-8af9b6ec216a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:48:52.362412 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:52.362371    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7957j9jj" podUID="d1a5bf44-3583-43ac-bd68-8af9b6ec216a" containerName="main" containerID="cri-o://74d15e944ee107a9a891329cce5b6c5204d3a12d19f010737a978793cf156e83" gracePeriod=45
Jul 28 17:48:52.618779 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:52.618723    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7957j9jj"
Jul 28 17:48:52.655001 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:52.654978    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a5bf44-3583-43ac-bd68-8af9b6ec216a-cachi2-cache\") pod \"d1a5bf44-3583-43ac-bd68-8af9b6ec216a\" (UID: \"d1a5bf44-3583-43ac-bd68-8af9b6ec216a\") "
Jul 28 17:48:52.655161 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:52.655017    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffjvk\" (UniqueName: \"kubernetes.io/projected/d1a5bf44-3583-43ac-bd68-8af9b6ec216a-kube-api-access-ffjvk\") pod \"d1a5bf44-3583-43ac-bd68-8af9b6ec216a\" (UID: \"d1a5bf44-3583-43ac-bd68-8af9b6ec216a\") "
Jul 28 17:48:52.655161 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:52.655059    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a5bf44-3583-43ac-bd68-8af9b6ec216a-tls-certs\") pod \"d1a5bf44-3583-43ac-bd68-8af9b6ec216a\" (UID: \"d1a5bf44-3583-43ac-bd68-8af9b6ec216a\") "
Jul 28 17:48:52.655161 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:52.655082    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a5bf44-3583-43ac-bd68-8af9b6ec216a-tokenizer-cache\") pod \"d1a5bf44-3583-43ac-bd68-8af9b6ec216a\" (UID: \"d1a5bf44-3583-43ac-bd68-8af9b6ec216a\") "
Jul 28 17:48:52.655325 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:52.655264    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a5bf44-3583-43ac-bd68-8af9b6ec216a-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "d1a5bf44-3583-43ac-bd68-8af9b6ec216a" (UID: "d1a5bf44-3583-43ac-bd68-8af9b6ec216a"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:48:52.655472 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:52.655434    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a5bf44-3583-43ac-bd68-8af9b6ec216a-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "d1a5bf44-3583-43ac-bd68-8af9b6ec216a" (UID: "d1a5bf44-3583-43ac-bd68-8af9b6ec216a"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:48:52.657183 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:52.657159    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a5bf44-3583-43ac-bd68-8af9b6ec216a-kube-api-access-ffjvk" (OuterVolumeSpecName: "kube-api-access-ffjvk") pod "d1a5bf44-3583-43ac-bd68-8af9b6ec216a" (UID: "d1a5bf44-3583-43ac-bd68-8af9b6ec216a"). InnerVolumeSpecName "kube-api-access-ffjvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:48:52.657183 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:52.657171    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1a5bf44-3583-43ac-bd68-8af9b6ec216a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d1a5bf44-3583-43ac-bd68-8af9b6ec216a" (UID: "d1a5bf44-3583-43ac-bd68-8af9b6ec216a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:48:52.755783 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:52.755760    2543 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a5bf44-3583-43ac-bd68-8af9b6ec216a-cachi2-cache\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:48:52.755783 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:52.755782    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffjvk\" (UniqueName: \"kubernetes.io/projected/d1a5bf44-3583-43ac-bd68-8af9b6ec216a-kube-api-access-ffjvk\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:48:52.755940 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:52.755793    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a5bf44-3583-43ac-bd68-8af9b6ec216a-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:48:52.755940 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:52.755801    2543 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1a5bf44-3583-43ac-bd68-8af9b6ec216a-tokenizer-cache\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:48:52.825929 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:52.825896    2543 generic.go:356] "Generic (PLEG): container finished" podID="d1a5bf44-3583-43ac-bd68-8af9b6ec216a" containerID="74d15e944ee107a9a891329cce5b6c5204d3a12d19f010737a978793cf156e83" exitCode=0
Jul 28 17:48:52.826062 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:52.825945    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7957j9jj"
Jul 28 17:48:52.826062 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:52.825981    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7957j9jj" event={"ID":"d1a5bf44-3583-43ac-bd68-8af9b6ec216a","Type":"ContainerDied","Data":"74d15e944ee107a9a891329cce5b6c5204d3a12d19f010737a978793cf156e83"}
Jul 28 17:48:52.826062 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:52.826035    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7957j9jj" event={"ID":"d1a5bf44-3583-43ac-bd68-8af9b6ec216a","Type":"ContainerDied","Data":"054c3de01a8d05d4cd261025547b23c9f6297ba25f10ebaf2d2722ba4a3a09ee"}
Jul 28 17:48:52.826062 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:52.826057    2543 scope.go:117] "RemoveContainer" containerID="74d15e944ee107a9a891329cce5b6c5204d3a12d19f010737a978793cf156e83"
Jul 28 17:48:52.834681 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:52.834666    2543 scope.go:117] "RemoveContainer" containerID="74d15e944ee107a9a891329cce5b6c5204d3a12d19f010737a978793cf156e83"
Jul 28 17:48:52.834927 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:48:52.834911    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74d15e944ee107a9a891329cce5b6c5204d3a12d19f010737a978793cf156e83\": container with ID starting with 74d15e944ee107a9a891329cce5b6c5204d3a12d19f010737a978793cf156e83 not found: ID does not exist" containerID="74d15e944ee107a9a891329cce5b6c5204d3a12d19f010737a978793cf156e83"
Jul 28 17:48:52.834974 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:52.834934    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74d15e944ee107a9a891329cce5b6c5204d3a12d19f010737a978793cf156e83"} err="failed to get container status \"74d15e944ee107a9a891329cce5b6c5204d3a12d19f010737a978793cf156e83\": rpc error: code = NotFound desc = could not find container \"74d15e944ee107a9a891329cce5b6c5204d3a12d19f010737a978793cf156e83\": container with ID starting with 74d15e944ee107a9a891329cce5b6c5204d3a12d19f010737a978793cf156e83 not found: ID does not exist"
Jul 28 17:48:52.848099 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:52.848078    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7957j9jj"]
Jul 28 17:48:52.857695 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:52.854475    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-7957j9jj"]
Jul 28 17:48:53.152285 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:48:53.152257    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1a5bf44-3583-43ac-bd68-8af9b6ec216a" path="/var/lib/kubelet/pods/d1a5bf44-3583-43ac-bd68-8af9b6ec216a/volumes"
Jul 28 17:49:01.862965 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:01.862929    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"]
Jul 28 17:49:01.863457 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:01.863437    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1a5bf44-3583-43ac-bd68-8af9b6ec216a" containerName="main"
Jul 28 17:49:01.863505 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:01.863463    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1a5bf44-3583-43ac-bd68-8af9b6ec216a" containerName="main"
Jul 28 17:49:01.863505 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:01.863485    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71926e8e-aece-4c27-b48e-20ac6721ecd6" containerName="main"
Jul 28 17:49:01.863505 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:01.863494    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="71926e8e-aece-4c27-b48e-20ac6721ecd6" containerName="main"
Jul 28 17:49:01.863605 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:01.863593    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1a5bf44-3583-43ac-bd68-8af9b6ec216a" containerName="main"
Jul 28 17:49:01.863641 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:01.863616    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="71926e8e-aece-4c27-b48e-20ac6721ecd6" containerName="main"
Jul 28 17:49:01.868579 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:01.868559    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"
Jul 28 17:49:01.871056 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:01.871037    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-lvv7g\""
Jul 28 17:49:01.871229 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:01.871210    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-722p8\""
Jul 28 17:49:01.872429 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:01.872399    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Jul 28 17:49:01.872552 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:01.872436    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 28 17:49:01.872552 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:01.872475    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Jul 28 17:49:01.879809 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:01.879788    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"]
Jul 28 17:49:01.936004 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:01.935963    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/156fe330-1e45-49fa-b552-3761d0c58935-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq\" (UID: \"156fe330-1e45-49fa-b552-3761d0c58935\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"
Jul 28 17:49:01.936168 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:01.936024    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/156fe330-1e45-49fa-b552-3761d0c58935-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq\" (UID: \"156fe330-1e45-49fa-b552-3761d0c58935\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"
Jul 28 17:49:01.936168 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:01.936082    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/156fe330-1e45-49fa-b552-3761d0c58935-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq\" (UID: \"156fe330-1e45-49fa-b552-3761d0c58935\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"
Jul 28 17:49:01.936168 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:01.936108    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/156fe330-1e45-49fa-b552-3761d0c58935-home\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq\" (UID: \"156fe330-1e45-49fa-b552-3761d0c58935\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"
Jul 28 17:49:01.936291 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:01.936218    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/156fe330-1e45-49fa-b552-3761d0c58935-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq\" (UID: \"156fe330-1e45-49fa-b552-3761d0c58935\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"
Jul 28 17:49:01.936291 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:01.936256    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xglct\" (UniqueName: \"kubernetes.io/projected/156fe330-1e45-49fa-b552-3761d0c58935-kube-api-access-xglct\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq\" (UID: \"156fe330-1e45-49fa-b552-3761d0c58935\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"
Jul 28 17:49:02.036995 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:02.036962    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/156fe330-1e45-49fa-b552-3761d0c58935-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq\" (UID: \"156fe330-1e45-49fa-b552-3761d0c58935\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"
Jul 28 17:49:02.037154 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:02.037008    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xglct\" (UniqueName: \"kubernetes.io/projected/156fe330-1e45-49fa-b552-3761d0c58935-kube-api-access-xglct\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq\" (UID: \"156fe330-1e45-49fa-b552-3761d0c58935\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"
Jul 28 17:49:02.037154 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:02.037043    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/156fe330-1e45-49fa-b552-3761d0c58935-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq\" (UID: \"156fe330-1e45-49fa-b552-3761d0c58935\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"
Jul 28 17:49:02.037154 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:02.037095    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/156fe330-1e45-49fa-b552-3761d0c58935-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq\" (UID: \"156fe330-1e45-49fa-b552-3761d0c58935\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"
Jul 28 17:49:02.037154 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:02.037144    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/156fe330-1e45-49fa-b552-3761d0c58935-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq\" (UID: \"156fe330-1e45-49fa-b552-3761d0c58935\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"
Jul 28 17:49:02.037348 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:02.037167    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/156fe330-1e45-49fa-b552-3761d0c58935-home\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq\" (UID: \"156fe330-1e45-49fa-b552-3761d0c58935\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"
Jul 28 17:49:02.037464 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:02.037443    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/156fe330-1e45-49fa-b552-3761d0c58935-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq\" (UID: \"156fe330-1e45-49fa-b552-3761d0c58935\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"
Jul 28 17:49:02.037551 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:02.037530    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/156fe330-1e45-49fa-b552-3761d0c58935-home\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq\" (UID: \"156fe330-1e45-49fa-b552-3761d0c58935\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"
Jul 28 17:49:02.037702 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:02.037685    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/156fe330-1e45-49fa-b552-3761d0c58935-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq\" (UID: \"156fe330-1e45-49fa-b552-3761d0c58935\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"
Jul 28 17:49:02.039537 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:02.039515    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/156fe330-1e45-49fa-b552-3761d0c58935-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq\" (UID: \"156fe330-1e45-49fa-b552-3761d0c58935\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"
Jul 28 17:49:02.039688 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:02.039673    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/156fe330-1e45-49fa-b552-3761d0c58935-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq\" (UID: \"156fe330-1e45-49fa-b552-3761d0c58935\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"
Jul 28 17:49:02.046184 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:02.046161    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xglct\" (UniqueName: \"kubernetes.io/projected/156fe330-1e45-49fa-b552-3761d0c58935-kube-api-access-xglct\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq\" (UID: \"156fe330-1e45-49fa-b552-3761d0c58935\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"
Jul 28 17:49:02.156768 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:02.156683    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f92pd2q"]
Jul 28 17:49:02.160894 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:02.160871    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f92pd2q"
Jul 28 17:49:02.163789 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:02.163766    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-vzb22\""
Jul 28 17:49:02.172971 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:02.172947    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f92pd2q"]
Jul 28 17:49:02.179715 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:02.179697    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"
Jul 28 17:49:02.238672 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:02.238641    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ff28b47-9a34-4fd4-b850-0e097fa16217-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f92pd2q\" (UID: \"6ff28b47-9a34-4fd4-b850-0e097fa16217\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f92pd2q"
Jul 28 17:49:02.238919 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:02.238720    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6prp\" (UniqueName: \"kubernetes.io/projected/6ff28b47-9a34-4fd4-b850-0e097fa16217-kube-api-access-x6prp\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f92pd2q\" (UID: \"6ff28b47-9a34-4fd4-b850-0e097fa16217\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f92pd2q"
Jul 28 17:49:02.314872 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:02.314840    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"]
Jul 28 17:49:02.316355 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:49:02.316328    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156fe330_1e45_49fa_b552_3761d0c58935.slice/crio-2b92fee77b7ef6abd8af66d971d1e5a31175c2e102b4483184b90010e3d6c0bf WatchSource:0}: Error finding container 2b92fee77b7ef6abd8af66d971d1e5a31175c2e102b4483184b90010e3d6c0bf: Status 404 returned error can't find the container with id 2b92fee77b7ef6abd8af66d971d1e5a31175c2e102b4483184b90010e3d6c0bf
Jul 28 17:49:02.339930 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:02.339907    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6prp\" (UniqueName: \"kubernetes.io/projected/6ff28b47-9a34-4fd4-b850-0e097fa16217-kube-api-access-x6prp\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f92pd2q\" (UID: \"6ff28b47-9a34-4fd4-b850-0e097fa16217\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f92pd2q"
Jul 28 17:49:02.340041 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:02.339957    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ff28b47-9a34-4fd4-b850-0e097fa16217-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f92pd2q\" (UID: \"6ff28b47-9a34-4fd4-b850-0e097fa16217\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f92pd2q"
Jul 28 17:49:02.342559 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:02.342535    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ff28b47-9a34-4fd4-b850-0e097fa16217-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f92pd2q\" (UID: \"6ff28b47-9a34-4fd4-b850-0e097fa16217\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f92pd2q"
Jul 28 17:49:02.348720 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:02.348700    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6prp\" (UniqueName: \"kubernetes.io/projected/6ff28b47-9a34-4fd4-b850-0e097fa16217-kube-api-access-x6prp\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f92pd2q\" (UID: \"6ff28b47-9a34-4fd4-b850-0e097fa16217\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f92pd2q"
Jul 28 17:49:02.471857 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:02.471781    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f92pd2q"
Jul 28 17:49:02.605061 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:02.605030    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f92pd2q"]
Jul 28 17:49:02.606906 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:49:02.606864    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff28b47_9a34_4fd4_b850_0e097fa16217.slice/crio-6445ab7b69dbd61f2f6181078e69561916ffe5cba9253cfaffee1a124c6ecfd3 WatchSource:0}: Error finding container 6445ab7b69dbd61f2f6181078e69561916ffe5cba9253cfaffee1a124c6ecfd3: Status 404 returned error can't find the container with id 6445ab7b69dbd61f2f6181078e69561916ffe5cba9253cfaffee1a124c6ecfd3
Jul 28 17:49:02.877593 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:02.877556    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f92pd2q" event={"ID":"6ff28b47-9a34-4fd4-b850-0e097fa16217","Type":"ContainerStarted","Data":"29ba8d83743ed38ee4991f223b6b0fcc1e6793d8075c90db0fe094a58bffba3f"}
Jul 28 17:49:02.878080 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:02.877619    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f92pd2q" event={"ID":"6ff28b47-9a34-4fd4-b850-0e097fa16217","Type":"ContainerStarted","Data":"6445ab7b69dbd61f2f6181078e69561916ffe5cba9253cfaffee1a124c6ecfd3"}
Jul 28 17:49:02.878080 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:02.877743    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f92pd2q"
Jul 28 17:49:02.878781 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:02.878759    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq" event={"ID":"156fe330-1e45-49fa-b552-3761d0c58935","Type":"ContainerStarted","Data":"2b92fee77b7ef6abd8af66d971d1e5a31175c2e102b4483184b90010e3d6c0bf"}
Jul 28 17:49:02.897946 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:02.897896    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f92pd2q" podStartSLOduration=0.897881023 podStartE2EDuration="897.881023ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:49:02.896673186 +0000 UTC m=+1546.302946929" watchObservedRunningTime="2026-07-28 17:49:02.897881023 +0000 UTC m=+1546.304154763"
Jul 28 17:49:04.889796 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:04.889751    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq" event={"ID":"156fe330-1e45-49fa-b552-3761d0c58935","Type":"ContainerStarted","Data":"847a301a0a2a8407eb0d9865f28501e15509ec666c29b9ee768c842c296f8449"}
Jul 28 17:49:04.890171 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:04.889890    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"
Jul 28 17:49:05.897394 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:05.897362    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq" event={"ID":"156fe330-1e45-49fa-b552-3761d0c58935","Type":"ContainerStarted","Data":"eeed75de1af1b630a4170f41288ec810c43f30364c1114fb3762f62eefa93880"}
Jul 28 17:49:05.919368 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:05.919320    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq" podStartSLOduration=3.134827406 podStartE2EDuration="4.919305107s" totalImagesPullingTime="1.784477701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:49:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:49:05.917425479 +0000 UTC m=+1549.323699224" watchObservedRunningTime="2026-07-28 17:49:05.919305107 +0000 UTC m=+1549.325578847"
Jul 28 17:49:06.179984 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:06.179897    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"
Jul 28 17:49:12.180198 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:12.180159    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"
Jul 28 17:49:12.192508 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:12.192477    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"
Jul 28 17:49:12.948196 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:12.948162    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"
Jul 28 17:49:23.949349 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:23.949318    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"
Jul 28 17:49:33.885341 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:49:33.885311    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f92pd2q"
Jul 28 17:50:36.174249 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:36.174208    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f92pd2q"]
Jul 28 17:50:36.176662 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:36.176635    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"]
Jul 28 17:50:36.184750 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:50:36.184718    2543 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:50:36.184750 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:50:36.184750 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:50:36.184750 ip-10-0-135-177 kubenswrapper[2543]:  > containerID="eeed75de1af1b630a4170f41288ec810c43f30364c1114fb3762f62eefa93880" cmd=["/bin/sleep","15"]
Jul 28 17:50:36.185010 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:36.184768    2543 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq" message="" err=<
Jul 28 17:50:36.185010 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:50:36.185010 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:50:36.185010 ip-10-0-135-177 kubenswrapper[2543]:  >
Jul 28 17:50:36.185010 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:50:36.184780    2543 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:50:36.185010 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:50:36.185010 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:50:36.185010 ip-10-0-135-177 kubenswrapper[2543]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq" podUID="156fe330-1e45-49fa-b552-3761d0c58935" containerName="main" containerID="cri-o://eeed75de1af1b630a4170f41288ec810c43f30364c1114fb3762f62eefa93880"
Jul 28 17:50:36.185010 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:36.184824    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq" podUID="156fe330-1e45-49fa-b552-3761d0c58935" containerName="main" containerID="cri-o://eeed75de1af1b630a4170f41288ec810c43f30364c1114fb3762f62eefa93880" gracePeriod=60
Jul 28 17:50:36.402073 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:36.402040    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq" podUID="156fe330-1e45-49fa-b552-3761d0c58935" containerName="llm-d-routing-sidecar" containerID="cri-o://847a301a0a2a8407eb0d9865f28501e15509ec666c29b9ee768c842c296f8449" gracePeriod=60
Jul 28 17:50:36.657273 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:36.657250    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"
Jul 28 17:50:36.761299 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:36.761268    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/156fe330-1e45-49fa-b552-3761d0c58935-tmp-dir\") pod \"156fe330-1e45-49fa-b552-3761d0c58935\" (UID: \"156fe330-1e45-49fa-b552-3761d0c58935\") "
Jul 28 17:50:36.761482 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:36.761330    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/156fe330-1e45-49fa-b552-3761d0c58935-tls-certs\") pod \"156fe330-1e45-49fa-b552-3761d0c58935\" (UID: \"156fe330-1e45-49fa-b552-3761d0c58935\") "
Jul 28 17:50:36.761482 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:36.761414    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/156fe330-1e45-49fa-b552-3761d0c58935-model-cache\") pod \"156fe330-1e45-49fa-b552-3761d0c58935\" (UID: \"156fe330-1e45-49fa-b552-3761d0c58935\") "
Jul 28 17:50:36.761482 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:36.761442    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/156fe330-1e45-49fa-b552-3761d0c58935-dshm\") pod \"156fe330-1e45-49fa-b552-3761d0c58935\" (UID: \"156fe330-1e45-49fa-b552-3761d0c58935\") "
Jul 28 17:50:36.761482 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:36.761463    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/156fe330-1e45-49fa-b552-3761d0c58935-home\") pod \"156fe330-1e45-49fa-b552-3761d0c58935\" (UID: \"156fe330-1e45-49fa-b552-3761d0c58935\") "
Jul 28 17:50:36.761705 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:36.761486    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xglct\" (UniqueName: \"kubernetes.io/projected/156fe330-1e45-49fa-b552-3761d0c58935-kube-api-access-xglct\") pod \"156fe330-1e45-49fa-b552-3761d0c58935\" (UID: \"156fe330-1e45-49fa-b552-3761d0c58935\") "
Jul 28 17:50:36.761705 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:36.761568    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156fe330-1e45-49fa-b552-3761d0c58935-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "156fe330-1e45-49fa-b552-3761d0c58935" (UID: "156fe330-1e45-49fa-b552-3761d0c58935"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:50:36.761807 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:36.761710    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156fe330-1e45-49fa-b552-3761d0c58935-model-cache" (OuterVolumeSpecName: "model-cache") pod "156fe330-1e45-49fa-b552-3761d0c58935" (UID: "156fe330-1e45-49fa-b552-3761d0c58935"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:50:36.761807 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:36.761738    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156fe330-1e45-49fa-b552-3761d0c58935-home" (OuterVolumeSpecName: "home") pod "156fe330-1e45-49fa-b552-3761d0c58935" (UID: "156fe330-1e45-49fa-b552-3761d0c58935"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:50:36.761807 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:36.761793    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/156fe330-1e45-49fa-b552-3761d0c58935-tmp-dir\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:50:36.763726 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:36.763699    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156fe330-1e45-49fa-b552-3761d0c58935-dshm" (OuterVolumeSpecName: "dshm") pod "156fe330-1e45-49fa-b552-3761d0c58935" (UID: "156fe330-1e45-49fa-b552-3761d0c58935"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:50:36.763829 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:36.763724    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/156fe330-1e45-49fa-b552-3761d0c58935-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "156fe330-1e45-49fa-b552-3761d0c58935" (UID: "156fe330-1e45-49fa-b552-3761d0c58935"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:50:36.763829 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:36.763766    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156fe330-1e45-49fa-b552-3761d0c58935-kube-api-access-xglct" (OuterVolumeSpecName: "kube-api-access-xglct") pod "156fe330-1e45-49fa-b552-3761d0c58935" (UID: "156fe330-1e45-49fa-b552-3761d0c58935"). InnerVolumeSpecName "kube-api-access-xglct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:50:36.863248 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:36.863200    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/156fe330-1e45-49fa-b552-3761d0c58935-model-cache\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:50:36.863248 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:36.863238    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/156fe330-1e45-49fa-b552-3761d0c58935-dshm\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:50:36.863248 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:36.863248    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/156fe330-1e45-49fa-b552-3761d0c58935-home\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:50:36.863248 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:36.863257    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xglct\" (UniqueName: \"kubernetes.io/projected/156fe330-1e45-49fa-b552-3761d0c58935-kube-api-access-xglct\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:50:36.863248 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:36.863266    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/156fe330-1e45-49fa-b552-3761d0c58935-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:50:37.352790 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:37.352757    2543 generic.go:356] "Generic (PLEG): container finished" podID="156fe330-1e45-49fa-b552-3761d0c58935" containerID="eeed75de1af1b630a4170f41288ec810c43f30364c1114fb3762f62eefa93880" exitCode=0
Jul 28 17:50:37.352790 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:37.352782    2543 generic.go:356] "Generic (PLEG): container finished" podID="156fe330-1e45-49fa-b552-3761d0c58935" containerID="847a301a0a2a8407eb0d9865f28501e15509ec666c29b9ee768c842c296f8449" exitCode=0
Jul 28 17:50:37.353285 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:37.352822    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"
Jul 28 17:50:37.353285 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:37.352848    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq" event={"ID":"156fe330-1e45-49fa-b552-3761d0c58935","Type":"ContainerDied","Data":"eeed75de1af1b630a4170f41288ec810c43f30364c1114fb3762f62eefa93880"}
Jul 28 17:50:37.353285 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:37.352900    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq" event={"ID":"156fe330-1e45-49fa-b552-3761d0c58935","Type":"ContainerDied","Data":"847a301a0a2a8407eb0d9865f28501e15509ec666c29b9ee768c842c296f8449"}
Jul 28 17:50:37.353285 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:37.352919    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq" event={"ID":"156fe330-1e45-49fa-b552-3761d0c58935","Type":"ContainerDied","Data":"2b92fee77b7ef6abd8af66d971d1e5a31175c2e102b4483184b90010e3d6c0bf"}
Jul 28 17:50:37.353285 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:37.352940    2543 scope.go:117] "RemoveContainer" containerID="eeed75de1af1b630a4170f41288ec810c43f30364c1114fb3762f62eefa93880"
Jul 28 17:50:37.361841 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:37.361795    2543 scope.go:117] "RemoveContainer" containerID="847a301a0a2a8407eb0d9865f28501e15509ec666c29b9ee768c842c296f8449"
Jul 28 17:50:37.370211 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:37.370192    2543 scope.go:117] "RemoveContainer" containerID="eeed75de1af1b630a4170f41288ec810c43f30364c1114fb3762f62eefa93880"
Jul 28 17:50:37.370477 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:50:37.370460    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeed75de1af1b630a4170f41288ec810c43f30364c1114fb3762f62eefa93880\": container with ID starting with eeed75de1af1b630a4170f41288ec810c43f30364c1114fb3762f62eefa93880 not found: ID does not exist" containerID="eeed75de1af1b630a4170f41288ec810c43f30364c1114fb3762f62eefa93880"
Jul 28 17:50:37.370552 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:37.370486    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeed75de1af1b630a4170f41288ec810c43f30364c1114fb3762f62eefa93880"} err="failed to get container status \"eeed75de1af1b630a4170f41288ec810c43f30364c1114fb3762f62eefa93880\": rpc error: code = NotFound desc = could not find container \"eeed75de1af1b630a4170f41288ec810c43f30364c1114fb3762f62eefa93880\": container with ID starting with eeed75de1af1b630a4170f41288ec810c43f30364c1114fb3762f62eefa93880 not found: ID does not exist"
Jul 28 17:50:37.370552 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:37.370502    2543 scope.go:117] "RemoveContainer" containerID="847a301a0a2a8407eb0d9865f28501e15509ec666c29b9ee768c842c296f8449"
Jul 28 17:50:37.370744 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:50:37.370727    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"847a301a0a2a8407eb0d9865f28501e15509ec666c29b9ee768c842c296f8449\": container with ID starting with 847a301a0a2a8407eb0d9865f28501e15509ec666c29b9ee768c842c296f8449 not found: ID does not exist" containerID="847a301a0a2a8407eb0d9865f28501e15509ec666c29b9ee768c842c296f8449"
Jul 28 17:50:37.370814 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:37.370755    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"847a301a0a2a8407eb0d9865f28501e15509ec666c29b9ee768c842c296f8449"} err="failed to get container status \"847a301a0a2a8407eb0d9865f28501e15509ec666c29b9ee768c842c296f8449\": rpc error: code = NotFound desc = could not find container \"847a301a0a2a8407eb0d9865f28501e15509ec666c29b9ee768c842c296f8449\": container with ID starting with 847a301a0a2a8407eb0d9865f28501e15509ec666c29b9ee768c842c296f8449 not found: ID does not exist"
Jul 28 17:50:37.370814 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:37.370778    2543 scope.go:117] "RemoveContainer" containerID="eeed75de1af1b630a4170f41288ec810c43f30364c1114fb3762f62eefa93880"
Jul 28 17:50:37.371054 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:37.371033    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeed75de1af1b630a4170f41288ec810c43f30364c1114fb3762f62eefa93880"} err="failed to get container status \"eeed75de1af1b630a4170f41288ec810c43f30364c1114fb3762f62eefa93880\": rpc error: code = NotFound desc = could not find container \"eeed75de1af1b630a4170f41288ec810c43f30364c1114fb3762f62eefa93880\": container with ID starting with eeed75de1af1b630a4170f41288ec810c43f30364c1114fb3762f62eefa93880 not found: ID does not exist"
Jul 28 17:50:37.371104 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:37.371055    2543 scope.go:117] "RemoveContainer" containerID="847a301a0a2a8407eb0d9865f28501e15509ec666c29b9ee768c842c296f8449"
Jul 28 17:50:37.371307 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:37.371281    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"847a301a0a2a8407eb0d9865f28501e15509ec666c29b9ee768c842c296f8449"} err="failed to get container status \"847a301a0a2a8407eb0d9865f28501e15509ec666c29b9ee768c842c296f8449\": rpc error: code = NotFound desc = could not find container \"847a301a0a2a8407eb0d9865f28501e15509ec666c29b9ee768c842c296f8449\": container with ID starting with 847a301a0a2a8407eb0d9865f28501e15509ec666c29b9ee768c842c296f8449 not found: ID does not exist"
Jul 28 17:50:37.371739 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:37.371724    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"]
Jul 28 17:50:37.375659 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:37.375640    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ff6gq"]
Jul 28 17:50:39.152668 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:39.152634    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="156fe330-1e45-49fa-b552-3761d0c58935" path="/var/lib/kubelet/pods/156fe330-1e45-49fa-b552-3761d0c58935/volumes"
Jul 28 17:50:43.885695 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:43.885662    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f92pd2q" podUID="6ff28b47-9a34-4fd4-b850-0e097fa16217" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:50:51.193758 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:51.193716    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f92pd2q" podUID="6ff28b47-9a34-4fd4-b850-0e097fa16217" containerName="main" containerID="cri-o://29ba8d83743ed38ee4991f223b6b0fcc1e6793d8075c90db0fe094a58bffba3f" gracePeriod=45
Jul 28 17:50:51.423724 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:51.423688    2543 generic.go:356] "Generic (PLEG): container finished" podID="6ff28b47-9a34-4fd4-b850-0e097fa16217" containerID="29ba8d83743ed38ee4991f223b6b0fcc1e6793d8075c90db0fe094a58bffba3f" exitCode=0
Jul 28 17:50:51.423858 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:51.423752    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f92pd2q" event={"ID":"6ff28b47-9a34-4fd4-b850-0e097fa16217","Type":"ContainerDied","Data":"29ba8d83743ed38ee4991f223b6b0fcc1e6793d8075c90db0fe094a58bffba3f"}
Jul 28 17:50:51.450675 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:51.450624    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f92pd2q"
Jul 28 17:50:51.584840 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:51.584806    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6prp\" (UniqueName: \"kubernetes.io/projected/6ff28b47-9a34-4fd4-b850-0e097fa16217-kube-api-access-x6prp\") pod \"6ff28b47-9a34-4fd4-b850-0e097fa16217\" (UID: \"6ff28b47-9a34-4fd4-b850-0e097fa16217\") "
Jul 28 17:50:51.585003 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:51.584895    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ff28b47-9a34-4fd4-b850-0e097fa16217-tls-certs\") pod \"6ff28b47-9a34-4fd4-b850-0e097fa16217\" (UID: \"6ff28b47-9a34-4fd4-b850-0e097fa16217\") "
Jul 28 17:50:51.586868 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:51.586830    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ff28b47-9a34-4fd4-b850-0e097fa16217-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ff28b47-9a34-4fd4-b850-0e097fa16217" (UID: "6ff28b47-9a34-4fd4-b850-0e097fa16217"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:50:51.587001 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:51.586899    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff28b47-9a34-4fd4-b850-0e097fa16217-kube-api-access-x6prp" (OuterVolumeSpecName: "kube-api-access-x6prp") pod "6ff28b47-9a34-4fd4-b850-0e097fa16217" (UID: "6ff28b47-9a34-4fd4-b850-0e097fa16217"). InnerVolumeSpecName "kube-api-access-x6prp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:50:51.686009 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:51.685979    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ff28b47-9a34-4fd4-b850-0e097fa16217-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:50:51.686009 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:51.686005    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6prp\" (UniqueName: \"kubernetes.io/projected/6ff28b47-9a34-4fd4-b850-0e097fa16217-kube-api-access-x6prp\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:50:52.430019 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:52.429983    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f92pd2q"
Jul 28 17:50:52.430019 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:52.430017    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f92pd2q" event={"ID":"6ff28b47-9a34-4fd4-b850-0e097fa16217","Type":"ContainerDied","Data":"6445ab7b69dbd61f2f6181078e69561916ffe5cba9253cfaffee1a124c6ecfd3"}
Jul 28 17:50:52.430539 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:52.430067    2543 scope.go:117] "RemoveContainer" containerID="29ba8d83743ed38ee4991f223b6b0fcc1e6793d8075c90db0fe094a58bffba3f"
Jul 28 17:50:52.452213 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:52.452186    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f92pd2q"]
Jul 28 17:50:52.456610 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:52.456589    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f92pd2q"]
Jul 28 17:50:53.152424 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:50:53.152390    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ff28b47-9a34-4fd4-b850-0e097fa16217" path="/var/lib/kubelet/pods/6ff28b47-9a34-4fd4-b850-0e097fa16217/volumes"
Jul 28 17:51:12.365077 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.365043    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9"]
Jul 28 17:51:12.365556 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.365419    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="156fe330-1e45-49fa-b552-3761d0c58935" containerName="main"
Jul 28 17:51:12.365556 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.365432    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="156fe330-1e45-49fa-b552-3761d0c58935" containerName="main"
Jul 28 17:51:12.365556 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.365440    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="156fe330-1e45-49fa-b552-3761d0c58935" containerName="llm-d-routing-sidecar"
Jul 28 17:51:12.365556 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.365447    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="156fe330-1e45-49fa-b552-3761d0c58935" containerName="llm-d-routing-sidecar"
Jul 28 17:51:12.365556 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.365453    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ff28b47-9a34-4fd4-b850-0e097fa16217" containerName="main"
Jul 28 17:51:12.365556 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.365458    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ff28b47-9a34-4fd4-b850-0e097fa16217" containerName="main"
Jul 28 17:51:12.365556 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.365508    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="156fe330-1e45-49fa-b552-3761d0c58935" containerName="main"
Jul 28 17:51:12.365556 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.365517    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="156fe330-1e45-49fa-b552-3761d0c58935" containerName="llm-d-routing-sidecar"
Jul 28 17:51:12.365556 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.365525    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ff28b47-9a34-4fd4-b850-0e097fa16217" containerName="main"
Jul 28 17:51:12.370107 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.370080    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9"
Jul 28 17:51:12.374440 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.374413    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 28 17:51:12.374569 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.374453    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Jul 28 17:51:12.374569 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.374422    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Jul 28 17:51:12.375426 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.375404    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-5xhl9\""
Jul 28 17:51:12.382680 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.382658    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9"]
Jul 28 17:51:12.457400 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.457373    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae9c33e2-441e-427e-8142-3edd58532be6-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9\" (UID: \"ae9c33e2-441e-427e-8142-3edd58532be6\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9"
Jul 28 17:51:12.457558 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.457409    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae9c33e2-441e-427e-8142-3edd58532be6-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9\" (UID: \"ae9c33e2-441e-427e-8142-3edd58532be6\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9"
Jul 28 17:51:12.457558 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.457447    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae9c33e2-441e-427e-8142-3edd58532be6-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9\" (UID: \"ae9c33e2-441e-427e-8142-3edd58532be6\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9"
Jul 28 17:51:12.457558 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.457466    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae9c33e2-441e-427e-8142-3edd58532be6-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9\" (UID: \"ae9c33e2-441e-427e-8142-3edd58532be6\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9"
Jul 28 17:51:12.457558 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.457481    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn6vx\" (UniqueName: \"kubernetes.io/projected/ae9c33e2-441e-427e-8142-3edd58532be6-kube-api-access-kn6vx\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9\" (UID: \"ae9c33e2-441e-427e-8142-3edd58532be6\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9"
Jul 28 17:51:12.457558 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.457535    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae9c33e2-441e-427e-8142-3edd58532be6-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9\" (UID: \"ae9c33e2-441e-427e-8142-3edd58532be6\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9"
Jul 28 17:51:12.558800 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.558770    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae9c33e2-441e-427e-8142-3edd58532be6-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9\" (UID: \"ae9c33e2-441e-427e-8142-3edd58532be6\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9"
Jul 28 17:51:12.558800 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.558803    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae9c33e2-441e-427e-8142-3edd58532be6-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9\" (UID: \"ae9c33e2-441e-427e-8142-3edd58532be6\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9"
Jul 28 17:51:12.559047 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.558824    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn6vx\" (UniqueName: \"kubernetes.io/projected/ae9c33e2-441e-427e-8142-3edd58532be6-kube-api-access-kn6vx\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9\" (UID: \"ae9c33e2-441e-427e-8142-3edd58532be6\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9"
Jul 28 17:51:12.559047 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.558858    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae9c33e2-441e-427e-8142-3edd58532be6-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9\" (UID: \"ae9c33e2-441e-427e-8142-3edd58532be6\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9"
Jul 28 17:51:12.559047 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.558900    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae9c33e2-441e-427e-8142-3edd58532be6-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9\" (UID: \"ae9c33e2-441e-427e-8142-3edd58532be6\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9"
Jul 28 17:51:12.559047 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.558923    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae9c33e2-441e-427e-8142-3edd58532be6-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9\" (UID: \"ae9c33e2-441e-427e-8142-3edd58532be6\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9"
Jul 28 17:51:12.559288 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.559213    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae9c33e2-441e-427e-8142-3edd58532be6-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9\" (UID: \"ae9c33e2-441e-427e-8142-3edd58532be6\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9"
Jul 28 17:51:12.559288 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.559275    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae9c33e2-441e-427e-8142-3edd58532be6-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9\" (UID: \"ae9c33e2-441e-427e-8142-3edd58532be6\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9"
Jul 28 17:51:12.559401 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.559355    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae9c33e2-441e-427e-8142-3edd58532be6-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9\" (UID: \"ae9c33e2-441e-427e-8142-3edd58532be6\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9"
Jul 28 17:51:12.561327 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.561302    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae9c33e2-441e-427e-8142-3edd58532be6-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9\" (UID: \"ae9c33e2-441e-427e-8142-3edd58532be6\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9"
Jul 28 17:51:12.561462 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.561445    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae9c33e2-441e-427e-8142-3edd58532be6-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9\" (UID: \"ae9c33e2-441e-427e-8142-3edd58532be6\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9"
Jul 28 17:51:12.568689 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.568665    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn6vx\" (UniqueName: \"kubernetes.io/projected/ae9c33e2-441e-427e-8142-3edd58532be6-kube-api-access-kn6vx\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9\" (UID: \"ae9c33e2-441e-427e-8142-3edd58532be6\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9"
Jul 28 17:51:12.599449 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.599409    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedtqgnd"]
Jul 28 17:51:12.604072 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.604045    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedtqgnd"
Jul 28 17:51:12.606622 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.606602    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-mfjgj\""
Jul 28 17:51:12.611866 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.611839    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedtqgnd"]
Jul 28 17:51:12.659918 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.659828    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngm9n\" (UniqueName: \"kubernetes.io/projected/49a05f98-d633-4069-ba25-b9d632fb72e5-kube-api-access-ngm9n\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedtqgnd\" (UID: \"49a05f98-d633-4069-ba25-b9d632fb72e5\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedtqgnd"
Jul 28 17:51:12.659918 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.659872    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49a05f98-d633-4069-ba25-b9d632fb72e5-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedtqgnd\" (UID: \"49a05f98-d633-4069-ba25-b9d632fb72e5\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedtqgnd"
Jul 28 17:51:12.681580 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.681554    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9"
Jul 28 17:51:12.761013 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.760914    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngm9n\" (UniqueName: \"kubernetes.io/projected/49a05f98-d633-4069-ba25-b9d632fb72e5-kube-api-access-ngm9n\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedtqgnd\" (UID: \"49a05f98-d633-4069-ba25-b9d632fb72e5\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedtqgnd"
Jul 28 17:51:12.761013 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.760959    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49a05f98-d633-4069-ba25-b9d632fb72e5-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedtqgnd\" (UID: \"49a05f98-d633-4069-ba25-b9d632fb72e5\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedtqgnd"
Jul 28 17:51:12.763615 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.763590    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49a05f98-d633-4069-ba25-b9d632fb72e5-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedtqgnd\" (UID: \"49a05f98-d633-4069-ba25-b9d632fb72e5\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedtqgnd"
Jul 28 17:51:12.780581 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.780552    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngm9n\" (UniqueName: \"kubernetes.io/projected/49a05f98-d633-4069-ba25-b9d632fb72e5-kube-api-access-ngm9n\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedtqgnd\" (UID: \"49a05f98-d633-4069-ba25-b9d632fb72e5\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedtqgnd"
Jul 28 17:51:12.824229 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.824201    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9"]
Jul 28 17:51:12.826302 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:51:12.826277    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae9c33e2_441e_427e_8142_3edd58532be6.slice/crio-c38474c59b376625bab8e4eeb82ef2b7f26daa3ccfd135246f57909402d77d0c WatchSource:0}: Error finding container c38474c59b376625bab8e4eeb82ef2b7f26daa3ccfd135246f57909402d77d0c: Status 404 returned error can't find the container with id c38474c59b376625bab8e4eeb82ef2b7f26daa3ccfd135246f57909402d77d0c
Jul 28 17:51:12.916895 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:12.916827    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedtqgnd"
Jul 28 17:51:13.057385 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:13.057360    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedtqgnd"]
Jul 28 17:51:13.059272 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:51:13.059240    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a05f98_d633_4069_ba25_b9d632fb72e5.slice/crio-86ed315eddbd7eb5078f4852d207ce773e98c8fc395bc0b3e9a136ba0ba60339 WatchSource:0}: Error finding container 86ed315eddbd7eb5078f4852d207ce773e98c8fc395bc0b3e9a136ba0ba60339: Status 404 returned error can't find the container with id 86ed315eddbd7eb5078f4852d207ce773e98c8fc395bc0b3e9a136ba0ba60339
Jul 28 17:51:13.531050 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:13.531013    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedtqgnd" event={"ID":"49a05f98-d633-4069-ba25-b9d632fb72e5","Type":"ContainerStarted","Data":"a984a5063896324266de673d5982442b2c641a75f140d98cde9f21aa115c7cba"}
Jul 28 17:51:13.531517 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:13.531075    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedtqgnd" event={"ID":"49a05f98-d633-4069-ba25-b9d632fb72e5","Type":"ContainerStarted","Data":"86ed315eddbd7eb5078f4852d207ce773e98c8fc395bc0b3e9a136ba0ba60339"}
Jul 28 17:51:13.531517 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:13.531107    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedtqgnd"
Jul 28 17:51:13.532536 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:13.532515    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9" event={"ID":"ae9c33e2-441e-427e-8142-3edd58532be6","Type":"ContainerStarted","Data":"d288ece80a313a5008886a12a9ea9b07cc53e70fa310ff4aa21dc52c9af2a8e1"}
Jul 28 17:51:13.532591 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:13.532558    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9" event={"ID":"ae9c33e2-441e-427e-8142-3edd58532be6","Type":"ContainerStarted","Data":"c38474c59b376625bab8e4eeb82ef2b7f26daa3ccfd135246f57909402d77d0c"}
Jul 28 17:51:13.551313 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:13.551264    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedtqgnd" podStartSLOduration=1.551250541 podStartE2EDuration="1.551250541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:51:13.548725047 +0000 UTC m=+1676.954998787" watchObservedRunningTime="2026-07-28 17:51:13.551250541 +0000 UTC m=+1676.957524261"
Jul 28 17:51:13.570623 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:13.570580    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9" podStartSLOduration=1.5705675000000001 podStartE2EDuration="1.5705675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:51:13.568261731 +0000 UTC m=+1676.974535473" watchObservedRunningTime="2026-07-28 17:51:13.5705675 +0000 UTC m=+1676.976841240"
Jul 28 17:51:13.682078 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:13.682043    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9"
Jul 28 17:51:22.682097 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:22.682062    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9"
Jul 28 17:51:22.694670 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:22.694648    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9"
Jul 28 17:51:23.593750 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:23.593723    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9"
Jul 28 17:51:44.539171 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:44.539068    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedtqgnd"
Jul 28 17:51:46.206248 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:46.206216    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9"]
Jul 28 17:51:46.209564 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:46.209534    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedtqgnd"]
Jul 28 17:51:46.216192 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:51:46.216160    2543 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:51:46.216192 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:51:46.216192 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:51:46.216192 ip-10-0-135-177 kubenswrapper[2543]:  > containerID="d288ece80a313a5008886a12a9ea9b07cc53e70fa310ff4aa21dc52c9af2a8e1" cmd=["/bin/sleep","15"]
Jul 28 17:51:46.216417 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:46.216208    2543 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9" message="" err=<
Jul 28 17:51:46.216417 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:51:46.216417 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:51:46.216417 ip-10-0-135-177 kubenswrapper[2543]:  >
Jul 28 17:51:46.216417 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:51:46.216223    2543 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:51:46.216417 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:51:46.216417 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:51:46.216417 ip-10-0-135-177 kubenswrapper[2543]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9" podUID="ae9c33e2-441e-427e-8142-3edd58532be6" containerName="main" containerID="cri-o://d288ece80a313a5008886a12a9ea9b07cc53e70fa310ff4aa21dc52c9af2a8e1"
Jul 28 17:51:46.216417 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:46.216266    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9" podUID="ae9c33e2-441e-427e-8142-3edd58532be6" containerName="main" containerID="cri-o://d288ece80a313a5008886a12a9ea9b07cc53e70fa310ff4aa21dc52c9af2a8e1" gracePeriod=60
Jul 28 17:51:46.563018 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:46.562997    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9"
Jul 28 17:51:46.658972 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:46.658939    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae9c33e2-441e-427e-8142-3edd58532be6-home\") pod \"ae9c33e2-441e-427e-8142-3edd58532be6\" (UID: \"ae9c33e2-441e-427e-8142-3edd58532be6\") "
Jul 28 17:51:46.659169 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:46.658992    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae9c33e2-441e-427e-8142-3edd58532be6-tmp-dir\") pod \"ae9c33e2-441e-427e-8142-3edd58532be6\" (UID: \"ae9c33e2-441e-427e-8142-3edd58532be6\") "
Jul 28 17:51:46.659169 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:46.659028    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae9c33e2-441e-427e-8142-3edd58532be6-tls-certs\") pod \"ae9c33e2-441e-427e-8142-3edd58532be6\" (UID: \"ae9c33e2-441e-427e-8142-3edd58532be6\") "
Jul 28 17:51:46.659169 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:46.659071    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae9c33e2-441e-427e-8142-3edd58532be6-model-cache\") pod \"ae9c33e2-441e-427e-8142-3edd58532be6\" (UID: \"ae9c33e2-441e-427e-8142-3edd58532be6\") "
Jul 28 17:51:46.659169 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:46.659108    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kn6vx\" (UniqueName: \"kubernetes.io/projected/ae9c33e2-441e-427e-8142-3edd58532be6-kube-api-access-kn6vx\") pod \"ae9c33e2-441e-427e-8142-3edd58532be6\" (UID: \"ae9c33e2-441e-427e-8142-3edd58532be6\") "
Jul 28 17:51:46.659386 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:46.659173    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae9c33e2-441e-427e-8142-3edd58532be6-dshm\") pod \"ae9c33e2-441e-427e-8142-3edd58532be6\" (UID: \"ae9c33e2-441e-427e-8142-3edd58532be6\") "
Jul 28 17:51:46.659386 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:46.659273    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae9c33e2-441e-427e-8142-3edd58532be6-home" (OuterVolumeSpecName: "home") pod "ae9c33e2-441e-427e-8142-3edd58532be6" (UID: "ae9c33e2-441e-427e-8142-3edd58532be6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:51:46.659386 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:46.659291    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae9c33e2-441e-427e-8142-3edd58532be6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae9c33e2-441e-427e-8142-3edd58532be6" (UID: "ae9c33e2-441e-427e-8142-3edd58532be6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:51:46.659551 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:46.659383    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae9c33e2-441e-427e-8142-3edd58532be6-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae9c33e2-441e-427e-8142-3edd58532be6" (UID: "ae9c33e2-441e-427e-8142-3edd58532be6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:51:46.659551 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:46.659480    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae9c33e2-441e-427e-8142-3edd58532be6-home\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:51:46.659551 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:46.659492    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae9c33e2-441e-427e-8142-3edd58532be6-tmp-dir\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:51:46.659551 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:46.659501    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae9c33e2-441e-427e-8142-3edd58532be6-model-cache\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:51:46.661341 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:46.661318    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae9c33e2-441e-427e-8142-3edd58532be6-dshm" (OuterVolumeSpecName: "dshm") pod "ae9c33e2-441e-427e-8142-3edd58532be6" (UID: "ae9c33e2-441e-427e-8142-3edd58532be6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:51:46.661445 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:46.661333    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae9c33e2-441e-427e-8142-3edd58532be6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae9c33e2-441e-427e-8142-3edd58532be6" (UID: "ae9c33e2-441e-427e-8142-3edd58532be6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:51:46.661445 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:46.661396    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae9c33e2-441e-427e-8142-3edd58532be6-kube-api-access-kn6vx" (OuterVolumeSpecName: "kube-api-access-kn6vx") pod "ae9c33e2-441e-427e-8142-3edd58532be6" (UID: "ae9c33e2-441e-427e-8142-3edd58532be6"). InnerVolumeSpecName "kube-api-access-kn6vx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:51:46.697924 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:46.697898    2543 generic.go:356] "Generic (PLEG): container finished" podID="ae9c33e2-441e-427e-8142-3edd58532be6" containerID="d288ece80a313a5008886a12a9ea9b07cc53e70fa310ff4aa21dc52c9af2a8e1" exitCode=0
Jul 28 17:51:46.698037 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:46.697941    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9" event={"ID":"ae9c33e2-441e-427e-8142-3edd58532be6","Type":"ContainerDied","Data":"d288ece80a313a5008886a12a9ea9b07cc53e70fa310ff4aa21dc52c9af2a8e1"}
Jul 28 17:51:46.698037 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:46.697945    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9"
Jul 28 17:51:46.698037 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:46.697982    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9" event={"ID":"ae9c33e2-441e-427e-8142-3edd58532be6","Type":"ContainerDied","Data":"c38474c59b376625bab8e4eeb82ef2b7f26daa3ccfd135246f57909402d77d0c"}
Jul 28 17:51:46.698037 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:46.698005    2543 scope.go:117] "RemoveContainer" containerID="d288ece80a313a5008886a12a9ea9b07cc53e70fa310ff4aa21dc52c9af2a8e1"
Jul 28 17:51:46.706815 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:46.706800    2543 scope.go:117] "RemoveContainer" containerID="d288ece80a313a5008886a12a9ea9b07cc53e70fa310ff4aa21dc52c9af2a8e1"
Jul 28 17:51:46.707068 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:51:46.707048    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d288ece80a313a5008886a12a9ea9b07cc53e70fa310ff4aa21dc52c9af2a8e1\": container with ID starting with d288ece80a313a5008886a12a9ea9b07cc53e70fa310ff4aa21dc52c9af2a8e1 not found: ID does not exist" containerID="d288ece80a313a5008886a12a9ea9b07cc53e70fa310ff4aa21dc52c9af2a8e1"
Jul 28 17:51:46.707150 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:46.707075    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d288ece80a313a5008886a12a9ea9b07cc53e70fa310ff4aa21dc52c9af2a8e1"} err="failed to get container status \"d288ece80a313a5008886a12a9ea9b07cc53e70fa310ff4aa21dc52c9af2a8e1\": rpc error: code = NotFound desc = could not find container \"d288ece80a313a5008886a12a9ea9b07cc53e70fa310ff4aa21dc52c9af2a8e1\": container with ID starting with d288ece80a313a5008886a12a9ea9b07cc53e70fa310ff4aa21dc52c9af2a8e1 not found: ID does not exist"
Jul 28 17:51:46.724779 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:46.724754    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9"]
Jul 28 17:51:46.733971 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:46.733950    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-b58flmlb9"]
Jul 28 17:51:46.759892 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:46.759871    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae9c33e2-441e-427e-8142-3edd58532be6-dshm\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:51:46.759892 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:46.759890    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae9c33e2-441e-427e-8142-3edd58532be6-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:51:46.760023 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:46.759899    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn6vx\" (UniqueName: \"kubernetes.io/projected/ae9c33e2-441e-427e-8142-3edd58532be6-kube-api-access-kn6vx\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:51:47.152111 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:47.152033    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae9c33e2-441e-427e-8142-3edd58532be6" path="/var/lib/kubelet/pods/ae9c33e2-441e-427e-8142-3edd58532be6/volumes"
Jul 28 17:51:54.539360 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:51:54.539317    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedtqgnd" podUID="49a05f98-d633-4069-ba25-b9d632fb72e5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:52:01.227533 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:01.227472    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedtqgnd" podUID="49a05f98-d633-4069-ba25-b9d632fb72e5" containerName="main" containerID="cri-o://a984a5063896324266de673d5982442b2c641a75f140d98cde9f21aa115c7cba" gracePeriod=45
Jul 28 17:52:01.476344 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:01.476320    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedtqgnd"
Jul 28 17:52:01.585006 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:01.584973    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49a05f98-d633-4069-ba25-b9d632fb72e5-tls-certs\") pod \"49a05f98-d633-4069-ba25-b9d632fb72e5\" (UID: \"49a05f98-d633-4069-ba25-b9d632fb72e5\") "
Jul 28 17:52:01.585249 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:01.585043    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngm9n\" (UniqueName: \"kubernetes.io/projected/49a05f98-d633-4069-ba25-b9d632fb72e5-kube-api-access-ngm9n\") pod \"49a05f98-d633-4069-ba25-b9d632fb72e5\" (UID: \"49a05f98-d633-4069-ba25-b9d632fb72e5\") "
Jul 28 17:52:01.587159 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:01.587103    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a05f98-d633-4069-ba25-b9d632fb72e5-kube-api-access-ngm9n" (OuterVolumeSpecName: "kube-api-access-ngm9n") pod "49a05f98-d633-4069-ba25-b9d632fb72e5" (UID: "49a05f98-d633-4069-ba25-b9d632fb72e5"). InnerVolumeSpecName "kube-api-access-ngm9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:52:01.587288 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:01.587161    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49a05f98-d633-4069-ba25-b9d632fb72e5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "49a05f98-d633-4069-ba25-b9d632fb72e5" (UID: "49a05f98-d633-4069-ba25-b9d632fb72e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:52:01.685769 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:01.685734    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49a05f98-d633-4069-ba25-b9d632fb72e5-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:52:01.685769 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:01.685771    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngm9n\" (UniqueName: \"kubernetes.io/projected/49a05f98-d633-4069-ba25-b9d632fb72e5-kube-api-access-ngm9n\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:52:01.779550 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:01.779513    2543 generic.go:356] "Generic (PLEG): container finished" podID="49a05f98-d633-4069-ba25-b9d632fb72e5" containerID="a984a5063896324266de673d5982442b2c641a75f140d98cde9f21aa115c7cba" exitCode=0
Jul 28 17:52:01.779735 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:01.779649    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedtqgnd" event={"ID":"49a05f98-d633-4069-ba25-b9d632fb72e5","Type":"ContainerDied","Data":"a984a5063896324266de673d5982442b2c641a75f140d98cde9f21aa115c7cba"}
Jul 28 17:52:01.779796 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:01.779754    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedtqgnd" event={"ID":"49a05f98-d633-4069-ba25-b9d632fb72e5","Type":"ContainerDied","Data":"86ed315eddbd7eb5078f4852d207ce773e98c8fc395bc0b3e9a136ba0ba60339"}
Jul 28 17:52:01.779852 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:01.779799    2543 scope.go:117] "RemoveContainer" containerID="a984a5063896324266de673d5982442b2c641a75f140d98cde9f21aa115c7cba"
Jul 28 17:52:01.779991 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:01.779976    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedtqgnd"
Jul 28 17:52:01.789025 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:01.789006    2543 scope.go:117] "RemoveContainer" containerID="a984a5063896324266de673d5982442b2c641a75f140d98cde9f21aa115c7cba"
Jul 28 17:52:01.789302 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:52:01.789284    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a984a5063896324266de673d5982442b2c641a75f140d98cde9f21aa115c7cba\": container with ID starting with a984a5063896324266de673d5982442b2c641a75f140d98cde9f21aa115c7cba not found: ID does not exist" containerID="a984a5063896324266de673d5982442b2c641a75f140d98cde9f21aa115c7cba"
Jul 28 17:52:01.789365 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:01.789309    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a984a5063896324266de673d5982442b2c641a75f140d98cde9f21aa115c7cba"} err="failed to get container status \"a984a5063896324266de673d5982442b2c641a75f140d98cde9f21aa115c7cba\": rpc error: code = NotFound desc = could not find container \"a984a5063896324266de673d5982442b2c641a75f140d98cde9f21aa115c7cba\": container with ID starting with a984a5063896324266de673d5982442b2c641a75f140d98cde9f21aa115c7cba not found: ID does not exist"
Jul 28 17:52:01.801237 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:01.801213    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedtqgnd"]
Jul 28 17:52:01.807349 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:01.807328    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedtqgnd"]
Jul 28 17:52:03.151913 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:03.151869    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49a05f98-d633-4069-ba25-b9d632fb72e5" path="/var/lib/kubelet/pods/49a05f98-d633-4069-ba25-b9d632fb72e5/volumes"
Jul 28 17:52:17.715943 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.715911    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"]
Jul 28 17:52:17.716347 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.716241    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49a05f98-d633-4069-ba25-b9d632fb72e5" containerName="main"
Jul 28 17:52:17.716347 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.716253    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="49a05f98-d633-4069-ba25-b9d632fb72e5" containerName="main"
Jul 28 17:52:17.716347 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.716279    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae9c33e2-441e-427e-8142-3edd58532be6" containerName="main"
Jul 28 17:52:17.716347 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.716284    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae9c33e2-441e-427e-8142-3edd58532be6" containerName="main"
Jul 28 17:52:17.716347 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.716330    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae9c33e2-441e-427e-8142-3edd58532be6" containerName="main"
Jul 28 17:52:17.716347 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.716341    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="49a05f98-d633-4069-ba25-b9d632fb72e5" containerName="main"
Jul 28 17:52:17.721008 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.720987    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"
Jul 28 17:52:17.730327 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.730304    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Jul 28 17:52:17.730327 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.730325    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Jul 28 17:52:17.731229 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.731212    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-fx57c\""
Jul 28 17:52:17.731229 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.731214    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Jul 28 17:52:17.731346 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.731237    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Jul 28 17:52:17.753151 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.753079    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"]
Jul 28 17:52:17.809579 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.809551    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/409a42cc-81b4-4a3a-8580-08722c69bd22-home\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d\" (UID: \"409a42cc-81b4-4a3a-8580-08722c69bd22\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"
Jul 28 17:52:17.809751 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.809592    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/409a42cc-81b4-4a3a-8580-08722c69bd22-dshm\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d\" (UID: \"409a42cc-81b4-4a3a-8580-08722c69bd22\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"
Jul 28 17:52:17.809751 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.809622    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/409a42cc-81b4-4a3a-8580-08722c69bd22-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d\" (UID: \"409a42cc-81b4-4a3a-8580-08722c69bd22\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"
Jul 28 17:52:17.809751 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.809703    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/409a42cc-81b4-4a3a-8580-08722c69bd22-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d\" (UID: \"409a42cc-81b4-4a3a-8580-08722c69bd22\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"
Jul 28 17:52:17.809751 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.809740    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/409a42cc-81b4-4a3a-8580-08722c69bd22-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d\" (UID: \"409a42cc-81b4-4a3a-8580-08722c69bd22\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"
Jul 28 17:52:17.809891 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.809764    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rdqx\" (UniqueName: \"kubernetes.io/projected/409a42cc-81b4-4a3a-8580-08722c69bd22-kube-api-access-4rdqx\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d\" (UID: \"409a42cc-81b4-4a3a-8580-08722c69bd22\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"
Jul 28 17:52:17.809891 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.809798    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/409a42cc-81b4-4a3a-8580-08722c69bd22-model-cache\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d\" (UID: \"409a42cc-81b4-4a3a-8580-08722c69bd22\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"
Jul 28 17:52:17.911224 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.911191    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/409a42cc-81b4-4a3a-8580-08722c69bd22-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d\" (UID: \"409a42cc-81b4-4a3a-8580-08722c69bd22\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"
Jul 28 17:52:17.911224 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.911228    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/409a42cc-81b4-4a3a-8580-08722c69bd22-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d\" (UID: \"409a42cc-81b4-4a3a-8580-08722c69bd22\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"
Jul 28 17:52:17.911445 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.911249    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/409a42cc-81b4-4a3a-8580-08722c69bd22-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d\" (UID: \"409a42cc-81b4-4a3a-8580-08722c69bd22\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"
Jul 28 17:52:17.911445 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.911374    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rdqx\" (UniqueName: \"kubernetes.io/projected/409a42cc-81b4-4a3a-8580-08722c69bd22-kube-api-access-4rdqx\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d\" (UID: \"409a42cc-81b4-4a3a-8580-08722c69bd22\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"
Jul 28 17:52:17.911553 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.911454    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/409a42cc-81b4-4a3a-8580-08722c69bd22-model-cache\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d\" (UID: \"409a42cc-81b4-4a3a-8580-08722c69bd22\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"
Jul 28 17:52:17.911553 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.911520    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/409a42cc-81b4-4a3a-8580-08722c69bd22-home\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d\" (UID: \"409a42cc-81b4-4a3a-8580-08722c69bd22\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"
Jul 28 17:52:17.911660 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.911589    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/409a42cc-81b4-4a3a-8580-08722c69bd22-dshm\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d\" (UID: \"409a42cc-81b4-4a3a-8580-08722c69bd22\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"
Jul 28 17:52:17.911660 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.911636    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/409a42cc-81b4-4a3a-8580-08722c69bd22-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d\" (UID: \"409a42cc-81b4-4a3a-8580-08722c69bd22\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"
Jul 28 17:52:17.911758 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.911684    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/409a42cc-81b4-4a3a-8580-08722c69bd22-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d\" (UID: \"409a42cc-81b4-4a3a-8580-08722c69bd22\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"
Jul 28 17:52:17.911822 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.911801    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/409a42cc-81b4-4a3a-8580-08722c69bd22-model-cache\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d\" (UID: \"409a42cc-81b4-4a3a-8580-08722c69bd22\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"
Jul 28 17:52:17.911903 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.911878    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/409a42cc-81b4-4a3a-8580-08722c69bd22-home\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d\" (UID: \"409a42cc-81b4-4a3a-8580-08722c69bd22\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"
Jul 28 17:52:17.913563 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.913536    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/409a42cc-81b4-4a3a-8580-08722c69bd22-dshm\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d\" (UID: \"409a42cc-81b4-4a3a-8580-08722c69bd22\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"
Jul 28 17:52:17.913829 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.913808    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/409a42cc-81b4-4a3a-8580-08722c69bd22-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d\" (UID: \"409a42cc-81b4-4a3a-8580-08722c69bd22\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"
Jul 28 17:52:17.920886 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:17.920863    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rdqx\" (UniqueName: \"kubernetes.io/projected/409a42cc-81b4-4a3a-8580-08722c69bd22-kube-api-access-4rdqx\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d\" (UID: \"409a42cc-81b4-4a3a-8580-08722c69bd22\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"
Jul 28 17:52:18.031406 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:18.031333    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"
Jul 28 17:52:18.071328 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:18.071296    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx"]
Jul 28 17:52:18.075492 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:18.075466    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx"
Jul 28 17:52:18.098062 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:18.098032    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx"]
Jul 28 17:52:18.113866 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:18.113835    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e48c6ef-7d93-4882-b556-f00ad3832595-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx\" (UID: \"4e48c6ef-7d93-4882-b556-f00ad3832595\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx"
Jul 28 17:52:18.114001 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:18.113911    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e48c6ef-7d93-4882-b556-f00ad3832595-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx\" (UID: \"4e48c6ef-7d93-4882-b556-f00ad3832595\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx"
Jul 28 17:52:18.114001 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:18.113958    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e48c6ef-7d93-4882-b556-f00ad3832595-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx\" (UID: \"4e48c6ef-7d93-4882-b556-f00ad3832595\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx"
Jul 28 17:52:18.214671 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:18.214644    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e48c6ef-7d93-4882-b556-f00ad3832595-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx\" (UID: \"4e48c6ef-7d93-4882-b556-f00ad3832595\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx"
Jul 28 17:52:18.214781 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:18.214755    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e48c6ef-7d93-4882-b556-f00ad3832595-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx\" (UID: \"4e48c6ef-7d93-4882-b556-f00ad3832595\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx"
Jul 28 17:52:18.214863 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:18.214817    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e48c6ef-7d93-4882-b556-f00ad3832595-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx\" (UID: \"4e48c6ef-7d93-4882-b556-f00ad3832595\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx"
Jul 28 17:52:18.215305 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:18.215053    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e48c6ef-7d93-4882-b556-f00ad3832595-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx\" (UID: \"4e48c6ef-7d93-4882-b556-f00ad3832595\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx"
Jul 28 17:52:18.215305 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:18.215113    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e48c6ef-7d93-4882-b556-f00ad3832595-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx\" (UID: \"4e48c6ef-7d93-4882-b556-f00ad3832595\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx"
Jul 28 17:52:18.216069 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:18.216046    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"]
Jul 28 17:52:18.217461 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:52:18.217439    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod409a42cc_81b4_4a3a_8580_08722c69bd22.slice/crio-0985cd2ee3af26fe1c082c83b34efc07d277a25888072d9feaa4b646ee318ff6 WatchSource:0}: Error finding container 0985cd2ee3af26fe1c082c83b34efc07d277a25888072d9feaa4b646ee318ff6: Status 404 returned error can't find the container with id 0985cd2ee3af26fe1c082c83b34efc07d277a25888072d9feaa4b646ee318ff6
Jul 28 17:52:18.217646 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:18.217630    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e48c6ef-7d93-4882-b556-f00ad3832595-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx\" (UID: \"4e48c6ef-7d93-4882-b556-f00ad3832595\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx"
Jul 28 17:52:18.388803 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:18.388713    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx"
Jul 28 17:52:18.563476 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:52:18.563441    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e48c6ef_7d93_4882_b556_f00ad3832595.slice/crio-96c3b2959a66879f28647eb78c68075cf0763655d8097ea95f5441a5666c1c83 WatchSource:0}: Error finding container 96c3b2959a66879f28647eb78c68075cf0763655d8097ea95f5441a5666c1c83: Status 404 returned error can't find the container with id 96c3b2959a66879f28647eb78c68075cf0763655d8097ea95f5441a5666c1c83
Jul 28 17:52:18.567120 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:18.567095    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx"]
Jul 28 17:52:18.868072 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:18.868035    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx" event={"ID":"4e48c6ef-7d93-4882-b556-f00ad3832595","Type":"ContainerStarted","Data":"45be7804e0ef341d5b9b756a88eb40b0f88e6d4ea525bb7e1e19e5ef9fc0d6eb"}
Jul 28 17:52:18.868542 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:18.868098    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx" event={"ID":"4e48c6ef-7d93-4882-b556-f00ad3832595","Type":"ContainerStarted","Data":"96c3b2959a66879f28647eb78c68075cf0763655d8097ea95f5441a5666c1c83"}
Jul 28 17:52:18.869632 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:18.869607    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d" event={"ID":"409a42cc-81b4-4a3a-8580-08722c69bd22","Type":"ContainerStarted","Data":"f1b41a67d9d3feafca7df99654e9d5b0830a73040d31c800cba1291c09f287fc"}
Jul 28 17:52:18.869789 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:18.869656    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d" event={"ID":"409a42cc-81b4-4a3a-8580-08722c69bd22","Type":"ContainerStarted","Data":"0985cd2ee3af26fe1c082c83b34efc07d277a25888072d9feaa4b646ee318ff6"}
Jul 28 17:52:22.902039 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:22.901954    2543 generic.go:356] "Generic (PLEG): container finished" podID="4e48c6ef-7d93-4882-b556-f00ad3832595" containerID="45be7804e0ef341d5b9b756a88eb40b0f88e6d4ea525bb7e1e19e5ef9fc0d6eb" exitCode=0
Jul 28 17:52:22.902430 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:22.902041    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx" event={"ID":"4e48c6ef-7d93-4882-b556-f00ad3832595","Type":"ContainerDied","Data":"45be7804e0ef341d5b9b756a88eb40b0f88e6d4ea525bb7e1e19e5ef9fc0d6eb"}
Jul 28 17:52:22.903454 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:22.903438    2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:52:23.910241 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:23.910200    2543 generic.go:356] "Generic (PLEG): container finished" podID="409a42cc-81b4-4a3a-8580-08722c69bd22" containerID="f1b41a67d9d3feafca7df99654e9d5b0830a73040d31c800cba1291c09f287fc" exitCode=0
Jul 28 17:52:23.910730 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:23.910300    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d" event={"ID":"409a42cc-81b4-4a3a-8580-08722c69bd22","Type":"ContainerDied","Data":"f1b41a67d9d3feafca7df99654e9d5b0830a73040d31c800cba1291c09f287fc"}
Jul 28 17:52:24.919662 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:24.919620    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d" event={"ID":"409a42cc-81b4-4a3a-8580-08722c69bd22","Type":"ContainerStarted","Data":"26ca818ae80172f185318394af33ede9dff937c6a42552ea131cad3a31950421"}
Jul 28 17:52:24.947413 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:24.947360    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d" podStartSLOduration=3.217418982 podStartE2EDuration="7.94734559s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.729926608s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:52:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:52:24.946243702 +0000 UTC m=+1748.352517443" watchObservedRunningTime="2026-07-28 17:52:24.94734559 +0000 UTC m=+1748.353619330"
Jul 28 17:52:25.031517 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:25.031474    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"
Jul 28 17:52:28.032228 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:28.032192    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"
Jul 28 17:52:28.047671 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:28.047643    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"
Jul 28 17:52:28.952395 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:28.952367    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"
Jul 28 17:52:59.127844 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:59.127813    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx" event={"ID":"4e48c6ef-7d93-4882-b556-f00ad3832595","Type":"ContainerStarted","Data":"733c52941d58c06e8bd31224a8a63277d23aa5f989472c3e92da80e73a3fe17b"}
Jul 28 17:52:59.128316 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:59.128013    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx"
Jul 28 17:52:59.148080 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:52:59.148022    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx" podStartSLOduration=1.563550018 podStartE2EDuration="41.148003487s" totalImagesPullingTime="35.577640123s" totalInitContainerRuntime="4.006813346s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:52:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:52:59.145557695 +0000 UTC m=+1782.551831436" watchObservedRunningTime="2026-07-28 17:52:59.148003487 +0000 UTC m=+1782.554277229"
Jul 28 17:53:30.144706 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:30.144673    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx"
Jul 28 17:53:31.930963 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:31.930931    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx_4e48c6ef-7d93-4882-b556-f00ad3832595/main/0.log"
Jul 28 17:53:32.276257 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:32.276223    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx"]
Jul 28 17:53:32.276500 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:32.276474    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx" podUID="4e48c6ef-7d93-4882-b556-f00ad3832595" containerName="main" containerID="cri-o://733c52941d58c06e8bd31224a8a63277d23aa5f989472c3e92da80e73a3fe17b" gracePeriod=30
Jul 28 17:53:32.278442 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:32.278417    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"]
Jul 28 17:53:32.286999 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:53:32.286967    2543 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:53:32.286999 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:53:32.286999 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:53:32.286999 ip-10-0-135-177 kubenswrapper[2543]:  > containerID="26ca818ae80172f185318394af33ede9dff937c6a42552ea131cad3a31950421" cmd=["/bin/sleep","15"]
Jul 28 17:53:32.287327 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:32.287018    2543 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d" message="" err=<
Jul 28 17:53:32.287327 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:53:32.287327 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:53:32.287327 ip-10-0-135-177 kubenswrapper[2543]:  >
Jul 28 17:53:32.287327 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:53:32.287031    2543 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:53:32.287327 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:53:32.287327 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:53:32.287327 ip-10-0-135-177 kubenswrapper[2543]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d" podUID="409a42cc-81b4-4a3a-8580-08722c69bd22" containerName="main" containerID="cri-o://26ca818ae80172f185318394af33ede9dff937c6a42552ea131cad3a31950421"
Jul 28 17:53:32.287327 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:32.287073    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d" podUID="409a42cc-81b4-4a3a-8580-08722c69bd22" containerName="main" containerID="cri-o://26ca818ae80172f185318394af33ede9dff937c6a42552ea131cad3a31950421" gracePeriod=60
Jul 28 17:53:32.638194 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:32.638166    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"
Jul 28 17:53:32.805161 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:32.805109    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/409a42cc-81b4-4a3a-8580-08722c69bd22-tmp-dir\") pod \"409a42cc-81b4-4a3a-8580-08722c69bd22\" (UID: \"409a42cc-81b4-4a3a-8580-08722c69bd22\") "
Jul 28 17:53:32.805328 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:32.805194    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/409a42cc-81b4-4a3a-8580-08722c69bd22-model-cache\") pod \"409a42cc-81b4-4a3a-8580-08722c69bd22\" (UID: \"409a42cc-81b4-4a3a-8580-08722c69bd22\") "
Jul 28 17:53:32.805328 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:32.805261    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/409a42cc-81b4-4a3a-8580-08722c69bd22-tls-certs\") pod \"409a42cc-81b4-4a3a-8580-08722c69bd22\" (UID: \"409a42cc-81b4-4a3a-8580-08722c69bd22\") "
Jul 28 17:53:32.805328 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:32.805296    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/409a42cc-81b4-4a3a-8580-08722c69bd22-dshm\") pod \"409a42cc-81b4-4a3a-8580-08722c69bd22\" (UID: \"409a42cc-81b4-4a3a-8580-08722c69bd22\") "
Jul 28 17:53:32.805504 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:32.805345    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/409a42cc-81b4-4a3a-8580-08722c69bd22-home\") pod \"409a42cc-81b4-4a3a-8580-08722c69bd22\" (UID: \"409a42cc-81b4-4a3a-8580-08722c69bd22\") "
Jul 28 17:53:32.805504 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:32.805385    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rdqx\" (UniqueName: \"kubernetes.io/projected/409a42cc-81b4-4a3a-8580-08722c69bd22-kube-api-access-4rdqx\") pod \"409a42cc-81b4-4a3a-8580-08722c69bd22\" (UID: \"409a42cc-81b4-4a3a-8580-08722c69bd22\") "
Jul 28 17:53:32.805504 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:32.805400    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409a42cc-81b4-4a3a-8580-08722c69bd22-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "409a42cc-81b4-4a3a-8580-08722c69bd22" (UID: "409a42cc-81b4-4a3a-8580-08722c69bd22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:53:32.805504 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:32.805428    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/409a42cc-81b4-4a3a-8580-08722c69bd22-kserve-provision-location\") pod \"409a42cc-81b4-4a3a-8580-08722c69bd22\" (UID: \"409a42cc-81b4-4a3a-8580-08722c69bd22\") "
Jul 28 17:53:32.805504 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:32.805469    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409a42cc-81b4-4a3a-8580-08722c69bd22-model-cache" (OuterVolumeSpecName: "model-cache") pod "409a42cc-81b4-4a3a-8580-08722c69bd22" (UID: "409a42cc-81b4-4a3a-8580-08722c69bd22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:53:32.805780 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:32.805759    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/409a42cc-81b4-4a3a-8580-08722c69bd22-tmp-dir\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:53:32.805848 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:32.805786    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/409a42cc-81b4-4a3a-8580-08722c69bd22-model-cache\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:53:32.805848 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:32.805782    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409a42cc-81b4-4a3a-8580-08722c69bd22-home" (OuterVolumeSpecName: "home") pod "409a42cc-81b4-4a3a-8580-08722c69bd22" (UID: "409a42cc-81b4-4a3a-8580-08722c69bd22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:53:32.807656 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:32.807591    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409a42cc-81b4-4a3a-8580-08722c69bd22-dshm" (OuterVolumeSpecName: "dshm") pod "409a42cc-81b4-4a3a-8580-08722c69bd22" (UID: "409a42cc-81b4-4a3a-8580-08722c69bd22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:53:32.807656 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:32.807641    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/409a42cc-81b4-4a3a-8580-08722c69bd22-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "409a42cc-81b4-4a3a-8580-08722c69bd22" (UID: "409a42cc-81b4-4a3a-8580-08722c69bd22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:53:32.807792 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:32.807721    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409a42cc-81b4-4a3a-8580-08722c69bd22-kube-api-access-4rdqx" (OuterVolumeSpecName: "kube-api-access-4rdqx") pod "409a42cc-81b4-4a3a-8580-08722c69bd22" (UID: "409a42cc-81b4-4a3a-8580-08722c69bd22"). InnerVolumeSpecName "kube-api-access-4rdqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:53:32.879101 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:32.879062    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409a42cc-81b4-4a3a-8580-08722c69bd22-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "409a42cc-81b4-4a3a-8580-08722c69bd22" (UID: "409a42cc-81b4-4a3a-8580-08722c69bd22"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:53:32.906808 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:32.906783    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/409a42cc-81b4-4a3a-8580-08722c69bd22-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:53:32.906808 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:32.906805    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/409a42cc-81b4-4a3a-8580-08722c69bd22-dshm\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:53:32.906808 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:32.906813    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/409a42cc-81b4-4a3a-8580-08722c69bd22-home\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:53:32.907008 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:32.906821    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rdqx\" (UniqueName: \"kubernetes.io/projected/409a42cc-81b4-4a3a-8580-08722c69bd22-kube-api-access-4rdqx\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:53:32.907008 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:32.906831    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/409a42cc-81b4-4a3a-8580-08722c69bd22-kserve-provision-location\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:53:32.925578 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:32.925557    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx"
Jul 28 17:53:33.108372 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.108285    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e48c6ef-7d93-4882-b556-f00ad3832595-tls-certs\") pod \"4e48c6ef-7d93-4882-b556-f00ad3832595\" (UID: \"4e48c6ef-7d93-4882-b556-f00ad3832595\") "
Jul 28 17:53:33.108372 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.108330    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e48c6ef-7d93-4882-b556-f00ad3832595-kserve-provision-location\") pod \"4e48c6ef-7d93-4882-b556-f00ad3832595\" (UID: \"4e48c6ef-7d93-4882-b556-f00ad3832595\") "
Jul 28 17:53:33.108372 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.108355    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e48c6ef-7d93-4882-b556-f00ad3832595-tmp-dir\") pod \"4e48c6ef-7d93-4882-b556-f00ad3832595\" (UID: \"4e48c6ef-7d93-4882-b556-f00ad3832595\") "
Jul 28 17:53:33.108948 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.108739    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e48c6ef-7d93-4882-b556-f00ad3832595-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e48c6ef-7d93-4882-b556-f00ad3832595" (UID: "4e48c6ef-7d93-4882-b556-f00ad3832595"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:53:33.110408 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.110389    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e48c6ef-7d93-4882-b556-f00ad3832595-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e48c6ef-7d93-4882-b556-f00ad3832595" (UID: "4e48c6ef-7d93-4882-b556-f00ad3832595"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:53:33.166087 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.166058    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e48c6ef-7d93-4882-b556-f00ad3832595-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4e48c6ef-7d93-4882-b556-f00ad3832595" (UID: "4e48c6ef-7d93-4882-b556-f00ad3832595"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:53:33.210046 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.210021    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e48c6ef-7d93-4882-b556-f00ad3832595-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:53:33.210046 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.210047    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e48c6ef-7d93-4882-b556-f00ad3832595-kserve-provision-location\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:53:33.210220 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.210056    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e48c6ef-7d93-4882-b556-f00ad3832595-tmp-dir\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:53:33.313402 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.313376    2543 generic.go:356] "Generic (PLEG): container finished" podID="4e48c6ef-7d93-4882-b556-f00ad3832595" containerID="733c52941d58c06e8bd31224a8a63277d23aa5f989472c3e92da80e73a3fe17b" exitCode=0
Jul 28 17:53:33.313579 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.313435    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx"
Jul 28 17:53:33.313579 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.313465    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx" event={"ID":"4e48c6ef-7d93-4882-b556-f00ad3832595","Type":"ContainerDied","Data":"733c52941d58c06e8bd31224a8a63277d23aa5f989472c3e92da80e73a3fe17b"}
Jul 28 17:53:33.313579 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.313520    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx" event={"ID":"4e48c6ef-7d93-4882-b556-f00ad3832595","Type":"ContainerDied","Data":"96c3b2959a66879f28647eb78c68075cf0763655d8097ea95f5441a5666c1c83"}
Jul 28 17:53:33.313579 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.313553    2543 scope.go:117] "RemoveContainer" containerID="733c52941d58c06e8bd31224a8a63277d23aa5f989472c3e92da80e73a3fe17b"
Jul 28 17:53:33.315331 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.315307    2543 generic.go:356] "Generic (PLEG): container finished" podID="409a42cc-81b4-4a3a-8580-08722c69bd22" containerID="26ca818ae80172f185318394af33ede9dff937c6a42552ea131cad3a31950421" exitCode=0
Jul 28 17:53:33.315453 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.315332    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d" event={"ID":"409a42cc-81b4-4a3a-8580-08722c69bd22","Type":"ContainerDied","Data":"26ca818ae80172f185318394af33ede9dff937c6a42552ea131cad3a31950421"}
Jul 28 17:53:33.315453 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.315361    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"
Jul 28 17:53:33.315453 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.315379    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d" event={"ID":"409a42cc-81b4-4a3a-8580-08722c69bd22","Type":"ContainerDied","Data":"0985cd2ee3af26fe1c082c83b34efc07d277a25888072d9feaa4b646ee318ff6"}
Jul 28 17:53:33.322915 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.322900    2543 scope.go:117] "RemoveContainer" containerID="45be7804e0ef341d5b9b756a88eb40b0f88e6d4ea525bb7e1e19e5ef9fc0d6eb"
Jul 28 17:53:33.331952 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.331933    2543 scope.go:117] "RemoveContainer" containerID="733c52941d58c06e8bd31224a8a63277d23aa5f989472c3e92da80e73a3fe17b"
Jul 28 17:53:33.332251 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:53:33.332227    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"733c52941d58c06e8bd31224a8a63277d23aa5f989472c3e92da80e73a3fe17b\": container with ID starting with 733c52941d58c06e8bd31224a8a63277d23aa5f989472c3e92da80e73a3fe17b not found: ID does not exist" containerID="733c52941d58c06e8bd31224a8a63277d23aa5f989472c3e92da80e73a3fe17b"
Jul 28 17:53:33.332331 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.332259    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"733c52941d58c06e8bd31224a8a63277d23aa5f989472c3e92da80e73a3fe17b"} err="failed to get container status \"733c52941d58c06e8bd31224a8a63277d23aa5f989472c3e92da80e73a3fe17b\": rpc error: code = NotFound desc = could not find container \"733c52941d58c06e8bd31224a8a63277d23aa5f989472c3e92da80e73a3fe17b\": container with ID starting with 733c52941d58c06e8bd31224a8a63277d23aa5f989472c3e92da80e73a3fe17b not found: ID does not exist"
Jul 28 17:53:33.332331 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.332284    2543 scope.go:117] "RemoveContainer" containerID="45be7804e0ef341d5b9b756a88eb40b0f88e6d4ea525bb7e1e19e5ef9fc0d6eb"
Jul 28 17:53:33.332500 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:53:33.332484    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45be7804e0ef341d5b9b756a88eb40b0f88e6d4ea525bb7e1e19e5ef9fc0d6eb\": container with ID starting with 45be7804e0ef341d5b9b756a88eb40b0f88e6d4ea525bb7e1e19e5ef9fc0d6eb not found: ID does not exist" containerID="45be7804e0ef341d5b9b756a88eb40b0f88e6d4ea525bb7e1e19e5ef9fc0d6eb"
Jul 28 17:53:33.332541 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.332506    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45be7804e0ef341d5b9b756a88eb40b0f88e6d4ea525bb7e1e19e5ef9fc0d6eb"} err="failed to get container status \"45be7804e0ef341d5b9b756a88eb40b0f88e6d4ea525bb7e1e19e5ef9fc0d6eb\": rpc error: code = NotFound desc = could not find container \"45be7804e0ef341d5b9b756a88eb40b0f88e6d4ea525bb7e1e19e5ef9fc0d6eb\": container with ID starting with 45be7804e0ef341d5b9b756a88eb40b0f88e6d4ea525bb7e1e19e5ef9fc0d6eb not found: ID does not exist"
Jul 28 17:53:33.332541 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.332519    2543 scope.go:117] "RemoveContainer" containerID="26ca818ae80172f185318394af33ede9dff937c6a42552ea131cad3a31950421"
Jul 28 17:53:33.332977 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.332956    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"]
Jul 28 17:53:33.336015 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.335992    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-8kn6d"]
Jul 28 17:53:33.340720 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.340695    2543 scope.go:117] "RemoveContainer" containerID="f1b41a67d9d3feafca7df99654e9d5b0830a73040d31c800cba1291c09f287fc"
Jul 28 17:53:33.347223 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.347199    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx"]
Jul 28 17:53:33.349997 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.349976    2543 scope.go:117] "RemoveContainer" containerID="26ca818ae80172f185318394af33ede9dff937c6a42552ea131cad3a31950421"
Jul 28 17:53:33.350311 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:53:33.350293    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26ca818ae80172f185318394af33ede9dff937c6a42552ea131cad3a31950421\": container with ID starting with 26ca818ae80172f185318394af33ede9dff937c6a42552ea131cad3a31950421 not found: ID does not exist" containerID="26ca818ae80172f185318394af33ede9dff937c6a42552ea131cad3a31950421"
Jul 28 17:53:33.350370 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.350319    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26ca818ae80172f185318394af33ede9dff937c6a42552ea131cad3a31950421"} err="failed to get container status \"26ca818ae80172f185318394af33ede9dff937c6a42552ea131cad3a31950421\": rpc error: code = NotFound desc = could not find container \"26ca818ae80172f185318394af33ede9dff937c6a42552ea131cad3a31950421\": container with ID starting with 26ca818ae80172f185318394af33ede9dff937c6a42552ea131cad3a31950421 not found: ID does not exist"
Jul 28 17:53:33.350370 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.350335    2543 scope.go:117] "RemoveContainer" containerID="f1b41a67d9d3feafca7df99654e9d5b0830a73040d31c800cba1291c09f287fc"
Jul 28 17:53:33.350581 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:53:33.350566    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1b41a67d9d3feafca7df99654e9d5b0830a73040d31c800cba1291c09f287fc\": container with ID starting with f1b41a67d9d3feafca7df99654e9d5b0830a73040d31c800cba1291c09f287fc not found: ID does not exist" containerID="f1b41a67d9d3feafca7df99654e9d5b0830a73040d31c800cba1291c09f287fc"
Jul 28 17:53:33.350626 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.350584    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1b41a67d9d3feafca7df99654e9d5b0830a73040d31c800cba1291c09f287fc"} err="failed to get container status \"f1b41a67d9d3feafca7df99654e9d5b0830a73040d31c800cba1291c09f287fc\": rpc error: code = NotFound desc = could not find container \"f1b41a67d9d3feafca7df99654e9d5b0830a73040d31c800cba1291c09f287fc\": container with ID starting with f1b41a67d9d3feafca7df99654e9d5b0830a73040d31c800cba1291c09f287fc not found: ID does not exist"
Jul 28 17:53:33.351318 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:33.351302    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-66tvx"]
Jul 28 17:53:35.153535 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:35.153504    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="409a42cc-81b4-4a3a-8580-08722c69bd22" path="/var/lib/kubelet/pods/409a42cc-81b4-4a3a-8580-08722c69bd22/volumes"
Jul 28 17:53:35.153923 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:35.153913    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e48c6ef-7d93-4882-b556-f00ad3832595" path="/var/lib/kubelet/pods/4e48c6ef-7d93-4882-b556-f00ad3832595/volumes"
Jul 28 17:53:57.259186 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.259151    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"]
Jul 28 17:53:57.259588 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.259481    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e48c6ef-7d93-4882-b556-f00ad3832595" containerName="storage-initializer"
Jul 28 17:53:57.259588 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.259491    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e48c6ef-7d93-4882-b556-f00ad3832595" containerName="storage-initializer"
Jul 28 17:53:57.259588 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.259501    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e48c6ef-7d93-4882-b556-f00ad3832595" containerName="main"
Jul 28 17:53:57.259588 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.259507    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e48c6ef-7d93-4882-b556-f00ad3832595" containerName="main"
Jul 28 17:53:57.259588 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.259517    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="409a42cc-81b4-4a3a-8580-08722c69bd22" containerName="storage-initializer"
Jul 28 17:53:57.259588 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.259522    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="409a42cc-81b4-4a3a-8580-08722c69bd22" containerName="storage-initializer"
Jul 28 17:53:57.259588 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.259536    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="409a42cc-81b4-4a3a-8580-08722c69bd22" containerName="main"
Jul 28 17:53:57.259588 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.259542    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="409a42cc-81b4-4a3a-8580-08722c69bd22" containerName="main"
Jul 28 17:53:57.259849 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.259606    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="409a42cc-81b4-4a3a-8580-08722c69bd22" containerName="main"
Jul 28 17:53:57.259849 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.259612    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e48c6ef-7d93-4882-b556-f00ad3832595" containerName="main"
Jul 28 17:53:57.261413 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.261397    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"
Jul 28 17:53:57.263729 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.263708    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Jul 28 17:53:57.264436 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.264421    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Jul 28 17:53:57.264540 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.264525    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-lpvmq\""
Jul 28 17:53:57.264593 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.264529    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Jul 28 17:53:57.265733 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.265714    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Jul 28 17:53:57.277312 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.277288    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"]
Jul 28 17:53:57.288590 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.288558    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d115270-e811-4224-b762-6bc2fd829799-dshm\") pod \"tokenizer-migration-path-test-kserve-6666787596-qn5x2\" (UID: \"0d115270-e811-4224-b762-6bc2fd829799\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"
Jul 28 17:53:57.288590 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.288587    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d115270-e811-4224-b762-6bc2fd829799-home\") pod \"tokenizer-migration-path-test-kserve-6666787596-qn5x2\" (UID: \"0d115270-e811-4224-b762-6bc2fd829799\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"
Jul 28 17:53:57.288769 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.288610    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d115270-e811-4224-b762-6bc2fd829799-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6666787596-qn5x2\" (UID: \"0d115270-e811-4224-b762-6bc2fd829799\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"
Jul 28 17:53:57.288769 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.288686    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj842\" (UniqueName: \"kubernetes.io/projected/0d115270-e811-4224-b762-6bc2fd829799-kube-api-access-zj842\") pod \"tokenizer-migration-path-test-kserve-6666787596-qn5x2\" (UID: \"0d115270-e811-4224-b762-6bc2fd829799\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"
Jul 28 17:53:57.288769 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.288715    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d115270-e811-4224-b762-6bc2fd829799-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6666787596-qn5x2\" (UID: \"0d115270-e811-4224-b762-6bc2fd829799\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"
Jul 28 17:53:57.288769 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.288747    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d115270-e811-4224-b762-6bc2fd829799-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6666787596-qn5x2\" (UID: \"0d115270-e811-4224-b762-6bc2fd829799\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"
Jul 28 17:53:57.288998 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.288784    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d115270-e811-4224-b762-6bc2fd829799-model-cache\") pod \"tokenizer-migration-path-test-kserve-6666787596-qn5x2\" (UID: \"0d115270-e811-4224-b762-6bc2fd829799\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"
Jul 28 17:53:57.389768 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.389737    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj842\" (UniqueName: \"kubernetes.io/projected/0d115270-e811-4224-b762-6bc2fd829799-kube-api-access-zj842\") pod \"tokenizer-migration-path-test-kserve-6666787596-qn5x2\" (UID: \"0d115270-e811-4224-b762-6bc2fd829799\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"
Jul 28 17:53:57.389768 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.389769    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d115270-e811-4224-b762-6bc2fd829799-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6666787596-qn5x2\" (UID: \"0d115270-e811-4224-b762-6bc2fd829799\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"
Jul 28 17:53:57.389989 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.389795    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d115270-e811-4224-b762-6bc2fd829799-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6666787596-qn5x2\" (UID: \"0d115270-e811-4224-b762-6bc2fd829799\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"
Jul 28 17:53:57.389989 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.389833    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d115270-e811-4224-b762-6bc2fd829799-model-cache\") pod \"tokenizer-migration-path-test-kserve-6666787596-qn5x2\" (UID: \"0d115270-e811-4224-b762-6bc2fd829799\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"
Jul 28 17:53:57.389989 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.389873    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d115270-e811-4224-b762-6bc2fd829799-dshm\") pod \"tokenizer-migration-path-test-kserve-6666787596-qn5x2\" (UID: \"0d115270-e811-4224-b762-6bc2fd829799\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"
Jul 28 17:53:57.390167 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.389997    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d115270-e811-4224-b762-6bc2fd829799-home\") pod \"tokenizer-migration-path-test-kserve-6666787596-qn5x2\" (UID: \"0d115270-e811-4224-b762-6bc2fd829799\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"
Jul 28 17:53:57.390167 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.390040    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d115270-e811-4224-b762-6bc2fd829799-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6666787596-qn5x2\" (UID: \"0d115270-e811-4224-b762-6bc2fd829799\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"
Jul 28 17:53:57.390422 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.390401    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d115270-e811-4224-b762-6bc2fd829799-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6666787596-qn5x2\" (UID: \"0d115270-e811-4224-b762-6bc2fd829799\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"
Jul 28 17:53:57.390493 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.390415    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d115270-e811-4224-b762-6bc2fd829799-model-cache\") pod \"tokenizer-migration-path-test-kserve-6666787596-qn5x2\" (UID: \"0d115270-e811-4224-b762-6bc2fd829799\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"
Jul 28 17:53:57.390536 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.390501    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d115270-e811-4224-b762-6bc2fd829799-home\") pod \"tokenizer-migration-path-test-kserve-6666787596-qn5x2\" (UID: \"0d115270-e811-4224-b762-6bc2fd829799\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"
Jul 28 17:53:57.390536 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.390515    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d115270-e811-4224-b762-6bc2fd829799-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6666787596-qn5x2\" (UID: \"0d115270-e811-4224-b762-6bc2fd829799\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"
Jul 28 17:53:57.392084 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.392066    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d115270-e811-4224-b762-6bc2fd829799-dshm\") pod \"tokenizer-migration-path-test-kserve-6666787596-qn5x2\" (UID: \"0d115270-e811-4224-b762-6bc2fd829799\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"
Jul 28 17:53:57.392390 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.392373    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d115270-e811-4224-b762-6bc2fd829799-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6666787596-qn5x2\" (UID: \"0d115270-e811-4224-b762-6bc2fd829799\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"
Jul 28 17:53:57.398614 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.398591    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj842\" (UniqueName: \"kubernetes.io/projected/0d115270-e811-4224-b762-6bc2fd829799-kube-api-access-zj842\") pod \"tokenizer-migration-path-test-kserve-6666787596-qn5x2\" (UID: \"0d115270-e811-4224-b762-6bc2fd829799\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"
Jul 28 17:53:57.557335 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.557258    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw"]
Jul 28 17:53:57.559739 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.559723    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw"
Jul 28 17:53:57.572240 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.572222    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"
Jul 28 17:53:57.591163 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.591109    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/192443aa-a6d6-4bb0-9d46-99d465899ce6-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw\" (UID: \"192443aa-a6d6-4bb0-9d46-99d465899ce6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw"
Jul 28 17:53:57.591403 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.591375    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/192443aa-a6d6-4bb0-9d46-99d465899ce6-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw\" (UID: \"192443aa-a6d6-4bb0-9d46-99d465899ce6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw"
Jul 28 17:53:57.591582 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.591423    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/192443aa-a6d6-4bb0-9d46-99d465899ce6-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw\" (UID: \"192443aa-a6d6-4bb0-9d46-99d465899ce6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw"
Jul 28 17:53:57.596499 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.596474    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw"]
Jul 28 17:53:57.692747 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.692716    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/192443aa-a6d6-4bb0-9d46-99d465899ce6-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw\" (UID: \"192443aa-a6d6-4bb0-9d46-99d465899ce6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw"
Jul 28 17:53:57.692901 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.692771    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/192443aa-a6d6-4bb0-9d46-99d465899ce6-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw\" (UID: \"192443aa-a6d6-4bb0-9d46-99d465899ce6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw"
Jul 28 17:53:57.692901 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.692793    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/192443aa-a6d6-4bb0-9d46-99d465899ce6-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw\" (UID: \"192443aa-a6d6-4bb0-9d46-99d465899ce6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw"
Jul 28 17:53:57.693095 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.693078    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/192443aa-a6d6-4bb0-9d46-99d465899ce6-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw\" (UID: \"192443aa-a6d6-4bb0-9d46-99d465899ce6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw"
Jul 28 17:53:57.693162 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.693105    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/192443aa-a6d6-4bb0-9d46-99d465899ce6-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw\" (UID: \"192443aa-a6d6-4bb0-9d46-99d465899ce6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw"
Jul 28 17:53:57.695148 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.695112    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/192443aa-a6d6-4bb0-9d46-99d465899ce6-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw\" (UID: \"192443aa-a6d6-4bb0-9d46-99d465899ce6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw"
Jul 28 17:53:57.719017 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.718996    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"]
Jul 28 17:53:57.719855 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:53:57.719832    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d115270_e811_4224_b762_6bc2fd829799.slice/crio-e6a2cb6f85b51d39c1ec66360b21ec03f6c7526e76fe1be7ede1c83ab52caecf WatchSource:0}: Error finding container e6a2cb6f85b51d39c1ec66360b21ec03f6c7526e76fe1be7ede1c83ab52caecf: Status 404 returned error can't find the container with id e6a2cb6f85b51d39c1ec66360b21ec03f6c7526e76fe1be7ede1c83ab52caecf
Jul 28 17:53:57.870728 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:57.870625    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw"
Jul 28 17:53:58.009511 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:58.009491    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw"]
Jul 28 17:53:58.011768 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:53:58.011744    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod192443aa_a6d6_4bb0_9d46_99d465899ce6.slice/crio-0cef85996e287db0ce3aa8473cefc99fb7e35220a72327a00c20fc028fea6006 WatchSource:0}: Error finding container 0cef85996e287db0ce3aa8473cefc99fb7e35220a72327a00c20fc028fea6006: Status 404 returned error can't find the container with id 0cef85996e287db0ce3aa8473cefc99fb7e35220a72327a00c20fc028fea6006
Jul 28 17:53:58.447258 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:58.447205    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2" event={"ID":"0d115270-e811-4224-b762-6bc2fd829799","Type":"ContainerStarted","Data":"20921bfcd3595a6224c0af7d382db888f1fde52bb395868e7c12a7a20debd527"}
Jul 28 17:53:58.447759 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:58.447269    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2" event={"ID":"0d115270-e811-4224-b762-6bc2fd829799","Type":"ContainerStarted","Data":"e6a2cb6f85b51d39c1ec66360b21ec03f6c7526e76fe1be7ede1c83ab52caecf"}
Jul 28 17:53:58.448889 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:58.448865    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw" event={"ID":"192443aa-a6d6-4bb0-9d46-99d465899ce6","Type":"ContainerStarted","Data":"33b806f61a0c5db35fe82ab9b042c58f41578f22a04fc10069cecaa56521d6e2"}
Jul 28 17:53:58.449031 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:53:58.448904    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw" event={"ID":"192443aa-a6d6-4bb0-9d46-99d465899ce6","Type":"ContainerStarted","Data":"0cef85996e287db0ce3aa8473cefc99fb7e35220a72327a00c20fc028fea6006"}
Jul 28 17:54:03.480998 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:03.480960    2543 generic.go:356] "Generic (PLEG): container finished" podID="192443aa-a6d6-4bb0-9d46-99d465899ce6" containerID="33b806f61a0c5db35fe82ab9b042c58f41578f22a04fc10069cecaa56521d6e2" exitCode=0
Jul 28 17:54:03.481416 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:03.481034    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw" event={"ID":"192443aa-a6d6-4bb0-9d46-99d465899ce6","Type":"ContainerDied","Data":"33b806f61a0c5db35fe82ab9b042c58f41578f22a04fc10069cecaa56521d6e2"}
Jul 28 17:54:03.482735 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:03.482711    2543 generic.go:356] "Generic (PLEG): container finished" podID="0d115270-e811-4224-b762-6bc2fd829799" containerID="20921bfcd3595a6224c0af7d382db888f1fde52bb395868e7c12a7a20debd527" exitCode=0
Jul 28 17:54:03.482833 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:03.482775    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2" event={"ID":"0d115270-e811-4224-b762-6bc2fd829799","Type":"ContainerDied","Data":"20921bfcd3595a6224c0af7d382db888f1fde52bb395868e7c12a7a20debd527"}
Jul 28 17:54:04.489868 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:04.489834    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw" event={"ID":"192443aa-a6d6-4bb0-9d46-99d465899ce6","Type":"ContainerStarted","Data":"5a30a8e2b78f988e66682f0c09840e498bcdab5ec0b0f3d0f22e8892f8a2a68f"}
Jul 28 17:54:04.490343 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:04.490079    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw"
Jul 28 17:54:04.491804 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:04.491783    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2" event={"ID":"0d115270-e811-4224-b762-6bc2fd829799","Type":"ContainerStarted","Data":"1198d08474119b4ebf14be2fd009a8df04ba4905cb25b4853631b5a12d91569d"}
Jul 28 17:54:04.509034 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:04.508964    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw" podStartSLOduration=3.0773436690000002 podStartE2EDuration="7.50895114s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.431607471s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:54:04.50703609 +0000 UTC m=+1847.913309830" watchObservedRunningTime="2026-07-28 17:54:04.50895114 +0000 UTC m=+1847.915224881"
Jul 28 17:54:04.525853 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:04.525813    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2" podStartSLOduration=2.616643018 podStartE2EDuration="7.525801117s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.909158099s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:54:04.524872195 +0000 UTC m=+1847.931145947" watchObservedRunningTime="2026-07-28 17:54:04.525801117 +0000 UTC m=+1847.932074858"
Jul 28 17:54:04.573308 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:04.573284    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"
Jul 28 17:54:07.573115 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:07.573075    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"
Jul 28 17:54:07.585570 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:07.585548    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"
Jul 28 17:54:08.522614 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:08.522577    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"
Jul 28 17:54:35.509122 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:35.509087    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw"
Jul 28 17:54:37.054529 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.054498    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw_192443aa-a6d6-4bb0-9d46-99d465899ce6/main/0.log"
Jul 28 17:54:37.235336 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.235304    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"]
Jul 28 17:54:37.240390 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.240358    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw"]
Jul 28 17:54:37.240602 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.240577    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw" podUID="192443aa-a6d6-4bb0-9d46-99d465899ce6" containerName="main" containerID="cri-o://5a30a8e2b78f988e66682f0c09840e498bcdab5ec0b0f3d0f22e8892f8a2a68f" gracePeriod=30
Jul 28 17:54:37.245793 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:54:37.245756    2543 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:54:37.245793 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:54:37.245793 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:54:37.245793 ip-10-0-135-177 kubenswrapper[2543]:  > containerID="1198d08474119b4ebf14be2fd009a8df04ba4905cb25b4853631b5a12d91569d" cmd=["/bin/sleep","15"]
Jul 28 17:54:37.246083 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.245817    2543 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2" message="" err=<
Jul 28 17:54:37.246083 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:54:37.246083 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:54:37.246083 ip-10-0-135-177 kubenswrapper[2543]:  >
Jul 28 17:54:37.246083 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:54:37.245845    2543 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:54:37.246083 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:54:37.246083 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:54:37.246083 ip-10-0-135-177 kubenswrapper[2543]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2" podUID="0d115270-e811-4224-b762-6bc2fd829799" containerName="main" containerID="cri-o://1198d08474119b4ebf14be2fd009a8df04ba4905cb25b4853631b5a12d91569d"
Jul 28 17:54:37.246083 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.245911    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2" podUID="0d115270-e811-4224-b762-6bc2fd829799" containerName="main" containerID="cri-o://1198d08474119b4ebf14be2fd009a8df04ba4905cb25b4853631b5a12d91569d" gracePeriod=60
Jul 28 17:54:37.511595 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.511561    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2" podUID="0d115270-e811-4224-b762-6bc2fd829799" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 28 17:54:37.676918 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.676888    2543 generic.go:356] "Generic (PLEG): container finished" podID="0d115270-e811-4224-b762-6bc2fd829799" containerID="1198d08474119b4ebf14be2fd009a8df04ba4905cb25b4853631b5a12d91569d" exitCode=0
Jul 28 17:54:37.677059 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.676926    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2" event={"ID":"0d115270-e811-4224-b762-6bc2fd829799","Type":"ContainerDied","Data":"1198d08474119b4ebf14be2fd009a8df04ba4905cb25b4853631b5a12d91569d"}
Jul 28 17:54:37.677059 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.676982    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2" event={"ID":"0d115270-e811-4224-b762-6bc2fd829799","Type":"ContainerDied","Data":"e6a2cb6f85b51d39c1ec66360b21ec03f6c7526e76fe1be7ede1c83ab52caecf"}
Jul 28 17:54:37.677059 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.677003    2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6a2cb6f85b51d39c1ec66360b21ec03f6c7526e76fe1be7ede1c83ab52caecf"
Jul 28 17:54:37.678943 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.678920    2543 generic.go:356] "Generic (PLEG): container finished" podID="192443aa-a6d6-4bb0-9d46-99d465899ce6" containerID="5a30a8e2b78f988e66682f0c09840e498bcdab5ec0b0f3d0f22e8892f8a2a68f" exitCode=0
Jul 28 17:54:37.679047 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.678951    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw" event={"ID":"192443aa-a6d6-4bb0-9d46-99d465899ce6","Type":"ContainerDied","Data":"5a30a8e2b78f988e66682f0c09840e498bcdab5ec0b0f3d0f22e8892f8a2a68f"}
Jul 28 17:54:37.680646 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.680627    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"
Jul 28 17:54:37.781770 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.781749    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw"
Jul 28 17:54:37.846985 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.846957    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d115270-e811-4224-b762-6bc2fd829799-model-cache\") pod \"0d115270-e811-4224-b762-6bc2fd829799\" (UID: \"0d115270-e811-4224-b762-6bc2fd829799\") "
Jul 28 17:54:37.847184 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.847032    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d115270-e811-4224-b762-6bc2fd829799-dshm\") pod \"0d115270-e811-4224-b762-6bc2fd829799\" (UID: \"0d115270-e811-4224-b762-6bc2fd829799\") "
Jul 28 17:54:37.847184 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.847058    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zj842\" (UniqueName: \"kubernetes.io/projected/0d115270-e811-4224-b762-6bc2fd829799-kube-api-access-zj842\") pod \"0d115270-e811-4224-b762-6bc2fd829799\" (UID: \"0d115270-e811-4224-b762-6bc2fd829799\") "
Jul 28 17:54:37.847184 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.847094    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d115270-e811-4224-b762-6bc2fd829799-tls-certs\") pod \"0d115270-e811-4224-b762-6bc2fd829799\" (UID: \"0d115270-e811-4224-b762-6bc2fd829799\") "
Jul 28 17:54:37.847184 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.847118    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d115270-e811-4224-b762-6bc2fd829799-kserve-provision-location\") pod \"0d115270-e811-4224-b762-6bc2fd829799\" (UID: \"0d115270-e811-4224-b762-6bc2fd829799\") "
Jul 28 17:54:37.847415 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.847267    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d115270-e811-4224-b762-6bc2fd829799-model-cache" (OuterVolumeSpecName: "model-cache") pod "0d115270-e811-4224-b762-6bc2fd829799" (UID: "0d115270-e811-4224-b762-6bc2fd829799"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:54:37.847415 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.847306    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d115270-e811-4224-b762-6bc2fd829799-home\") pod \"0d115270-e811-4224-b762-6bc2fd829799\" (UID: \"0d115270-e811-4224-b762-6bc2fd829799\") "
Jul 28 17:54:37.847415 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.847347    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d115270-e811-4224-b762-6bc2fd829799-tmp-dir\") pod \"0d115270-e811-4224-b762-6bc2fd829799\" (UID: \"0d115270-e811-4224-b762-6bc2fd829799\") "
Jul 28 17:54:37.847631 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.847568    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d115270-e811-4224-b762-6bc2fd829799-home" (OuterVolumeSpecName: "home") pod "0d115270-e811-4224-b762-6bc2fd829799" (UID: "0d115270-e811-4224-b762-6bc2fd829799"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:54:37.847689 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.847654    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d115270-e811-4224-b762-6bc2fd829799-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0d115270-e811-4224-b762-6bc2fd829799" (UID: "0d115270-e811-4224-b762-6bc2fd829799"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:54:37.847748 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.847681    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d115270-e811-4224-b762-6bc2fd829799-home\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:54:37.847748 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.847701    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d115270-e811-4224-b762-6bc2fd829799-model-cache\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:54:37.849416 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.849393    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d115270-e811-4224-b762-6bc2fd829799-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0d115270-e811-4224-b762-6bc2fd829799" (UID: "0d115270-e811-4224-b762-6bc2fd829799"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:54:37.849519 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.849433    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d115270-e811-4224-b762-6bc2fd829799-dshm" (OuterVolumeSpecName: "dshm") pod "0d115270-e811-4224-b762-6bc2fd829799" (UID: "0d115270-e811-4224-b762-6bc2fd829799"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:54:37.849563 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.849532    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d115270-e811-4224-b762-6bc2fd829799-kube-api-access-zj842" (OuterVolumeSpecName: "kube-api-access-zj842") pod "0d115270-e811-4224-b762-6bc2fd829799" (UID: "0d115270-e811-4224-b762-6bc2fd829799"). InnerVolumeSpecName "kube-api-access-zj842". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:54:37.906582 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.906506    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d115270-e811-4224-b762-6bc2fd829799-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0d115270-e811-4224-b762-6bc2fd829799" (UID: "0d115270-e811-4224-b762-6bc2fd829799"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:54:37.948232 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.948208    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/192443aa-a6d6-4bb0-9d46-99d465899ce6-tmp-dir\") pod \"192443aa-a6d6-4bb0-9d46-99d465899ce6\" (UID: \"192443aa-a6d6-4bb0-9d46-99d465899ce6\") "
Jul 28 17:54:37.948382 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.948241    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/192443aa-a6d6-4bb0-9d46-99d465899ce6-tls-certs\") pod \"192443aa-a6d6-4bb0-9d46-99d465899ce6\" (UID: \"192443aa-a6d6-4bb0-9d46-99d465899ce6\") "
Jul 28 17:54:37.948382 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.948336    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/192443aa-a6d6-4bb0-9d46-99d465899ce6-kserve-provision-location\") pod \"192443aa-a6d6-4bb0-9d46-99d465899ce6\" (UID: \"192443aa-a6d6-4bb0-9d46-99d465899ce6\") "
Jul 28 17:54:37.948601 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.948583    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192443aa-a6d6-4bb0-9d46-99d465899ce6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "192443aa-a6d6-4bb0-9d46-99d465899ce6" (UID: "192443aa-a6d6-4bb0-9d46-99d465899ce6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:54:37.948881 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.948543    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d115270-e811-4224-b762-6bc2fd829799-dshm\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:54:37.948965 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.948903    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj842\" (UniqueName: \"kubernetes.io/projected/0d115270-e811-4224-b762-6bc2fd829799-kube-api-access-zj842\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:54:37.955499 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.949069    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d115270-e811-4224-b762-6bc2fd829799-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:54:37.955499 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.949090    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d115270-e811-4224-b762-6bc2fd829799-kserve-provision-location\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:54:37.955499 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.949115    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d115270-e811-4224-b762-6bc2fd829799-tmp-dir\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:54:37.955499 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:37.951254    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/192443aa-a6d6-4bb0-9d46-99d465899ce6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "192443aa-a6d6-4bb0-9d46-99d465899ce6" (UID: "192443aa-a6d6-4bb0-9d46-99d465899ce6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:54:38.008363 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:38.008324    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192443aa-a6d6-4bb0-9d46-99d465899ce6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "192443aa-a6d6-4bb0-9d46-99d465899ce6" (UID: "192443aa-a6d6-4bb0-9d46-99d465899ce6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:54:38.050504 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:38.050473    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/192443aa-a6d6-4bb0-9d46-99d465899ce6-kserve-provision-location\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:54:38.050504 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:38.050499    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/192443aa-a6d6-4bb0-9d46-99d465899ce6-tmp-dir\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:54:38.050504 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:38.050510    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/192443aa-a6d6-4bb0-9d46-99d465899ce6-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:54:38.685938 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:38.685834    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw"
Jul 28 17:54:38.686402 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:38.685838    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw" event={"ID":"192443aa-a6d6-4bb0-9d46-99d465899ce6","Type":"ContainerDied","Data":"0cef85996e287db0ce3aa8473cefc99fb7e35220a72327a00c20fc028fea6006"}
Jul 28 17:54:38.686402 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:38.686002    2543 scope.go:117] "RemoveContainer" containerID="5a30a8e2b78f988e66682f0c09840e498bcdab5ec0b0f3d0f22e8892f8a2a68f"
Jul 28 17:54:38.686402 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:38.686072    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"
Jul 28 17:54:38.695383 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:38.695361    2543 scope.go:117] "RemoveContainer" containerID="33b806f61a0c5db35fe82ab9b042c58f41578f22a04fc10069cecaa56521d6e2"
Jul 28 17:54:38.712392 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:38.712364    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"]
Jul 28 17:54:38.718349 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:38.718325    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-qn5x2"]
Jul 28 17:54:38.729242 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:38.729217    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw"]
Jul 28 17:54:38.734145 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:38.734112    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-wbxvw"]
Jul 28 17:54:39.152147 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:39.152088    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d115270-e811-4224-b762-6bc2fd829799" path="/var/lib/kubelet/pods/0d115270-e811-4224-b762-6bc2fd829799/volumes"
Jul 28 17:54:39.152601 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:54:39.152580    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="192443aa-a6d6-4bb0-9d46-99d465899ce6" path="/var/lib/kubelet/pods/192443aa-a6d6-4bb0-9d46-99d465899ce6/volumes"
Jul 28 17:55:02.828327 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.828292    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7"]
Jul 28 17:55:02.828761 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.828605    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="192443aa-a6d6-4bb0-9d46-99d465899ce6" containerName="main"
Jul 28 17:55:02.828761 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.828616    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="192443aa-a6d6-4bb0-9d46-99d465899ce6" containerName="main"
Jul 28 17:55:02.828761 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.828626    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d115270-e811-4224-b762-6bc2fd829799" containerName="storage-initializer"
Jul 28 17:55:02.828761 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.828631    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d115270-e811-4224-b762-6bc2fd829799" containerName="storage-initializer"
Jul 28 17:55:02.828761 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.828641    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="192443aa-a6d6-4bb0-9d46-99d465899ce6" containerName="storage-initializer"
Jul 28 17:55:02.828761 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.828647    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="192443aa-a6d6-4bb0-9d46-99d465899ce6" containerName="storage-initializer"
Jul 28 17:55:02.828761 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.828653    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d115270-e811-4224-b762-6bc2fd829799" containerName="main"
Jul 28 17:55:02.828761 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.828659    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d115270-e811-4224-b762-6bc2fd829799" containerName="main"
Jul 28 17:55:02.828761 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.828716    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d115270-e811-4224-b762-6bc2fd829799" containerName="main"
Jul 28 17:55:02.828761 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.828726    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="192443aa-a6d6-4bb0-9d46-99d465899ce6" containerName="main"
Jul 28 17:55:02.830655 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.830637    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7"
Jul 28 17:55:02.834010 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.833983    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 28 17:55:02.834124 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.833995    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Jul 28 17:55:02.835114 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.835088    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Jul 28 17:55:02.835266 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.835175    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-6p8gb\""
Jul 28 17:55:02.844790 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.844771    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7"]
Jul 28 17:55:02.848789 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.848770    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmrrb\" (UniqueName: \"kubernetes.io/projected/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-kube-api-access-kmrrb\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7\" (UID: \"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7"
Jul 28 17:55:02.848906 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.848807    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7\" (UID: \"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7"
Jul 28 17:55:02.848906 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.848841    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7\" (UID: \"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7"
Jul 28 17:55:02.848906 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.848879    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7\" (UID: \"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7"
Jul 28 17:55:02.849082 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.848961    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7\" (UID: \"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7"
Jul 28 17:55:02.849082 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.849031    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7\" (UID: \"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7"
Jul 28 17:55:02.949390 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.949356    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7\" (UID: \"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7"
Jul 28 17:55:02.949557 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.949419    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7\" (UID: \"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7"
Jul 28 17:55:02.949557 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.949449    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7\" (UID: \"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7"
Jul 28 17:55:02.949557 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.949490    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmrrb\" (UniqueName: \"kubernetes.io/projected/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-kube-api-access-kmrrb\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7\" (UID: \"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7"
Jul 28 17:55:02.949557 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.949520    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7\" (UID: \"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7"
Jul 28 17:55:02.949557 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.949549    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7\" (UID: \"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7"
Jul 28 17:55:02.949849 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.949769    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7\" (UID: \"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7"
Jul 28 17:55:02.949849 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.949811    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7\" (UID: \"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7"
Jul 28 17:55:02.949948 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.949854    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7\" (UID: \"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7"
Jul 28 17:55:02.951673 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.951646    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7\" (UID: \"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7"
Jul 28 17:55:02.951844 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.951827    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7\" (UID: \"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7"
Jul 28 17:55:02.957665 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:02.957644    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmrrb\" (UniqueName: \"kubernetes.io/projected/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-kube-api-access-kmrrb\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7\" (UID: \"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7"
Jul 28 17:55:03.140679 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:03.140590    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7"
Jul 28 17:55:03.277983 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:03.277958    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7"]
Jul 28 17:55:03.280100 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:55:03.280073    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode30b156b_fcd3_4dbf_abd0_ea40fda22f8d.slice/crio-9f78c4eddcc5a2966a44320c63833c5af1de3d92ecc65e315bd3fae5ce45fc04 WatchSource:0}: Error finding container 9f78c4eddcc5a2966a44320c63833c5af1de3d92ecc65e315bd3fae5ce45fc04: Status 404 returned error can't find the container with id 9f78c4eddcc5a2966a44320c63833c5af1de3d92ecc65e315bd3fae5ce45fc04
Jul 28 17:55:03.808869 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:03.808836    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7" event={"ID":"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d","Type":"ContainerStarted","Data":"4e8468fa8e860dc9eaa77a0a27c4343c3eb93f21d3dc0227d11957596b5f270d"}
Jul 28 17:55:03.809044 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:03.808884    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7" event={"ID":"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d","Type":"ContainerStarted","Data":"9f78c4eddcc5a2966a44320c63833c5af1de3d92ecc65e315bd3fae5ce45fc04"}
Jul 28 17:55:03.833601 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:03.833547    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7" podStartSLOduration=1.833530819 podStartE2EDuration="1.833530819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:55:03.830122142 +0000 UTC m=+1907.236395876" watchObservedRunningTime="2026-07-28 17:55:03.833530819 +0000 UTC m=+1907.239804560"
Jul 28 17:55:04.141221 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:04.141124    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7"
Jul 28 17:55:13.141634 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:13.141591    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7"
Jul 28 17:55:13.154780 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:13.154754    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7"
Jul 28 17:55:13.871002 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:13.870969    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7"
Jul 28 17:55:36.229895 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:36.229861    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7"]
Jul 28 17:55:36.239588 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:55:36.239553    2543 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:55:36.239588 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:55:36.239588 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:55:36.239588 ip-10-0-135-177 kubenswrapper[2543]:  > containerID="4e8468fa8e860dc9eaa77a0a27c4343c3eb93f21d3dc0227d11957596b5f270d" cmd=["/bin/sleep","15"]
Jul 28 17:55:36.239905 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:36.239608    2543 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7" message="" err=<
Jul 28 17:55:36.239905 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:55:36.239905 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:55:36.239905 ip-10-0-135-177 kubenswrapper[2543]:  >
Jul 28 17:55:36.239905 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:55:36.239624    2543 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:55:36.239905 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:55:36.239905 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:55:36.239905 ip-10-0-135-177 kubenswrapper[2543]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7" podUID="e30b156b-fcd3-4dbf-abd0-ea40fda22f8d" containerName="main" containerID="cri-o://4e8468fa8e860dc9eaa77a0a27c4343c3eb93f21d3dc0227d11957596b5f270d"
Jul 28 17:55:36.239905 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:36.239676    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7" podUID="e30b156b-fcd3-4dbf-abd0-ea40fda22f8d" containerName="main" containerID="cri-o://4e8468fa8e860dc9eaa77a0a27c4343c3eb93f21d3dc0227d11957596b5f270d" gracePeriod=60
Jul 28 17:55:36.583943 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:36.583918    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7"
Jul 28 17:55:36.746982 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:36.746947    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmrrb\" (UniqueName: \"kubernetes.io/projected/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-kube-api-access-kmrrb\") pod \"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d\" (UID: \"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d\") "
Jul 28 17:55:36.746982 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:36.746987    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-tmp-dir\") pod \"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d\" (UID: \"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d\") "
Jul 28 17:55:36.747322 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:36.747015    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-tls-certs\") pod \"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d\" (UID: \"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d\") "
Jul 28 17:55:36.747322 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:36.747064    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-dshm\") pod \"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d\" (UID: \"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d\") "
Jul 28 17:55:36.747322 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:36.747087    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-home\") pod \"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d\" (UID: \"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d\") "
Jul 28 17:55:36.747322 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:36.747161    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-model-cache\") pod \"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d\" (UID: \"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d\") "
Jul 28 17:55:36.747322 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:36.747297    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e30b156b-fcd3-4dbf-abd0-ea40fda22f8d" (UID: "e30b156b-fcd3-4dbf-abd0-ea40fda22f8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:55:36.747580 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:36.747409    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-home" (OuterVolumeSpecName: "home") pod "e30b156b-fcd3-4dbf-abd0-ea40fda22f8d" (UID: "e30b156b-fcd3-4dbf-abd0-ea40fda22f8d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:55:36.747580 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:36.747460    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-tmp-dir\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:55:36.747580 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:36.747532    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-model-cache" (OuterVolumeSpecName: "model-cache") pod "e30b156b-fcd3-4dbf-abd0-ea40fda22f8d" (UID: "e30b156b-fcd3-4dbf-abd0-ea40fda22f8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:55:36.749359 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:36.749335    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-dshm" (OuterVolumeSpecName: "dshm") pod "e30b156b-fcd3-4dbf-abd0-ea40fda22f8d" (UID: "e30b156b-fcd3-4dbf-abd0-ea40fda22f8d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:55:36.749476 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:36.749454    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-kube-api-access-kmrrb" (OuterVolumeSpecName: "kube-api-access-kmrrb") pod "e30b156b-fcd3-4dbf-abd0-ea40fda22f8d" (UID: "e30b156b-fcd3-4dbf-abd0-ea40fda22f8d"). InnerVolumeSpecName "kube-api-access-kmrrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:55:36.749528 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:36.749473    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e30b156b-fcd3-4dbf-abd0-ea40fda22f8d" (UID: "e30b156b-fcd3-4dbf-abd0-ea40fda22f8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:55:36.848260 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:36.848224    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmrrb\" (UniqueName: \"kubernetes.io/projected/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-kube-api-access-kmrrb\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:55:36.848260 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:36.848254    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:55:36.848260 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:36.848263    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-dshm\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:55:36.848502 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:36.848271    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-home\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:55:36.848502 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:36.848280    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d-model-cache\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:55:36.977438 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:36.977405    2543 generic.go:356] "Generic (PLEG): container finished" podID="e30b156b-fcd3-4dbf-abd0-ea40fda22f8d" containerID="4e8468fa8e860dc9eaa77a0a27c4343c3eb93f21d3dc0227d11957596b5f270d" exitCode=0
Jul 28 17:55:36.977595 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:36.977450    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7"
Jul 28 17:55:36.977595 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:36.977486    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7" event={"ID":"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d","Type":"ContainerDied","Data":"4e8468fa8e860dc9eaa77a0a27c4343c3eb93f21d3dc0227d11957596b5f270d"}
Jul 28 17:55:36.977595 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:36.977534    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7" event={"ID":"e30b156b-fcd3-4dbf-abd0-ea40fda22f8d","Type":"ContainerDied","Data":"9f78c4eddcc5a2966a44320c63833c5af1de3d92ecc65e315bd3fae5ce45fc04"}
Jul 28 17:55:36.977595 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:36.977559    2543 scope.go:117] "RemoveContainer" containerID="4e8468fa8e860dc9eaa77a0a27c4343c3eb93f21d3dc0227d11957596b5f270d"
Jul 28 17:55:36.989103 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:36.989085    2543 scope.go:117] "RemoveContainer" containerID="4e8468fa8e860dc9eaa77a0a27c4343c3eb93f21d3dc0227d11957596b5f270d"
Jul 28 17:55:36.989405 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:55:36.989386    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e8468fa8e860dc9eaa77a0a27c4343c3eb93f21d3dc0227d11957596b5f270d\": container with ID starting with 4e8468fa8e860dc9eaa77a0a27c4343c3eb93f21d3dc0227d11957596b5f270d not found: ID does not exist" containerID="4e8468fa8e860dc9eaa77a0a27c4343c3eb93f21d3dc0227d11957596b5f270d"
Jul 28 17:55:36.989484 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:36.989411    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e8468fa8e860dc9eaa77a0a27c4343c3eb93f21d3dc0227d11957596b5f270d"} err="failed to get container status \"4e8468fa8e860dc9eaa77a0a27c4343c3eb93f21d3dc0227d11957596b5f270d\": rpc error: code = NotFound desc = could not find container \"4e8468fa8e860dc9eaa77a0a27c4343c3eb93f21d3dc0227d11957596b5f270d\": container with ID starting with 4e8468fa8e860dc9eaa77a0a27c4343c3eb93f21d3dc0227d11957596b5f270d not found: ID does not exist"
Jul 28 17:55:37.001161 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:37.001121    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7"]
Jul 28 17:55:37.005000 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:37.004948    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-8q2p7"]
Jul 28 17:55:37.153280 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:55:37.153247    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e30b156b-fcd3-4dbf-abd0-ea40fda22f8d" path="/var/lib/kubelet/pods/e30b156b-fcd3-4dbf-abd0-ea40fda22f8d/volumes"
Jul 28 17:56:04.722555 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:04.722520    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52"]
Jul 28 17:56:04.723065 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:04.722852    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e30b156b-fcd3-4dbf-abd0-ea40fda22f8d" containerName="main"
Jul 28 17:56:04.723065 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:04.722864    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="e30b156b-fcd3-4dbf-abd0-ea40fda22f8d" containerName="main"
Jul 28 17:56:04.723065 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:04.722921    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="e30b156b-fcd3-4dbf-abd0-ea40fda22f8d" containerName="main"
Jul 28 17:56:04.724909 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:04.724894    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52"
Jul 28 17:56:04.728480 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:04.728449    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Jul 28 17:56:04.728480 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:04.728468    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-z284l\""
Jul 28 17:56:04.728672 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:04.728451    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 28 17:56:04.728672 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:04.728475    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Jul 28 17:56:04.737055 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:04.737032    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52"]
Jul 28 17:56:04.766968 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:04.766929    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6c5f3d4-531f-4966-817e-b57fc0d211a5-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52\" (UID: \"b6c5f3d4-531f-4966-817e-b57fc0d211a5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52"
Jul 28 17:56:04.767094 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:04.767026    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6c5f3d4-531f-4966-817e-b57fc0d211a5-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52\" (UID: \"b6c5f3d4-531f-4966-817e-b57fc0d211a5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52"
Jul 28 17:56:04.767094 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:04.767054    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6c5f3d4-531f-4966-817e-b57fc0d211a5-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52\" (UID: \"b6c5f3d4-531f-4966-817e-b57fc0d211a5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52"
Jul 28 17:56:04.767197 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:04.767096    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6c5f3d4-531f-4966-817e-b57fc0d211a5-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52\" (UID: \"b6c5f3d4-531f-4966-817e-b57fc0d211a5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52"
Jul 28 17:56:04.767197 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:04.767119    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6c5f3d4-531f-4966-817e-b57fc0d211a5-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52\" (UID: \"b6c5f3d4-531f-4966-817e-b57fc0d211a5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52"
Jul 28 17:56:04.767269 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:04.767192    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57k4b\" (UniqueName: \"kubernetes.io/projected/b6c5f3d4-531f-4966-817e-b57fc0d211a5-kube-api-access-57k4b\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52\" (UID: \"b6c5f3d4-531f-4966-817e-b57fc0d211a5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52"
Jul 28 17:56:04.868189 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:04.868156    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6c5f3d4-531f-4966-817e-b57fc0d211a5-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52\" (UID: \"b6c5f3d4-531f-4966-817e-b57fc0d211a5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52"
Jul 28 17:56:04.868189 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:04.868191    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6c5f3d4-531f-4966-817e-b57fc0d211a5-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52\" (UID: \"b6c5f3d4-531f-4966-817e-b57fc0d211a5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52"
Jul 28 17:56:04.868384 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:04.868214    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6c5f3d4-531f-4966-817e-b57fc0d211a5-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52\" (UID: \"b6c5f3d4-531f-4966-817e-b57fc0d211a5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52"
Jul 28 17:56:04.868384 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:04.868325    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6c5f3d4-531f-4966-817e-b57fc0d211a5-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52\" (UID: \"b6c5f3d4-531f-4966-817e-b57fc0d211a5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52"
Jul 28 17:56:04.868384 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:04.868370    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57k4b\" (UniqueName: \"kubernetes.io/projected/b6c5f3d4-531f-4966-817e-b57fc0d211a5-kube-api-access-57k4b\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52\" (UID: \"b6c5f3d4-531f-4966-817e-b57fc0d211a5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52"
Jul 28 17:56:04.868493 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:04.868417    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6c5f3d4-531f-4966-817e-b57fc0d211a5-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52\" (UID: \"b6c5f3d4-531f-4966-817e-b57fc0d211a5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52"
Jul 28 17:56:04.868633 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:04.868611    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6c5f3d4-531f-4966-817e-b57fc0d211a5-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52\" (UID: \"b6c5f3d4-531f-4966-817e-b57fc0d211a5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52"
Jul 28 17:56:04.868697 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:04.868638    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6c5f3d4-531f-4966-817e-b57fc0d211a5-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52\" (UID: \"b6c5f3d4-531f-4966-817e-b57fc0d211a5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52"
Jul 28 17:56:04.868736 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:04.868700    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6c5f3d4-531f-4966-817e-b57fc0d211a5-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52\" (UID: \"b6c5f3d4-531f-4966-817e-b57fc0d211a5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52"
Jul 28 17:56:04.870800 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:04.870776    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6c5f3d4-531f-4966-817e-b57fc0d211a5-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52\" (UID: \"b6c5f3d4-531f-4966-817e-b57fc0d211a5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52"
Jul 28 17:56:04.870949 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:04.870934    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6c5f3d4-531f-4966-817e-b57fc0d211a5-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52\" (UID: \"b6c5f3d4-531f-4966-817e-b57fc0d211a5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52"
Jul 28 17:56:04.877411 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:04.877383    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57k4b\" (UniqueName: \"kubernetes.io/projected/b6c5f3d4-531f-4966-817e-b57fc0d211a5-kube-api-access-57k4b\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52\" (UID: \"b6c5f3d4-531f-4966-817e-b57fc0d211a5\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52"
Jul 28 17:56:05.035934 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:05.035898    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52"
Jul 28 17:56:05.167234 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:05.167211    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52"]
Jul 28 17:56:05.169769 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:56:05.169740    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c5f3d4_531f_4966_817e_b57fc0d211a5.slice/crio-ed449a2d3ae68641f6f776bad613281daaad0576bf3afa1d383fb560925cee33 WatchSource:0}: Error finding container ed449a2d3ae68641f6f776bad613281daaad0576bf3afa1d383fb560925cee33: Status 404 returned error can't find the container with id ed449a2d3ae68641f6f776bad613281daaad0576bf3afa1d383fb560925cee33
Jul 28 17:56:06.118221 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:06.118181    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52" event={"ID":"b6c5f3d4-531f-4966-817e-b57fc0d211a5","Type":"ContainerStarted","Data":"9fa7afbc6596929e5e4ef74bb3946a407ada966b121681894048656400609833"}
Jul 28 17:56:06.118725 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:06.118275    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52" event={"ID":"b6c5f3d4-531f-4966-817e-b57fc0d211a5","Type":"ContainerStarted","Data":"ed449a2d3ae68641f6f776bad613281daaad0576bf3afa1d383fb560925cee33"}
Jul 28 17:56:06.154014 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:06.153955    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52" podStartSLOduration=2.153939761 podStartE2EDuration="2.153939761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:56:06.15323182 +0000 UTC m=+1969.559505560" watchObservedRunningTime="2026-07-28 17:56:06.153939761 +0000 UTC m=+1969.560213502"
Jul 28 17:56:07.036508 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:07.036471    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52"
Jul 28 17:56:15.036417 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:15.036329    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52"
Jul 28 17:56:15.049094 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:15.049063    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52"
Jul 28 17:56:15.174800 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:15.174767    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52"
Jul 28 17:56:56.812297 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:56.812264    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp"]
Jul 28 17:56:56.815685 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:56.815665    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp"
Jul 28 17:56:56.818008 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:56.817988    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 28 17:56:56.828303 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:56.828280    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp"]
Jul 28 17:56:56.910010 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:56.909978    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp\" (UID: \"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp"
Jul 28 17:56:56.910175 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:56.910018    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp\" (UID: \"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp"
Jul 28 17:56:56.910175 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:56.910126    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hkss\" (UniqueName: \"kubernetes.io/projected/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-kube-api-access-4hkss\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp\" (UID: \"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp"
Jul 28 17:56:56.910260 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:56.910183    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp\" (UID: \"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp"
Jul 28 17:56:56.910260 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:56.910216    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp\" (UID: \"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp"
Jul 28 17:56:56.910327 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:56.910299    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp\" (UID: \"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp"
Jul 28 17:56:57.011456 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:57.011424    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hkss\" (UniqueName: \"kubernetes.io/projected/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-kube-api-access-4hkss\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp\" (UID: \"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp"
Jul 28 17:56:57.011586 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:57.011462    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp\" (UID: \"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp"
Jul 28 17:56:57.011586 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:57.011480    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp\" (UID: \"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp"
Jul 28 17:56:57.011586 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:57.011539    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp\" (UID: \"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp"
Jul 28 17:56:57.011586 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:57.011571    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp\" (UID: \"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp"
Jul 28 17:56:57.011749 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:57.011599    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp\" (UID: \"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp"
Jul 28 17:56:57.011919 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:57.011893    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp\" (UID: \"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp"
Jul 28 17:56:57.011977 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:57.011940    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp\" (UID: \"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp"
Jul 28 17:56:57.012099 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:57.012000    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp\" (UID: \"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp"
Jul 28 17:56:57.014279 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:57.014251    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp\" (UID: \"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp"
Jul 28 17:56:57.014450 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:57.014402    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp\" (UID: \"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp"
Jul 28 17:56:57.025240 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:57.025206    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hkss\" (UniqueName: \"kubernetes.io/projected/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-kube-api-access-4hkss\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp\" (UID: \"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp"
Jul 28 17:56:57.027345 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:57.027323    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schexqpmz"]
Jul 28 17:56:57.031310 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:57.030881    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schexqpmz"
Jul 28 17:56:57.033473 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:57.033449    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-x4ftj\""
Jul 28 17:56:57.041422 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:57.041400    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schexqpmz"]
Jul 28 17:56:57.112375 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:57.112297    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl4rp\" (UniqueName: \"kubernetes.io/projected/05a6a24c-457c-4a5e-bbaa-ff34fd2716af-kube-api-access-bl4rp\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schexqpmz\" (UID: \"05a6a24c-457c-4a5e-bbaa-ff34fd2716af\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schexqpmz"
Jul 28 17:56:57.112375 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:57.112359    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05a6a24c-457c-4a5e-bbaa-ff34fd2716af-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schexqpmz\" (UID: \"05a6a24c-457c-4a5e-bbaa-ff34fd2716af\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schexqpmz"
Jul 28 17:56:57.126273 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:57.126245    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp"
Jul 28 17:56:57.213699 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:57.213672    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl4rp\" (UniqueName: \"kubernetes.io/projected/05a6a24c-457c-4a5e-bbaa-ff34fd2716af-kube-api-access-bl4rp\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schexqpmz\" (UID: \"05a6a24c-457c-4a5e-bbaa-ff34fd2716af\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schexqpmz"
Jul 28 17:56:57.213862 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:57.213717    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05a6a24c-457c-4a5e-bbaa-ff34fd2716af-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schexqpmz\" (UID: \"05a6a24c-457c-4a5e-bbaa-ff34fd2716af\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schexqpmz"
Jul 28 17:56:57.216173 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:57.216123    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05a6a24c-457c-4a5e-bbaa-ff34fd2716af-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schexqpmz\" (UID: \"05a6a24c-457c-4a5e-bbaa-ff34fd2716af\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schexqpmz"
Jul 28 17:56:57.224404 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:57.224378    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl4rp\" (UniqueName: \"kubernetes.io/projected/05a6a24c-457c-4a5e-bbaa-ff34fd2716af-kube-api-access-bl4rp\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schexqpmz\" (UID: \"05a6a24c-457c-4a5e-bbaa-ff34fd2716af\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schexqpmz"
Jul 28 17:56:57.257589 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:57.257562    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp"]
Jul 28 17:56:57.259239 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:56:57.259214    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf5d66c6_ceb9_4f4f_b87d_cbcd1e7aa6e9.slice/crio-72de9737479b80bf076394bf3c2164a6e250cbe04c68fb37b2b6d4ad31960f2a WatchSource:0}: Error finding container 72de9737479b80bf076394bf3c2164a6e250cbe04c68fb37b2b6d4ad31960f2a: Status 404 returned error can't find the container with id 72de9737479b80bf076394bf3c2164a6e250cbe04c68fb37b2b6d4ad31960f2a
Jul 28 17:56:57.347799 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:57.347771    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schexqpmz"
Jul 28 17:56:57.372018 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:57.371953    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp" event={"ID":"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9","Type":"ContainerStarted","Data":"5544c6b85118e9a063b483f6a1eecb54c942072b8e939c7f8c5cf75a0faf42b6"}
Jul 28 17:56:57.372018 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:57.372006    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp" event={"ID":"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9","Type":"ContainerStarted","Data":"72de9737479b80bf076394bf3c2164a6e250cbe04c68fb37b2b6d4ad31960f2a"}
Jul 28 17:56:57.391861 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:57.391814    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp" podStartSLOduration=1.391798482 podStartE2EDuration="1.391798482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:56:57.389754614 +0000 UTC m=+2020.796028359" watchObservedRunningTime="2026-07-28 17:56:57.391798482 +0000 UTC m=+2020.798072254"
Jul 28 17:56:57.478999 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:57.478976    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schexqpmz"]
Jul 28 17:56:57.480211 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:56:57.480184    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a6a24c_457c_4a5e_bbaa_ff34fd2716af.slice/crio-cd38164581d1b6ab28ac6b703b327da89676b9f129275d13157ef50c77fcfb6d WatchSource:0}: Error finding container cd38164581d1b6ab28ac6b703b327da89676b9f129275d13157ef50c77fcfb6d: Status 404 returned error can't find the container with id cd38164581d1b6ab28ac6b703b327da89676b9f129275d13157ef50c77fcfb6d
Jul 28 17:56:58.127092 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:58.127046    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp"
Jul 28 17:56:58.378601 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:58.378504    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schexqpmz" event={"ID":"05a6a24c-457c-4a5e-bbaa-ff34fd2716af","Type":"ContainerStarted","Data":"d550955a4a4f4b510f0122736eff109e0415192eaf59411b18b4115e9b02aca4"}
Jul 28 17:56:58.378601 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:58.378567    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schexqpmz" event={"ID":"05a6a24c-457c-4a5e-bbaa-ff34fd2716af","Type":"ContainerStarted","Data":"cd38164581d1b6ab28ac6b703b327da89676b9f129275d13157ef50c77fcfb6d"}
Jul 28 17:56:58.378846 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:58.378708    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schexqpmz"
Jul 28 17:56:58.400672 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:56:58.400622    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schexqpmz" podStartSLOduration=1.400607431 podStartE2EDuration="1.400607431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:56:58.399043886 +0000 UTC m=+2021.805317629" watchObservedRunningTime="2026-07-28 17:56:58.400607431 +0000 UTC m=+2021.806881172"
Jul 28 17:57:07.127244 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:07.127206    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp"
Jul 28 17:57:07.139753 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:07.139732    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp"
Jul 28 17:57:07.435150 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:07.435059    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp"
Jul 28 17:57:29.385405 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:29.385373    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schexqpmz"
Jul 28 17:57:31.133247 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:31.133209    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schexqpmz"]
Jul 28 17:57:31.137819 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:31.137791    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp"]
Jul 28 17:57:31.147376 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:57:31.147202    2543 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:57:31.147376 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:57:31.147376 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:57:31.147376 ip-10-0-135-177 kubenswrapper[2543]:  > containerID="5544c6b85118e9a063b483f6a1eecb54c942072b8e939c7f8c5cf75a0faf42b6" cmd=["/bin/sleep","15"]
Jul 28 17:57:31.147376 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:31.147258    2543 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp" message="" err=<
Jul 28 17:57:31.147376 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:57:31.147376 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:57:31.147376 ip-10-0-135-177 kubenswrapper[2543]:  >
Jul 28 17:57:31.147376 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:57:31.147274    2543 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:57:31.147376 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:57:31.147376 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:57:31.147376 ip-10-0-135-177 kubenswrapper[2543]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp" podUID="af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9" containerName="main" containerID="cri-o://5544c6b85118e9a063b483f6a1eecb54c942072b8e939c7f8c5cf75a0faf42b6"
Jul 28 17:57:31.147376 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:31.147313    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp" podUID="af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9" containerName="main" containerID="cri-o://5544c6b85118e9a063b483f6a1eecb54c942072b8e939c7f8c5cf75a0faf42b6" gracePeriod=60
Jul 28 17:57:31.493645 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:31.493620    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp"
Jul 28 17:57:31.509416 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:31.509393    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-home\") pod \"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9\" (UID: \"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9\") "
Jul 28 17:57:31.509491 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:31.509442    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-tls-certs\") pod \"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9\" (UID: \"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9\") "
Jul 28 17:57:31.509491 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:31.509468    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-dshm\") pod \"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9\" (UID: \"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9\") "
Jul 28 17:57:31.509590 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:31.509493    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-model-cache\") pod \"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9\" (UID: \"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9\") "
Jul 28 17:57:31.509590 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:31.509567    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-tmp-dir\") pod \"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9\" (UID: \"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9\") "
Jul 28 17:57:31.509703 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:31.509600    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hkss\" (UniqueName: \"kubernetes.io/projected/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-kube-api-access-4hkss\") pod \"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9\" (UID: \"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9\") "
Jul 28 17:57:31.509703 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:31.509659    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-home" (OuterVolumeSpecName: "home") pod "af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9" (UID: "af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:57:31.509870 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:31.509838    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-model-cache" (OuterVolumeSpecName: "model-cache") pod "af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9" (UID: "af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:57:31.509988 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:31.509848    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9" (UID: "af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:57:31.509988 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:31.509854    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-home\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:57:31.511745 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:31.511723    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-kube-api-access-4hkss" (OuterVolumeSpecName: "kube-api-access-4hkss") pod "af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9" (UID: "af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9"). InnerVolumeSpecName "kube-api-access-4hkss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:57:31.511947 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:31.511924    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9" (UID: "af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:57:31.512030 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:31.511944    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-dshm" (OuterVolumeSpecName: "dshm") pod "af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9" (UID: "af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:57:31.555948 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:31.555919    2543 generic.go:356] "Generic (PLEG): container finished" podID="af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9" containerID="5544c6b85118e9a063b483f6a1eecb54c942072b8e939c7f8c5cf75a0faf42b6" exitCode=0
Jul 28 17:57:31.556081 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:31.555957    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp"
Jul 28 17:57:31.556081 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:31.555972    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp" event={"ID":"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9","Type":"ContainerDied","Data":"5544c6b85118e9a063b483f6a1eecb54c942072b8e939c7f8c5cf75a0faf42b6"}
Jul 28 17:57:31.556081 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:31.556024    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp" event={"ID":"af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9","Type":"ContainerDied","Data":"72de9737479b80bf076394bf3c2164a6e250cbe04c68fb37b2b6d4ad31960f2a"}
Jul 28 17:57:31.556081 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:31.556045    2543 scope.go:117] "RemoveContainer" containerID="5544c6b85118e9a063b483f6a1eecb54c942072b8e939c7f8c5cf75a0faf42b6"
Jul 28 17:57:31.565169 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:31.565124    2543 scope.go:117] "RemoveContainer" containerID="5544c6b85118e9a063b483f6a1eecb54c942072b8e939c7f8c5cf75a0faf42b6"
Jul 28 17:57:31.565433 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:57:31.565411    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5544c6b85118e9a063b483f6a1eecb54c942072b8e939c7f8c5cf75a0faf42b6\": container with ID starting with 5544c6b85118e9a063b483f6a1eecb54c942072b8e939c7f8c5cf75a0faf42b6 not found: ID does not exist" containerID="5544c6b85118e9a063b483f6a1eecb54c942072b8e939c7f8c5cf75a0faf42b6"
Jul 28 17:57:31.565491 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:31.565440    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5544c6b85118e9a063b483f6a1eecb54c942072b8e939c7f8c5cf75a0faf42b6"} err="failed to get container status \"5544c6b85118e9a063b483f6a1eecb54c942072b8e939c7f8c5cf75a0faf42b6\": rpc error: code = NotFound desc = could not find container \"5544c6b85118e9a063b483f6a1eecb54c942072b8e939c7f8c5cf75a0faf42b6\": container with ID starting with 5544c6b85118e9a063b483f6a1eecb54c942072b8e939c7f8c5cf75a0faf42b6 not found: ID does not exist"
Jul 28 17:57:31.584659 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:31.584634    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp"]
Jul 28 17:57:31.588952 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:31.588931    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp"]
Jul 28 17:57:31.610280 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:31.610256    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:57:31.610280 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:31.610279    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-dshm\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:57:31.610417 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:31.610292    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-model-cache\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:57:31.610417 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:31.610326    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-tmp-dir\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:57:31.610417 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:31.610340    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hkss\" (UniqueName: \"kubernetes.io/projected/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9-kube-api-access-4hkss\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:57:32.424450 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:32.424409    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9959z4fp" podUID="af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 28 17:57:33.153367 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:33.153333    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9" path="/var/lib/kubelet/pods/af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9/volumes"
Jul 28 17:57:39.384655 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:39.384566    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schexqpmz" podUID="05a6a24c-457c-4a5e-bbaa-ff34fd2716af" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:57:46.155121 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:46.155058    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schexqpmz" podUID="05a6a24c-457c-4a5e-bbaa-ff34fd2716af" containerName="main" containerID="cri-o://d550955a4a4f4b510f0122736eff109e0415192eaf59411b18b4115e9b02aca4" gracePeriod=45
Jul 28 17:57:46.400304 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:46.400280    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schexqpmz"
Jul 28 17:57:46.418992 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:46.418926    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bl4rp\" (UniqueName: \"kubernetes.io/projected/05a6a24c-457c-4a5e-bbaa-ff34fd2716af-kube-api-access-bl4rp\") pod \"05a6a24c-457c-4a5e-bbaa-ff34fd2716af\" (UID: \"05a6a24c-457c-4a5e-bbaa-ff34fd2716af\") "
Jul 28 17:57:46.418992 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:46.418980    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05a6a24c-457c-4a5e-bbaa-ff34fd2716af-tls-certs\") pod \"05a6a24c-457c-4a5e-bbaa-ff34fd2716af\" (UID: \"05a6a24c-457c-4a5e-bbaa-ff34fd2716af\") "
Jul 28 17:57:46.421479 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:46.421448    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05a6a24c-457c-4a5e-bbaa-ff34fd2716af-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "05a6a24c-457c-4a5e-bbaa-ff34fd2716af" (UID: "05a6a24c-457c-4a5e-bbaa-ff34fd2716af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:57:46.421631 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:46.421509    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a6a24c-457c-4a5e-bbaa-ff34fd2716af-kube-api-access-bl4rp" (OuterVolumeSpecName: "kube-api-access-bl4rp") pod "05a6a24c-457c-4a5e-bbaa-ff34fd2716af" (UID: "05a6a24c-457c-4a5e-bbaa-ff34fd2716af"). InnerVolumeSpecName "kube-api-access-bl4rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:57:46.519603 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:46.519576    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl4rp\" (UniqueName: \"kubernetes.io/projected/05a6a24c-457c-4a5e-bbaa-ff34fd2716af-kube-api-access-bl4rp\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:57:46.519603 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:46.519602    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05a6a24c-457c-4a5e-bbaa-ff34fd2716af-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:57:46.633532 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:46.633506    2543 generic.go:356] "Generic (PLEG): container finished" podID="05a6a24c-457c-4a5e-bbaa-ff34fd2716af" containerID="d550955a4a4f4b510f0122736eff109e0415192eaf59411b18b4115e9b02aca4" exitCode=0
Jul 28 17:57:46.633667 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:46.633549    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schexqpmz"
Jul 28 17:57:46.633667 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:46.633568    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schexqpmz" event={"ID":"05a6a24c-457c-4a5e-bbaa-ff34fd2716af","Type":"ContainerDied","Data":"d550955a4a4f4b510f0122736eff109e0415192eaf59411b18b4115e9b02aca4"}
Jul 28 17:57:46.633667 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:46.633615    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schexqpmz" event={"ID":"05a6a24c-457c-4a5e-bbaa-ff34fd2716af","Type":"ContainerDied","Data":"cd38164581d1b6ab28ac6b703b327da89676b9f129275d13157ef50c77fcfb6d"}
Jul 28 17:57:46.633667 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:46.633636    2543 scope.go:117] "RemoveContainer" containerID="d550955a4a4f4b510f0122736eff109e0415192eaf59411b18b4115e9b02aca4"
Jul 28 17:57:46.642824 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:46.642808    2543 scope.go:117] "RemoveContainer" containerID="d550955a4a4f4b510f0122736eff109e0415192eaf59411b18b4115e9b02aca4"
Jul 28 17:57:46.643079 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:57:46.643053    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d550955a4a4f4b510f0122736eff109e0415192eaf59411b18b4115e9b02aca4\": container with ID starting with d550955a4a4f4b510f0122736eff109e0415192eaf59411b18b4115e9b02aca4 not found: ID does not exist" containerID="d550955a4a4f4b510f0122736eff109e0415192eaf59411b18b4115e9b02aca4"
Jul 28 17:57:46.643156 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:46.643084    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d550955a4a4f4b510f0122736eff109e0415192eaf59411b18b4115e9b02aca4"} err="failed to get container status \"d550955a4a4f4b510f0122736eff109e0415192eaf59411b18b4115e9b02aca4\": rpc error: code = NotFound desc = could not find container \"d550955a4a4f4b510f0122736eff109e0415192eaf59411b18b4115e9b02aca4\": container with ID starting with d550955a4a4f4b510f0122736eff109e0415192eaf59411b18b4115e9b02aca4 not found: ID does not exist"
Jul 28 17:57:46.656840 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:46.656809    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schexqpmz"]
Jul 28 17:57:46.664289 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:46.664269    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schexqpmz"]
Jul 28 17:57:47.152553 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:47.152519    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05a6a24c-457c-4a5e-bbaa-ff34fd2716af" path="/var/lib/kubelet/pods/05a6a24c-457c-4a5e-bbaa-ff34fd2716af/volumes"
Jul 28 17:57:51.523520 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:51.523489    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52"]
Jul 28 17:57:51.533381 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:57:51.533347    2543 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:57:51.533381 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:57:51.533381 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:57:51.533381 ip-10-0-135-177 kubenswrapper[2543]:  > containerID="9fa7afbc6596929e5e4ef74bb3946a407ada966b121681894048656400609833" cmd=["/bin/sleep","15"]
Jul 28 17:57:51.533674 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:51.533395    2543 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52" message="" err=<
Jul 28 17:57:51.533674 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:57:51.533674 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:57:51.533674 ip-10-0-135-177 kubenswrapper[2543]:  >
Jul 28 17:57:51.533674 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:57:51.533408    2543 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:57:51.533674 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:57:51.533674 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:57:51.533674 ip-10-0-135-177 kubenswrapper[2543]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52" podUID="b6c5f3d4-531f-4966-817e-b57fc0d211a5" containerName="main" containerID="cri-o://9fa7afbc6596929e5e4ef74bb3946a407ada966b121681894048656400609833"
Jul 28 17:57:51.533674 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:51.533445    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52" podUID="b6c5f3d4-531f-4966-817e-b57fc0d211a5" containerName="main" containerID="cri-o://9fa7afbc6596929e5e4ef74bb3946a407ada966b121681894048656400609833" gracePeriod=60
Jul 28 17:57:51.894886 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:51.894856    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52"
Jul 28 17:57:51.962219 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:51.962183    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6c5f3d4-531f-4966-817e-b57fc0d211a5-model-cache\") pod \"b6c5f3d4-531f-4966-817e-b57fc0d211a5\" (UID: \"b6c5f3d4-531f-4966-817e-b57fc0d211a5\") "
Jul 28 17:57:51.962384 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:51.962241    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6c5f3d4-531f-4966-817e-b57fc0d211a5-home\") pod \"b6c5f3d4-531f-4966-817e-b57fc0d211a5\" (UID: \"b6c5f3d4-531f-4966-817e-b57fc0d211a5\") "
Jul 28 17:57:51.962384 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:51.962296    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6c5f3d4-531f-4966-817e-b57fc0d211a5-dshm\") pod \"b6c5f3d4-531f-4966-817e-b57fc0d211a5\" (UID: \"b6c5f3d4-531f-4966-817e-b57fc0d211a5\") "
Jul 28 17:57:51.962384 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:51.962317    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57k4b\" (UniqueName: \"kubernetes.io/projected/b6c5f3d4-531f-4966-817e-b57fc0d211a5-kube-api-access-57k4b\") pod \"b6c5f3d4-531f-4966-817e-b57fc0d211a5\" (UID: \"b6c5f3d4-531f-4966-817e-b57fc0d211a5\") "
Jul 28 17:57:51.962384 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:51.962354    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6c5f3d4-531f-4966-817e-b57fc0d211a5-tls-certs\") pod \"b6c5f3d4-531f-4966-817e-b57fc0d211a5\" (UID: \"b6c5f3d4-531f-4966-817e-b57fc0d211a5\") "
Jul 28 17:57:51.962384 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:51.962376    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6c5f3d4-531f-4966-817e-b57fc0d211a5-tmp-dir\") pod \"b6c5f3d4-531f-4966-817e-b57fc0d211a5\" (UID: \"b6c5f3d4-531f-4966-817e-b57fc0d211a5\") "
Jul 28 17:57:51.962639 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:51.962464    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6c5f3d4-531f-4966-817e-b57fc0d211a5-model-cache" (OuterVolumeSpecName: "model-cache") pod "b6c5f3d4-531f-4966-817e-b57fc0d211a5" (UID: "b6c5f3d4-531f-4966-817e-b57fc0d211a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:57:51.962639 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:51.962569    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6c5f3d4-531f-4966-817e-b57fc0d211a5-home" (OuterVolumeSpecName: "home") pod "b6c5f3d4-531f-4966-817e-b57fc0d211a5" (UID: "b6c5f3d4-531f-4966-817e-b57fc0d211a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:57:51.962756 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:51.962658    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6c5f3d4-531f-4966-817e-b57fc0d211a5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b6c5f3d4-531f-4966-817e-b57fc0d211a5" (UID: "b6c5f3d4-531f-4966-817e-b57fc0d211a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:57:51.962756 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:51.962715    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6c5f3d4-531f-4966-817e-b57fc0d211a5-tmp-dir\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:57:51.962756 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:51.962730    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6c5f3d4-531f-4966-817e-b57fc0d211a5-model-cache\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:57:51.962756 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:51.962744    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6c5f3d4-531f-4966-817e-b57fc0d211a5-home\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:57:51.964475 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:51.964451    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6c5f3d4-531f-4966-817e-b57fc0d211a5-kube-api-access-57k4b" (OuterVolumeSpecName: "kube-api-access-57k4b") pod "b6c5f3d4-531f-4966-817e-b57fc0d211a5" (UID: "b6c5f3d4-531f-4966-817e-b57fc0d211a5"). InnerVolumeSpecName "kube-api-access-57k4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:57:51.964635 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:51.964613    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6c5f3d4-531f-4966-817e-b57fc0d211a5-dshm" (OuterVolumeSpecName: "dshm") pod "b6c5f3d4-531f-4966-817e-b57fc0d211a5" (UID: "b6c5f3d4-531f-4966-817e-b57fc0d211a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:57:51.964700 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:51.964613    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6c5f3d4-531f-4966-817e-b57fc0d211a5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b6c5f3d4-531f-4966-817e-b57fc0d211a5" (UID: "b6c5f3d4-531f-4966-817e-b57fc0d211a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:57:52.063444 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:52.063383    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6c5f3d4-531f-4966-817e-b57fc0d211a5-dshm\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:57:52.063444 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:52.063406    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57k4b\" (UniqueName: \"kubernetes.io/projected/b6c5f3d4-531f-4966-817e-b57fc0d211a5-kube-api-access-57k4b\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:57:52.063444 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:52.063416    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6c5f3d4-531f-4966-817e-b57fc0d211a5-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:57:52.664770 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:52.664730    2543 generic.go:356] "Generic (PLEG): container finished" podID="b6c5f3d4-531f-4966-817e-b57fc0d211a5" containerID="9fa7afbc6596929e5e4ef74bb3946a407ada966b121681894048656400609833" exitCode=0
Jul 28 17:57:52.664770 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:52.664773    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52"
Jul 28 17:57:52.665317 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:52.664818    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52" event={"ID":"b6c5f3d4-531f-4966-817e-b57fc0d211a5","Type":"ContainerDied","Data":"9fa7afbc6596929e5e4ef74bb3946a407ada966b121681894048656400609833"}
Jul 28 17:57:52.665317 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:52.664870    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52" event={"ID":"b6c5f3d4-531f-4966-817e-b57fc0d211a5","Type":"ContainerDied","Data":"ed449a2d3ae68641f6f776bad613281daaad0576bf3afa1d383fb560925cee33"}
Jul 28 17:57:52.665317 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:52.664895    2543 scope.go:117] "RemoveContainer" containerID="9fa7afbc6596929e5e4ef74bb3946a407ada966b121681894048656400609833"
Jul 28 17:57:52.675043 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:52.675016    2543 scope.go:117] "RemoveContainer" containerID="9fa7afbc6596929e5e4ef74bb3946a407ada966b121681894048656400609833"
Jul 28 17:57:52.675321 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:57:52.675298    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fa7afbc6596929e5e4ef74bb3946a407ada966b121681894048656400609833\": container with ID starting with 9fa7afbc6596929e5e4ef74bb3946a407ada966b121681894048656400609833 not found: ID does not exist" containerID="9fa7afbc6596929e5e4ef74bb3946a407ada966b121681894048656400609833"
Jul 28 17:57:52.675377 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:52.675334    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fa7afbc6596929e5e4ef74bb3946a407ada966b121681894048656400609833"} err="failed to get container status \"9fa7afbc6596929e5e4ef74bb3946a407ada966b121681894048656400609833\": rpc error: code = NotFound desc = could not find container \"9fa7afbc6596929e5e4ef74bb3946a407ada966b121681894048656400609833\": container with ID starting with 9fa7afbc6596929e5e4ef74bb3946a407ada966b121681894048656400609833 not found: ID does not exist"
Jul 28 17:57:52.689480 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:52.689454    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52"]
Jul 28 17:57:52.693799 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:52.693779    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z2c52"]
Jul 28 17:57:53.154425 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:57:53.154355    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6c5f3d4-531f-4966-817e-b57fc0d211a5" path="/var/lib/kubelet/pods/b6c5f3d4-531f-4966-817e-b57fc0d211a5/volumes"
Jul 28 17:58:18.530426 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.530393    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59"]
Jul 28 17:58:18.530926 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.530909    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9" containerName="main"
Jul 28 17:58:18.530998 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.530928    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9" containerName="main"
Jul 28 17:58:18.530998 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.530946    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6c5f3d4-531f-4966-817e-b57fc0d211a5" containerName="main"
Jul 28 17:58:18.530998 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.530955    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6c5f3d4-531f-4966-817e-b57fc0d211a5" containerName="main"
Jul 28 17:58:18.530998 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.530971    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05a6a24c-457c-4a5e-bbaa-ff34fd2716af" containerName="main"
Jul 28 17:58:18.530998 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.530979    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="05a6a24c-457c-4a5e-bbaa-ff34fd2716af" containerName="main"
Jul 28 17:58:18.531298 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.531060    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="05a6a24c-457c-4a5e-bbaa-ff34fd2716af" containerName="main"
Jul 28 17:58:18.531298 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.531072    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6c5f3d4-531f-4966-817e-b57fc0d211a5" containerName="main"
Jul 28 17:58:18.531298 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.531083    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="af5d66c6-ceb9-4f4f-b87d-cbcd1e7aa6e9" containerName="main"
Jul 28 17:58:18.535545 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.535522    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59"
Jul 28 17:58:18.538448 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.538425    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Jul 28 17:58:18.538575 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.538555    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-r9kbj\""
Jul 28 17:58:18.539407 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.539389    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 28 17:58:18.539407 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.539401    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Jul 28 17:58:18.546257 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.546233    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59"]
Jul 28 17:58:18.685674 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.685642    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70bcd703-5a46-450f-9a40-20198f8eaf48-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59\" (UID: \"70bcd703-5a46-450f-9a40-20198f8eaf48\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59"
Jul 28 17:58:18.685674 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.685678    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwsr5\" (UniqueName: \"kubernetes.io/projected/70bcd703-5a46-450f-9a40-20198f8eaf48-kube-api-access-jwsr5\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59\" (UID: \"70bcd703-5a46-450f-9a40-20198f8eaf48\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59"
Jul 28 17:58:18.685915 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.685702    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70bcd703-5a46-450f-9a40-20198f8eaf48-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59\" (UID: \"70bcd703-5a46-450f-9a40-20198f8eaf48\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59"
Jul 28 17:58:18.685915 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.685769    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70bcd703-5a46-450f-9a40-20198f8eaf48-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59\" (UID: \"70bcd703-5a46-450f-9a40-20198f8eaf48\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59"
Jul 28 17:58:18.685915 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.685859    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70bcd703-5a46-450f-9a40-20198f8eaf48-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59\" (UID: \"70bcd703-5a46-450f-9a40-20198f8eaf48\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59"
Jul 28 17:58:18.685915 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.685894    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70bcd703-5a46-450f-9a40-20198f8eaf48-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59\" (UID: \"70bcd703-5a46-450f-9a40-20198f8eaf48\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59"
Jul 28 17:58:18.787129 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.787040    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70bcd703-5a46-450f-9a40-20198f8eaf48-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59\" (UID: \"70bcd703-5a46-450f-9a40-20198f8eaf48\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59"
Jul 28 17:58:18.787129 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.787085    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwsr5\" (UniqueName: \"kubernetes.io/projected/70bcd703-5a46-450f-9a40-20198f8eaf48-kube-api-access-jwsr5\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59\" (UID: \"70bcd703-5a46-450f-9a40-20198f8eaf48\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59"
Jul 28 17:58:18.787391 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.787179    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70bcd703-5a46-450f-9a40-20198f8eaf48-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59\" (UID: \"70bcd703-5a46-450f-9a40-20198f8eaf48\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59"
Jul 28 17:58:18.787391 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.787233    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70bcd703-5a46-450f-9a40-20198f8eaf48-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59\" (UID: \"70bcd703-5a46-450f-9a40-20198f8eaf48\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59"
Jul 28 17:58:18.787391 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.787285    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70bcd703-5a46-450f-9a40-20198f8eaf48-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59\" (UID: \"70bcd703-5a46-450f-9a40-20198f8eaf48\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59"
Jul 28 17:58:18.787391 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.787320    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70bcd703-5a46-450f-9a40-20198f8eaf48-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59\" (UID: \"70bcd703-5a46-450f-9a40-20198f8eaf48\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59"
Jul 28 17:58:18.787605 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.787526    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70bcd703-5a46-450f-9a40-20198f8eaf48-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59\" (UID: \"70bcd703-5a46-450f-9a40-20198f8eaf48\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59"
Jul 28 17:58:18.787605 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.787550    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70bcd703-5a46-450f-9a40-20198f8eaf48-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59\" (UID: \"70bcd703-5a46-450f-9a40-20198f8eaf48\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59"
Jul 28 17:58:18.787720 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.787695    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70bcd703-5a46-450f-9a40-20198f8eaf48-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59\" (UID: \"70bcd703-5a46-450f-9a40-20198f8eaf48\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59"
Jul 28 17:58:18.789493 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.789470    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70bcd703-5a46-450f-9a40-20198f8eaf48-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59\" (UID: \"70bcd703-5a46-450f-9a40-20198f8eaf48\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59"
Jul 28 17:58:18.789770 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.789751    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70bcd703-5a46-450f-9a40-20198f8eaf48-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59\" (UID: \"70bcd703-5a46-450f-9a40-20198f8eaf48\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59"
Jul 28 17:58:18.812332 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.812302    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwsr5\" (UniqueName: \"kubernetes.io/projected/70bcd703-5a46-450f-9a40-20198f8eaf48-kube-api-access-jwsr5\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59\" (UID: \"70bcd703-5a46-450f-9a40-20198f8eaf48\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59"
Jul 28 17:58:18.846177 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:18.846126    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59"
Jul 28 17:58:19.005968 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:19.005944    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59"]
Jul 28 17:58:19.008052 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:58:19.008022    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70bcd703_5a46_450f_9a40_20198f8eaf48.slice/crio-3b7a7c8b1bdc25149e03d095c5a3b0d18813a793aae8f6534f6e506bd77746c2 WatchSource:0}: Error finding container 3b7a7c8b1bdc25149e03d095c5a3b0d18813a793aae8f6534f6e506bd77746c2: Status 404 returned error can't find the container with id 3b7a7c8b1bdc25149e03d095c5a3b0d18813a793aae8f6534f6e506bd77746c2
Jul 28 17:58:19.807581 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:19.807543    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59" event={"ID":"70bcd703-5a46-450f-9a40-20198f8eaf48","Type":"ContainerStarted","Data":"390b05c8c2e82ce67f86deba6019ac88dc8cc58111cf89ee0da4ad7a47b25fc6"}
Jul 28 17:58:19.808013 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:19.807607    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59" event={"ID":"70bcd703-5a46-450f-9a40-20198f8eaf48","Type":"ContainerStarted","Data":"3b7a7c8b1bdc25149e03d095c5a3b0d18813a793aae8f6534f6e506bd77746c2"}
Jul 28 17:58:19.835580 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:19.835519    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59" podStartSLOduration=1.835501422 podStartE2EDuration="1.835501422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:58:19.835342042 +0000 UTC m=+2103.241615784" watchObservedRunningTime="2026-07-28 17:58:19.835501422 +0000 UTC m=+2103.241775164"
Jul 28 17:58:19.847181 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:19.847147    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59"
Jul 28 17:58:28.847357 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:28.847320    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59"
Jul 28 17:58:28.860819 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:28.860787    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59"
Jul 28 17:58:29.858886 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:58:29.858856    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59"
Jul 28 17:59:00.905294 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:00.905255    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq"]
Jul 28 17:59:00.908747 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:00.908731    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq"
Jul 28 17:59:00.911415 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:00.911392    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 28 17:59:00.918348 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:00.918323    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq"]
Jul 28 17:59:00.951254 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:00.951227    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g4bv\" (UniqueName: \"kubernetes.io/projected/ed66c090-fcd4-4937-8927-249cbc596710-kube-api-access-5g4bv\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq\" (UID: \"ed66c090-fcd4-4937-8927-249cbc596710\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq"
Jul 28 17:59:00.951400 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:00.951271    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed66c090-fcd4-4937-8927-249cbc596710-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq\" (UID: \"ed66c090-fcd4-4937-8927-249cbc596710\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq"
Jul 28 17:59:00.951400 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:00.951295    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed66c090-fcd4-4937-8927-249cbc596710-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq\" (UID: \"ed66c090-fcd4-4937-8927-249cbc596710\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq"
Jul 28 17:59:00.951480 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:00.951400    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed66c090-fcd4-4937-8927-249cbc596710-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq\" (UID: \"ed66c090-fcd4-4937-8927-249cbc596710\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq"
Jul 28 17:59:00.951480 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:00.951435    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed66c090-fcd4-4937-8927-249cbc596710-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq\" (UID: \"ed66c090-fcd4-4937-8927-249cbc596710\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq"
Jul 28 17:59:00.951550 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:00.951483    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed66c090-fcd4-4937-8927-249cbc596710-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq\" (UID: \"ed66c090-fcd4-4937-8927-249cbc596710\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq"
Jul 28 17:59:01.052082 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:01.052047    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed66c090-fcd4-4937-8927-249cbc596710-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq\" (UID: \"ed66c090-fcd4-4937-8927-249cbc596710\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq"
Jul 28 17:59:01.052301 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:01.052096    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g4bv\" (UniqueName: \"kubernetes.io/projected/ed66c090-fcd4-4937-8927-249cbc596710-kube-api-access-5g4bv\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq\" (UID: \"ed66c090-fcd4-4937-8927-249cbc596710\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq"
Jul 28 17:59:01.052301 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:01.052165    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed66c090-fcd4-4937-8927-249cbc596710-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq\" (UID: \"ed66c090-fcd4-4937-8927-249cbc596710\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq"
Jul 28 17:59:01.052301 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:01.052197    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed66c090-fcd4-4937-8927-249cbc596710-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq\" (UID: \"ed66c090-fcd4-4937-8927-249cbc596710\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq"
Jul 28 17:59:01.052301 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:01.052247    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed66c090-fcd4-4937-8927-249cbc596710-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq\" (UID: \"ed66c090-fcd4-4937-8927-249cbc596710\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq"
Jul 28 17:59:01.052301 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:01.052277    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed66c090-fcd4-4937-8927-249cbc596710-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq\" (UID: \"ed66c090-fcd4-4937-8927-249cbc596710\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq"
Jul 28 17:59:01.052636 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:01.052615    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed66c090-fcd4-4937-8927-249cbc596710-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq\" (UID: \"ed66c090-fcd4-4937-8927-249cbc596710\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq"
Jul 28 17:59:01.052729 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:01.052711    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed66c090-fcd4-4937-8927-249cbc596710-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq\" (UID: \"ed66c090-fcd4-4937-8927-249cbc596710\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq"
Jul 28 17:59:01.052772 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:01.052736    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed66c090-fcd4-4937-8927-249cbc596710-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq\" (UID: \"ed66c090-fcd4-4937-8927-249cbc596710\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq"
Jul 28 17:59:01.054538 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:01.054522    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed66c090-fcd4-4937-8927-249cbc596710-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq\" (UID: \"ed66c090-fcd4-4937-8927-249cbc596710\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq"
Jul 28 17:59:01.054654 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:01.054637    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed66c090-fcd4-4937-8927-249cbc596710-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq\" (UID: \"ed66c090-fcd4-4937-8927-249cbc596710\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq"
Jul 28 17:59:01.060920 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:01.060897    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g4bv\" (UniqueName: \"kubernetes.io/projected/ed66c090-fcd4-4937-8927-249cbc596710-kube-api-access-5g4bv\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq\" (UID: \"ed66c090-fcd4-4937-8927-249cbc596710\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq"
Jul 28 17:59:01.113228 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:01.113186    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scher62tn"]
Jul 28 17:59:01.116976 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:01.116954    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scher62tn"
Jul 28 17:59:01.119371 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:01.119348    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-vf8dr\""
Jul 28 17:59:01.129859 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:01.129831    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scher62tn"]
Jul 28 17:59:01.220351 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:01.220264    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq"
Jul 28 17:59:01.254570 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:01.254518    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrv9p\" (UniqueName: \"kubernetes.io/projected/12b1bdbb-82e4-472c-99f6-b8d7ab244b11-kube-api-access-nrv9p\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scher62tn\" (UID: \"12b1bdbb-82e4-472c-99f6-b8d7ab244b11\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scher62tn"
Jul 28 17:59:01.254756 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:01.254614    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12b1bdbb-82e4-472c-99f6-b8d7ab244b11-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scher62tn\" (UID: \"12b1bdbb-82e4-472c-99f6-b8d7ab244b11\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scher62tn"
Jul 28 17:59:01.354376 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:01.354353    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq"]
Jul 28 17:59:01.355300 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:01.355269    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrv9p\" (UniqueName: \"kubernetes.io/projected/12b1bdbb-82e4-472c-99f6-b8d7ab244b11-kube-api-access-nrv9p\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scher62tn\" (UID: \"12b1bdbb-82e4-472c-99f6-b8d7ab244b11\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scher62tn"
Jul 28 17:59:01.355426 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:01.355348    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12b1bdbb-82e4-472c-99f6-b8d7ab244b11-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scher62tn\" (UID: \"12b1bdbb-82e4-472c-99f6-b8d7ab244b11\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scher62tn"
Jul 28 17:59:01.356392 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:59:01.356369    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded66c090_fcd4_4937_8927_249cbc596710.slice/crio-9c6173fd7a6d0e081f37eea7882903847144eaa89594db63948be88a38c766ff WatchSource:0}: Error finding container 9c6173fd7a6d0e081f37eea7882903847144eaa89594db63948be88a38c766ff: Status 404 returned error can't find the container with id 9c6173fd7a6d0e081f37eea7882903847144eaa89594db63948be88a38c766ff
Jul 28 17:59:01.357993 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:01.357977    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12b1bdbb-82e4-472c-99f6-b8d7ab244b11-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scher62tn\" (UID: \"12b1bdbb-82e4-472c-99f6-b8d7ab244b11\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scher62tn"
Jul 28 17:59:01.365403 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:01.365382    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrv9p\" (UniqueName: \"kubernetes.io/projected/12b1bdbb-82e4-472c-99f6-b8d7ab244b11-kube-api-access-nrv9p\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scher62tn\" (UID: \"12b1bdbb-82e4-472c-99f6-b8d7ab244b11\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scher62tn"
Jul 28 17:59:01.429052 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:01.429028    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scher62tn"
Jul 28 17:59:01.563610 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:01.563585    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scher62tn"]
Jul 28 17:59:01.565434 ip-10-0-135-177 kubenswrapper[2543]: W0728 17:59:01.565410    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b1bdbb_82e4_472c_99f6_b8d7ab244b11.slice/crio-8d885a137b6d974fbb66584cbb12aa05a2c728918596bdf7996e7e6da40e7c37 WatchSource:0}: Error finding container 8d885a137b6d974fbb66584cbb12aa05a2c728918596bdf7996e7e6da40e7c37: Status 404 returned error can't find the container with id 8d885a137b6d974fbb66584cbb12aa05a2c728918596bdf7996e7e6da40e7c37
Jul 28 17:59:02.026617 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:02.026577    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scher62tn" event={"ID":"12b1bdbb-82e4-472c-99f6-b8d7ab244b11","Type":"ContainerStarted","Data":"66bc3fdb0c36a787a9a5eb449e2166ed170f7c83d0cbedf5cba9b1aab0977092"}
Jul 28 17:59:02.027147 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:02.026673    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scher62tn" event={"ID":"12b1bdbb-82e4-472c-99f6-b8d7ab244b11","Type":"ContainerStarted","Data":"8d885a137b6d974fbb66584cbb12aa05a2c728918596bdf7996e7e6da40e7c37"}
Jul 28 17:59:02.027147 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:02.026730    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scher62tn"
Jul 28 17:59:02.028321 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:02.028292    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq" event={"ID":"ed66c090-fcd4-4937-8927-249cbc596710","Type":"ContainerStarted","Data":"d1dd6d84be7abba9f3e8aee6e4d493ce9c51699767b1581f1940a321897d49f6"}
Jul 28 17:59:02.028461 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:02.028340    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq" event={"ID":"ed66c090-fcd4-4937-8927-249cbc596710","Type":"ContainerStarted","Data":"9c6173fd7a6d0e081f37eea7882903847144eaa89594db63948be88a38c766ff"}
Jul 28 17:59:02.070016 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:02.069952    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scher62tn" podStartSLOduration=1.069929949 podStartE2EDuration="1.069929949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:59:02.06836204 +0000 UTC m=+2145.474635783" watchObservedRunningTime="2026-07-28 17:59:02.069929949 +0000 UTC m=+2145.476203669"
Jul 28 17:59:02.130222 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:02.130170    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq" podStartSLOduration=2.130155546 podStartE2EDuration="2.130155546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:59:02.125758868 +0000 UTC m=+2145.532032610" watchObservedRunningTime="2026-07-28 17:59:02.130155546 +0000 UTC m=+2145.536429281"
Jul 28 17:59:02.220577 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:02.220542    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq"
Jul 28 17:59:11.220834 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:11.220749    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq"
Jul 28 17:59:11.233272 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:11.233249    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq"
Jul 28 17:59:12.097153 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:12.097101    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq"
Jul 28 17:59:33.035583 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:33.035554    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scher62tn"
Jul 28 17:59:34.348491 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:34.348457    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq"]
Jul 28 17:59:34.352010 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:34.351985    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scher62tn"]
Jul 28 17:59:34.358597 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:59:34.358563    2543 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:59:34.358597 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:59:34.358597 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:59:34.358597 ip-10-0-135-177 kubenswrapper[2543]:  > containerID="d1dd6d84be7abba9f3e8aee6e4d493ce9c51699767b1581f1940a321897d49f6" cmd=["/bin/sleep","15"]
Jul 28 17:59:34.358876 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:34.358613    2543 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq" message="" err=<
Jul 28 17:59:34.358876 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:59:34.358876 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:59:34.358876 ip-10-0-135-177 kubenswrapper[2543]:  >
Jul 28 17:59:34.358876 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:59:34.358625    2543 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:59:34.358876 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:59:34.358876 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:59:34.358876 ip-10-0-135-177 kubenswrapper[2543]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq" podUID="ed66c090-fcd4-4937-8927-249cbc596710" containerName="main" containerID="cri-o://d1dd6d84be7abba9f3e8aee6e4d493ce9c51699767b1581f1940a321897d49f6"
Jul 28 17:59:34.358876 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:34.358668    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq" podUID="ed66c090-fcd4-4937-8927-249cbc596710" containerName="main" containerID="cri-o://d1dd6d84be7abba9f3e8aee6e4d493ce9c51699767b1581f1940a321897d49f6" gracePeriod=60
Jul 28 17:59:34.696173 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:34.696128    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq"
Jul 28 17:59:34.740750 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:34.740716    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed66c090-fcd4-4937-8927-249cbc596710-model-cache\") pod \"ed66c090-fcd4-4937-8927-249cbc596710\" (UID: \"ed66c090-fcd4-4937-8927-249cbc596710\") "
Jul 28 17:59:34.740947 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:34.740778    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed66c090-fcd4-4937-8927-249cbc596710-home\") pod \"ed66c090-fcd4-4937-8927-249cbc596710\" (UID: \"ed66c090-fcd4-4937-8927-249cbc596710\") "
Jul 28 17:59:34.740947 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:34.740799    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5g4bv\" (UniqueName: \"kubernetes.io/projected/ed66c090-fcd4-4937-8927-249cbc596710-kube-api-access-5g4bv\") pod \"ed66c090-fcd4-4937-8927-249cbc596710\" (UID: \"ed66c090-fcd4-4937-8927-249cbc596710\") "
Jul 28 17:59:34.740947 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:34.740817    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed66c090-fcd4-4937-8927-249cbc596710-dshm\") pod \"ed66c090-fcd4-4937-8927-249cbc596710\" (UID: \"ed66c090-fcd4-4937-8927-249cbc596710\") "
Jul 28 17:59:34.740947 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:34.740836    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed66c090-fcd4-4937-8927-249cbc596710-tmp-dir\") pod \"ed66c090-fcd4-4937-8927-249cbc596710\" (UID: \"ed66c090-fcd4-4937-8927-249cbc596710\") "
Jul 28 17:59:34.741171 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:34.741024    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed66c090-fcd4-4937-8927-249cbc596710-model-cache" (OuterVolumeSpecName: "model-cache") pod "ed66c090-fcd4-4937-8927-249cbc596710" (UID: "ed66c090-fcd4-4937-8927-249cbc596710"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:59:34.741311 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:34.741278    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed66c090-fcd4-4937-8927-249cbc596710-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ed66c090-fcd4-4937-8927-249cbc596710" (UID: "ed66c090-fcd4-4937-8927-249cbc596710"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:59:34.741440 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:34.741349    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed66c090-fcd4-4937-8927-249cbc596710-home" (OuterVolumeSpecName: "home") pod "ed66c090-fcd4-4937-8927-249cbc596710" (UID: "ed66c090-fcd4-4937-8927-249cbc596710"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:59:34.743160 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:34.743116    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed66c090-fcd4-4937-8927-249cbc596710-dshm" (OuterVolumeSpecName: "dshm") pod "ed66c090-fcd4-4937-8927-249cbc596710" (UID: "ed66c090-fcd4-4937-8927-249cbc596710"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:59:34.743251 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:34.743122    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed66c090-fcd4-4937-8927-249cbc596710-kube-api-access-5g4bv" (OuterVolumeSpecName: "kube-api-access-5g4bv") pod "ed66c090-fcd4-4937-8927-249cbc596710" (UID: "ed66c090-fcd4-4937-8927-249cbc596710"). InnerVolumeSpecName "kube-api-access-5g4bv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:59:34.841530 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:34.841485    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed66c090-fcd4-4937-8927-249cbc596710-tls-certs\") pod \"ed66c090-fcd4-4937-8927-249cbc596710\" (UID: \"ed66c090-fcd4-4937-8927-249cbc596710\") "
Jul 28 17:59:34.841721 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:34.841633    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed66c090-fcd4-4937-8927-249cbc596710-model-cache\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:59:34.841721 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:34.841644    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed66c090-fcd4-4937-8927-249cbc596710-home\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:59:34.841721 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:34.841653    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g4bv\" (UniqueName: \"kubernetes.io/projected/ed66c090-fcd4-4937-8927-249cbc596710-kube-api-access-5g4bv\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:59:34.841721 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:34.841661    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed66c090-fcd4-4937-8927-249cbc596710-dshm\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:59:34.841721 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:34.841670    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed66c090-fcd4-4937-8927-249cbc596710-tmp-dir\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:59:34.843568 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:34.843540    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed66c090-fcd4-4937-8927-249cbc596710-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ed66c090-fcd4-4937-8927-249cbc596710" (UID: "ed66c090-fcd4-4937-8927-249cbc596710"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:59:34.942848 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:34.942753    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed66c090-fcd4-4937-8927-249cbc596710-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:59:35.208521 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:35.208440    2543 generic.go:356] "Generic (PLEG): container finished" podID="ed66c090-fcd4-4937-8927-249cbc596710" containerID="d1dd6d84be7abba9f3e8aee6e4d493ce9c51699767b1581f1940a321897d49f6" exitCode=0
Jul 28 17:59:35.208521 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:35.208485    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq"
Jul 28 17:59:35.208521 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:35.208502    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq" event={"ID":"ed66c090-fcd4-4937-8927-249cbc596710","Type":"ContainerDied","Data":"d1dd6d84be7abba9f3e8aee6e4d493ce9c51699767b1581f1940a321897d49f6"}
Jul 28 17:59:35.208780 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:35.208541    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq" event={"ID":"ed66c090-fcd4-4937-8927-249cbc596710","Type":"ContainerDied","Data":"9c6173fd7a6d0e081f37eea7882903847144eaa89594db63948be88a38c766ff"}
Jul 28 17:59:35.208780 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:35.208564    2543 scope.go:117] "RemoveContainer" containerID="d1dd6d84be7abba9f3e8aee6e4d493ce9c51699767b1581f1940a321897d49f6"
Jul 28 17:59:35.218070 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:35.218052    2543 scope.go:117] "RemoveContainer" containerID="d1dd6d84be7abba9f3e8aee6e4d493ce9c51699767b1581f1940a321897d49f6"
Jul 28 17:59:35.218338 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:59:35.218320    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1dd6d84be7abba9f3e8aee6e4d493ce9c51699767b1581f1940a321897d49f6\": container with ID starting with d1dd6d84be7abba9f3e8aee6e4d493ce9c51699767b1581f1940a321897d49f6 not found: ID does not exist" containerID="d1dd6d84be7abba9f3e8aee6e4d493ce9c51699767b1581f1940a321897d49f6"
Jul 28 17:59:35.218413 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:35.218349    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1dd6d84be7abba9f3e8aee6e4d493ce9c51699767b1581f1940a321897d49f6"} err="failed to get container status \"d1dd6d84be7abba9f3e8aee6e4d493ce9c51699767b1581f1940a321897d49f6\": rpc error: code = NotFound desc = could not find container \"d1dd6d84be7abba9f3e8aee6e4d493ce9c51699767b1581f1940a321897d49f6\": container with ID starting with d1dd6d84be7abba9f3e8aee6e4d493ce9c51699767b1581f1940a321897d49f6 not found: ID does not exist"
Jul 28 17:59:35.227065 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:35.227042    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq"]
Jul 28 17:59:35.232152 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:35.232119    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-hlwlq"]
Jul 28 17:59:37.152153 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:37.152115    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed66c090-fcd4-4937-8927-249cbc596710" path="/var/lib/kubelet/pods/ed66c090-fcd4-4937-8927-249cbc596710/volumes"
Jul 28 17:59:43.035519 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:43.035478    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scher62tn" podUID="12b1bdbb-82e4-472c-99f6-b8d7ab244b11" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:59:49.370689 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:49.370650    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scher62tn" podUID="12b1bdbb-82e4-472c-99f6-b8d7ab244b11" containerName="main" containerID="cri-o://66bc3fdb0c36a787a9a5eb449e2166ed170f7c83d0cbedf5cba9b1aab0977092" gracePeriod=45
Jul 28 17:59:49.622796 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:49.622732    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scher62tn"
Jul 28 17:59:49.657955 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:49.657921    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrv9p\" (UniqueName: \"kubernetes.io/projected/12b1bdbb-82e4-472c-99f6-b8d7ab244b11-kube-api-access-nrv9p\") pod \"12b1bdbb-82e4-472c-99f6-b8d7ab244b11\" (UID: \"12b1bdbb-82e4-472c-99f6-b8d7ab244b11\") "
Jul 28 17:59:49.658105 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:49.657985    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12b1bdbb-82e4-472c-99f6-b8d7ab244b11-tls-certs\") pod \"12b1bdbb-82e4-472c-99f6-b8d7ab244b11\" (UID: \"12b1bdbb-82e4-472c-99f6-b8d7ab244b11\") "
Jul 28 17:59:49.660153 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:49.660102    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b1bdbb-82e4-472c-99f6-b8d7ab244b11-kube-api-access-nrv9p" (OuterVolumeSpecName: "kube-api-access-nrv9p") pod "12b1bdbb-82e4-472c-99f6-b8d7ab244b11" (UID: "12b1bdbb-82e4-472c-99f6-b8d7ab244b11"). InnerVolumeSpecName "kube-api-access-nrv9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:59:49.660153 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:49.660123    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12b1bdbb-82e4-472c-99f6-b8d7ab244b11-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "12b1bdbb-82e4-472c-99f6-b8d7ab244b11" (UID: "12b1bdbb-82e4-472c-99f6-b8d7ab244b11"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:59:49.759305 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:49.759270    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrv9p\" (UniqueName: \"kubernetes.io/projected/12b1bdbb-82e4-472c-99f6-b8d7ab244b11-kube-api-access-nrv9p\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:59:49.759305 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:49.759298    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12b1bdbb-82e4-472c-99f6-b8d7ab244b11-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:59:50.291974 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:50.291935    2543 generic.go:356] "Generic (PLEG): container finished" podID="12b1bdbb-82e4-472c-99f6-b8d7ab244b11" containerID="66bc3fdb0c36a787a9a5eb449e2166ed170f7c83d0cbedf5cba9b1aab0977092" exitCode=0
Jul 28 17:59:50.292180 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:50.291995    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scher62tn"
Jul 28 17:59:50.292180 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:50.292022    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scher62tn" event={"ID":"12b1bdbb-82e4-472c-99f6-b8d7ab244b11","Type":"ContainerDied","Data":"66bc3fdb0c36a787a9a5eb449e2166ed170f7c83d0cbedf5cba9b1aab0977092"}
Jul 28 17:59:50.292180 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:50.292106    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scher62tn" event={"ID":"12b1bdbb-82e4-472c-99f6-b8d7ab244b11","Type":"ContainerDied","Data":"8d885a137b6d974fbb66584cbb12aa05a2c728918596bdf7996e7e6da40e7c37"}
Jul 28 17:59:50.292180 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:50.292143    2543 scope.go:117] "RemoveContainer" containerID="66bc3fdb0c36a787a9a5eb449e2166ed170f7c83d0cbedf5cba9b1aab0977092"
Jul 28 17:59:50.301200 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:50.301128    2543 scope.go:117] "RemoveContainer" containerID="66bc3fdb0c36a787a9a5eb449e2166ed170f7c83d0cbedf5cba9b1aab0977092"
Jul 28 17:59:50.301462 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:59:50.301441    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66bc3fdb0c36a787a9a5eb449e2166ed170f7c83d0cbedf5cba9b1aab0977092\": container with ID starting with 66bc3fdb0c36a787a9a5eb449e2166ed170f7c83d0cbedf5cba9b1aab0977092 not found: ID does not exist" containerID="66bc3fdb0c36a787a9a5eb449e2166ed170f7c83d0cbedf5cba9b1aab0977092"
Jul 28 17:59:50.301532 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:50.301473    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66bc3fdb0c36a787a9a5eb449e2166ed170f7c83d0cbedf5cba9b1aab0977092"} err="failed to get container status \"66bc3fdb0c36a787a9a5eb449e2166ed170f7c83d0cbedf5cba9b1aab0977092\": rpc error: code = NotFound desc = could not find container \"66bc3fdb0c36a787a9a5eb449e2166ed170f7c83d0cbedf5cba9b1aab0977092\": container with ID starting with 66bc3fdb0c36a787a9a5eb449e2166ed170f7c83d0cbedf5cba9b1aab0977092 not found: ID does not exist"
Jul 28 17:59:50.314293 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:50.314265    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scher62tn"]
Jul 28 17:59:50.318757 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:50.318735    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scher62tn"]
Jul 28 17:59:51.152569 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:51.152537    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12b1bdbb-82e4-472c-99f6-b8d7ab244b11" path="/var/lib/kubelet/pods/12b1bdbb-82e4-472c-99f6-b8d7ab244b11/volumes"
Jul 28 17:59:54.511790 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:54.511751    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59"]
Jul 28 17:59:54.525741 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:59:54.525699    2543 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:59:54.525741 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:59:54.525741 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:59:54.525741 ip-10-0-135-177 kubenswrapper[2543]:  > containerID="390b05c8c2e82ce67f86deba6019ac88dc8cc58111cf89ee0da4ad7a47b25fc6" cmd=["/bin/sleep","15"]
Jul 28 17:59:54.526089 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:54.525756    2543 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59" message="" err=<
Jul 28 17:59:54.526089 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:59:54.526089 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:59:54.526089 ip-10-0-135-177 kubenswrapper[2543]:  >
Jul 28 17:59:54.526089 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:59:54.525768    2543 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:59:54.526089 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:59:54.526089 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 17:59:54.526089 ip-10-0-135-177 kubenswrapper[2543]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59" podUID="70bcd703-5a46-450f-9a40-20198f8eaf48" containerName="main" containerID="cri-o://390b05c8c2e82ce67f86deba6019ac88dc8cc58111cf89ee0da4ad7a47b25fc6"
Jul 28 17:59:54.526089 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:54.525813    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59" podUID="70bcd703-5a46-450f-9a40-20198f8eaf48" containerName="main" containerID="cri-o://390b05c8c2e82ce67f86deba6019ac88dc8cc58111cf89ee0da4ad7a47b25fc6" gracePeriod=60
Jul 28 17:59:54.881623 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:54.881600    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59"
Jul 28 17:59:55.002353 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:55.002318    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70bcd703-5a46-450f-9a40-20198f8eaf48-tls-certs\") pod \"70bcd703-5a46-450f-9a40-20198f8eaf48\" (UID: \"70bcd703-5a46-450f-9a40-20198f8eaf48\") "
Jul 28 17:59:55.002554 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:55.002367    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70bcd703-5a46-450f-9a40-20198f8eaf48-home\") pod \"70bcd703-5a46-450f-9a40-20198f8eaf48\" (UID: \"70bcd703-5a46-450f-9a40-20198f8eaf48\") "
Jul 28 17:59:55.002554 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:55.002397    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwsr5\" (UniqueName: \"kubernetes.io/projected/70bcd703-5a46-450f-9a40-20198f8eaf48-kube-api-access-jwsr5\") pod \"70bcd703-5a46-450f-9a40-20198f8eaf48\" (UID: \"70bcd703-5a46-450f-9a40-20198f8eaf48\") "
Jul 28 17:59:55.002554 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:55.002444    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70bcd703-5a46-450f-9a40-20198f8eaf48-dshm\") pod \"70bcd703-5a46-450f-9a40-20198f8eaf48\" (UID: \"70bcd703-5a46-450f-9a40-20198f8eaf48\") "
Jul 28 17:59:55.002554 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:55.002474    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70bcd703-5a46-450f-9a40-20198f8eaf48-model-cache\") pod \"70bcd703-5a46-450f-9a40-20198f8eaf48\" (UID: \"70bcd703-5a46-450f-9a40-20198f8eaf48\") "
Jul 28 17:59:55.002554 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:55.002496    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70bcd703-5a46-450f-9a40-20198f8eaf48-tmp-dir\") pod \"70bcd703-5a46-450f-9a40-20198f8eaf48\" (UID: \"70bcd703-5a46-450f-9a40-20198f8eaf48\") "
Jul 28 17:59:55.002887 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:55.002746    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70bcd703-5a46-450f-9a40-20198f8eaf48-model-cache" (OuterVolumeSpecName: "model-cache") pod "70bcd703-5a46-450f-9a40-20198f8eaf48" (UID: "70bcd703-5a46-450f-9a40-20198f8eaf48"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:59:55.002887 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:55.002766    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70bcd703-5a46-450f-9a40-20198f8eaf48-home" (OuterVolumeSpecName: "home") pod "70bcd703-5a46-450f-9a40-20198f8eaf48" (UID: "70bcd703-5a46-450f-9a40-20198f8eaf48"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:59:55.002887 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:55.002845    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70bcd703-5a46-450f-9a40-20198f8eaf48-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "70bcd703-5a46-450f-9a40-20198f8eaf48" (UID: "70bcd703-5a46-450f-9a40-20198f8eaf48"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:59:55.004518 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:55.004494    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70bcd703-5a46-450f-9a40-20198f8eaf48-kube-api-access-jwsr5" (OuterVolumeSpecName: "kube-api-access-jwsr5") pod "70bcd703-5a46-450f-9a40-20198f8eaf48" (UID: "70bcd703-5a46-450f-9a40-20198f8eaf48"). InnerVolumeSpecName "kube-api-access-jwsr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:59:55.004979 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:55.004960    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70bcd703-5a46-450f-9a40-20198f8eaf48-dshm" (OuterVolumeSpecName: "dshm") pod "70bcd703-5a46-450f-9a40-20198f8eaf48" (UID: "70bcd703-5a46-450f-9a40-20198f8eaf48"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:59:55.004979 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:55.004964    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70bcd703-5a46-450f-9a40-20198f8eaf48-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "70bcd703-5a46-450f-9a40-20198f8eaf48" (UID: "70bcd703-5a46-450f-9a40-20198f8eaf48"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:59:55.103622 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:55.103541    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70bcd703-5a46-450f-9a40-20198f8eaf48-home\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:59:55.103622 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:55.103573    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwsr5\" (UniqueName: \"kubernetes.io/projected/70bcd703-5a46-450f-9a40-20198f8eaf48-kube-api-access-jwsr5\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:59:55.103622 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:55.103584    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70bcd703-5a46-450f-9a40-20198f8eaf48-dshm\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:59:55.103622 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:55.103592    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70bcd703-5a46-450f-9a40-20198f8eaf48-model-cache\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:59:55.103622 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:55.103623    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70bcd703-5a46-450f-9a40-20198f8eaf48-tmp-dir\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:59:55.103891 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:55.103631    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70bcd703-5a46-450f-9a40-20198f8eaf48-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 17:59:55.320718 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:55.320682    2543 generic.go:356] "Generic (PLEG): container finished" podID="70bcd703-5a46-450f-9a40-20198f8eaf48" containerID="390b05c8c2e82ce67f86deba6019ac88dc8cc58111cf89ee0da4ad7a47b25fc6" exitCode=0
Jul 28 17:59:55.320877 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:55.320726    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59"
Jul 28 17:59:55.320877 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:55.320768    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59" event={"ID":"70bcd703-5a46-450f-9a40-20198f8eaf48","Type":"ContainerDied","Data":"390b05c8c2e82ce67f86deba6019ac88dc8cc58111cf89ee0da4ad7a47b25fc6"}
Jul 28 17:59:55.320877 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:55.320817    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59" event={"ID":"70bcd703-5a46-450f-9a40-20198f8eaf48","Type":"ContainerDied","Data":"3b7a7c8b1bdc25149e03d095c5a3b0d18813a793aae8f6534f6e506bd77746c2"}
Jul 28 17:59:55.320877 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:55.320840    2543 scope.go:117] "RemoveContainer" containerID="390b05c8c2e82ce67f86deba6019ac88dc8cc58111cf89ee0da4ad7a47b25fc6"
Jul 28 17:59:55.329410 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:55.329387    2543 scope.go:117] "RemoveContainer" containerID="390b05c8c2e82ce67f86deba6019ac88dc8cc58111cf89ee0da4ad7a47b25fc6"
Jul 28 17:59:55.329649 ip-10-0-135-177 kubenswrapper[2543]: E0728 17:59:55.329632    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"390b05c8c2e82ce67f86deba6019ac88dc8cc58111cf89ee0da4ad7a47b25fc6\": container with ID starting with 390b05c8c2e82ce67f86deba6019ac88dc8cc58111cf89ee0da4ad7a47b25fc6 not found: ID does not exist" containerID="390b05c8c2e82ce67f86deba6019ac88dc8cc58111cf89ee0da4ad7a47b25fc6"
Jul 28 17:59:55.329693 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:55.329664    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"390b05c8c2e82ce67f86deba6019ac88dc8cc58111cf89ee0da4ad7a47b25fc6"} err="failed to get container status \"390b05c8c2e82ce67f86deba6019ac88dc8cc58111cf89ee0da4ad7a47b25fc6\": rpc error: code = NotFound desc = could not find container \"390b05c8c2e82ce67f86deba6019ac88dc8cc58111cf89ee0da4ad7a47b25fc6\": container with ID starting with 390b05c8c2e82ce67f86deba6019ac88dc8cc58111cf89ee0da4ad7a47b25fc6 not found: ID does not exist"
Jul 28 17:59:55.338185 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:55.338162    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59"]
Jul 28 17:59:55.344711 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:55.344689    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59"]
Jul 28 17:59:55.859916 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:55.859867    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969hln59" podUID="70bcd703-5a46-450f-9a40-20198f8eaf48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded"
Jul 28 17:59:57.152606 ip-10-0-135-177 kubenswrapper[2543]: I0728 17:59:57.152572    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70bcd703-5a46-450f-9a40-20198f8eaf48" path="/var/lib/kubelet/pods/70bcd703-5a46-450f-9a40-20198f8eaf48/volumes"
Jul 28 18:00:17.594982 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:17.594948    2543 scope.go:117] "RemoveContainer" containerID="1198d08474119b4ebf14be2fd009a8df04ba4905cb25b4853631b5a12d91569d"
Jul 28 18:00:17.604549 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:17.604523    2543 scope.go:117] "RemoveContainer" containerID="20921bfcd3595a6224c0af7d382db888f1fde52bb395868e7c12a7a20debd527"
Jul 28 18:00:24.264616 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.264578    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"]
Jul 28 18:00:24.265059 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.264922    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed66c090-fcd4-4937-8927-249cbc596710" containerName="main"
Jul 28 18:00:24.265059 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.264935    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed66c090-fcd4-4937-8927-249cbc596710" containerName="main"
Jul 28 18:00:24.265059 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.264952    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70bcd703-5a46-450f-9a40-20198f8eaf48" containerName="main"
Jul 28 18:00:24.265059 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.264957    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="70bcd703-5a46-450f-9a40-20198f8eaf48" containerName="main"
Jul 28 18:00:24.265059 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.264965    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12b1bdbb-82e4-472c-99f6-b8d7ab244b11" containerName="main"
Jul 28 18:00:24.265059 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.264971    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="12b1bdbb-82e4-472c-99f6-b8d7ab244b11" containerName="main"
Jul 28 18:00:24.265059 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.265032    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed66c090-fcd4-4937-8927-249cbc596710" containerName="main"
Jul 28 18:00:24.265059 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.265039    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="70bcd703-5a46-450f-9a40-20198f8eaf48" containerName="main"
Jul 28 18:00:24.265059 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.265046    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="12b1bdbb-82e4-472c-99f6-b8d7ab244b11" containerName="main"
Jul 28 18:00:24.267983 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.267963    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"
Jul 28 18:00:24.270528 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.270504    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:00:24.270668 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.270504    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Jul 28 18:00:24.270804 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.270788    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Jul 28 18:00:24.271026 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.271010    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Jul 28 18:00:24.271332 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.271312    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 28 18:00:24.271599 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.271586    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-wng8v\""
Jul 28 18:00:24.300558 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.300533    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"]
Jul 28 18:00:24.455032 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.455001    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b152ff41-62e7-4b37-810a-ade233a9cd70-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw\" (UID: \"b152ff41-62e7-4b37-810a-ade233a9cd70\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"
Jul 28 18:00:24.455253 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.455036    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b152ff41-62e7-4b37-810a-ade233a9cd70-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw\" (UID: \"b152ff41-62e7-4b37-810a-ade233a9cd70\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"
Jul 28 18:00:24.455253 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.455070    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b152ff41-62e7-4b37-810a-ade233a9cd70-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw\" (UID: \"b152ff41-62e7-4b37-810a-ade233a9cd70\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"
Jul 28 18:00:24.455253 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.455188    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b152ff41-62e7-4b37-810a-ade233a9cd70-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw\" (UID: \"b152ff41-62e7-4b37-810a-ade233a9cd70\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"
Jul 28 18:00:24.455253 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.455241    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b152ff41-62e7-4b37-810a-ade233a9cd70-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw\" (UID: \"b152ff41-62e7-4b37-810a-ade233a9cd70\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"
Jul 28 18:00:24.455451 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.455287    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b152ff41-62e7-4b37-810a-ade233a9cd70-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw\" (UID: \"b152ff41-62e7-4b37-810a-ade233a9cd70\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"
Jul 28 18:00:24.455451 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.455313    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b152ff41-62e7-4b37-810a-ade233a9cd70-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw\" (UID: \"b152ff41-62e7-4b37-810a-ade233a9cd70\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"
Jul 28 18:00:24.455451 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.455364    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkpgt\" (UniqueName: \"kubernetes.io/projected/b152ff41-62e7-4b37-810a-ade233a9cd70-kube-api-access-rkpgt\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw\" (UID: \"b152ff41-62e7-4b37-810a-ade233a9cd70\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"
Jul 28 18:00:24.556068 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.555992    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b152ff41-62e7-4b37-810a-ade233a9cd70-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw\" (UID: \"b152ff41-62e7-4b37-810a-ade233a9cd70\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"
Jul 28 18:00:24.556068 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.556034    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b152ff41-62e7-4b37-810a-ade233a9cd70-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw\" (UID: \"b152ff41-62e7-4b37-810a-ade233a9cd70\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"
Jul 28 18:00:24.556068 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.556058    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b152ff41-62e7-4b37-810a-ade233a9cd70-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw\" (UID: \"b152ff41-62e7-4b37-810a-ade233a9cd70\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"
Jul 28 18:00:24.556386 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.556178    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b152ff41-62e7-4b37-810a-ade233a9cd70-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw\" (UID: \"b152ff41-62e7-4b37-810a-ade233a9cd70\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"
Jul 28 18:00:24.556386 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.556228    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkpgt\" (UniqueName: \"kubernetes.io/projected/b152ff41-62e7-4b37-810a-ade233a9cd70-kube-api-access-rkpgt\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw\" (UID: \"b152ff41-62e7-4b37-810a-ade233a9cd70\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"
Jul 28 18:00:24.556386 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.556287    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b152ff41-62e7-4b37-810a-ade233a9cd70-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw\" (UID: \"b152ff41-62e7-4b37-810a-ade233a9cd70\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"
Jul 28 18:00:24.556386 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.556318    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b152ff41-62e7-4b37-810a-ade233a9cd70-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw\" (UID: \"b152ff41-62e7-4b37-810a-ade233a9cd70\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"
Jul 28 18:00:24.556386 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.556358    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b152ff41-62e7-4b37-810a-ade233a9cd70-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw\" (UID: \"b152ff41-62e7-4b37-810a-ade233a9cd70\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"
Jul 28 18:00:24.556660 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.556434    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b152ff41-62e7-4b37-810a-ade233a9cd70-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw\" (UID: \"b152ff41-62e7-4b37-810a-ade233a9cd70\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"
Jul 28 18:00:24.556660 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.556542    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b152ff41-62e7-4b37-810a-ade233a9cd70-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw\" (UID: \"b152ff41-62e7-4b37-810a-ade233a9cd70\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"
Jul 28 18:00:24.556776 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.556696    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b152ff41-62e7-4b37-810a-ade233a9cd70-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw\" (UID: \"b152ff41-62e7-4b37-810a-ade233a9cd70\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"
Jul 28 18:00:24.556776 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.556702    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b152ff41-62e7-4b37-810a-ade233a9cd70-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw\" (UID: \"b152ff41-62e7-4b37-810a-ade233a9cd70\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"
Jul 28 18:00:24.556888 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.556846    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b152ff41-62e7-4b37-810a-ade233a9cd70-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw\" (UID: \"b152ff41-62e7-4b37-810a-ade233a9cd70\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"
Jul 28 18:00:24.558363 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.558335    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b152ff41-62e7-4b37-810a-ade233a9cd70-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw\" (UID: \"b152ff41-62e7-4b37-810a-ade233a9cd70\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"
Jul 28 18:00:24.558653 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.558629    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b152ff41-62e7-4b37-810a-ade233a9cd70-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw\" (UID: \"b152ff41-62e7-4b37-810a-ade233a9cd70\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"
Jul 28 18:00:24.590378 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.590353    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkpgt\" (UniqueName: \"kubernetes.io/projected/b152ff41-62e7-4b37-810a-ade233a9cd70-kube-api-access-rkpgt\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw\" (UID: \"b152ff41-62e7-4b37-810a-ade233a9cd70\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"
Jul 28 18:00:24.878269 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:24.878169    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"
Jul 28 18:00:25.030767 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:25.030740    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"]
Jul 28 18:00:25.031584 ip-10-0-135-177 kubenswrapper[2543]: W0728 18:00:25.031559    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb152ff41_62e7_4b37_810a_ade233a9cd70.slice/crio-676ff5cbc34f18e8918780e8d01b44f8906685749be3a529a6146bc73c66ee70 WatchSource:0}: Error finding container 676ff5cbc34f18e8918780e8d01b44f8906685749be3a529a6146bc73c66ee70: Status 404 returned error can't find the container with id 676ff5cbc34f18e8918780e8d01b44f8906685749be3a529a6146bc73c66ee70
Jul 28 18:00:25.477699 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:25.477662    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw" event={"ID":"b152ff41-62e7-4b37-810a-ade233a9cd70","Type":"ContainerStarted","Data":"2835d61367d752112bab3e596e4cf5224609066f1656d252d8872af2afa7f425"}
Jul 28 18:00:25.478159 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:25.477717    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw" event={"ID":"b152ff41-62e7-4b37-810a-ade233a9cd70","Type":"ContainerStarted","Data":"676ff5cbc34f18e8918780e8d01b44f8906685749be3a529a6146bc73c66ee70"}
Jul 28 18:00:34.523947 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:34.523910    2543 generic.go:356] "Generic (PLEG): container finished" podID="b152ff41-62e7-4b37-810a-ade233a9cd70" containerID="2835d61367d752112bab3e596e4cf5224609066f1656d252d8872af2afa7f425" exitCode=0
Jul 28 18:00:34.524462 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:34.523988    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw" event={"ID":"b152ff41-62e7-4b37-810a-ade233a9cd70","Type":"ContainerDied","Data":"2835d61367d752112bab3e596e4cf5224609066f1656d252d8872af2afa7f425"}
Jul 28 18:00:35.531867 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:35.531826    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw" event={"ID":"b152ff41-62e7-4b37-810a-ade233a9cd70","Type":"ContainerStarted","Data":"68ef3aed34fc90261fb8a62a9fbcd168b097cb8c937b3cfcc860fe69dfe93ee7"}
Jul 28 18:00:35.574433 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:35.574378    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw" podStartSLOduration=2.810218901 podStartE2EDuration="11.574361457s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.764142556s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:00:35.573539943 +0000 UTC m=+2238.979813696" watchObservedRunningTime="2026-07-28 18:00:35.574361457 +0000 UTC m=+2238.980635197"
Jul 28 18:00:35.878908 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:35.878815    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"
Jul 28 18:00:44.879299 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:44.879205    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"
Jul 28 18:00:44.880864 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:44.880836    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw" podUID="b152ff41-62e7-4b37-810a-ade233a9cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 28 18:00:54.879229 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:00:54.879178    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw" podUID="b152ff41-62e7-4b37-810a-ade233a9cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 28 18:01:04.879247 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:01:04.879204    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw" podUID="b152ff41-62e7-4b37-810a-ade233a9cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 28 18:01:14.878991 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:01:14.878934    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw" podUID="b152ff41-62e7-4b37-810a-ade233a9cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 28 18:01:24.878986 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:01:24.878936    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw" podUID="b152ff41-62e7-4b37-810a-ade233a9cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 28 18:01:34.878966 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:01:34.878914    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw" podUID="b152ff41-62e7-4b37-810a-ade233a9cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 28 18:01:44.879528 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:01:44.879476    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw" podUID="b152ff41-62e7-4b37-810a-ade233a9cd70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Jul 28 18:01:54.889359 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:01:54.889328    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"
Jul 28 18:01:54.897372 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:01:54.897338    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"
Jul 28 18:02:20.198359 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:20.198279    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"]
Jul 28 18:02:35.221278 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:35.221235    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw" podUID="b152ff41-62e7-4b37-810a-ade233a9cd70" containerName="main" containerID="cri-o://68ef3aed34fc90261fb8a62a9fbcd168b097cb8c937b3cfcc860fe69dfe93ee7" gracePeriod=45
Jul 28 18:02:35.872360 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:35.872331    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"
Jul 28 18:02:36.005617 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.005581    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b152ff41-62e7-4b37-810a-ade233a9cd70-dshm\") pod \"b152ff41-62e7-4b37-810a-ade233a9cd70\" (UID: \"b152ff41-62e7-4b37-810a-ade233a9cd70\") "
Jul 28 18:02:36.005822 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.005628    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkpgt\" (UniqueName: \"kubernetes.io/projected/b152ff41-62e7-4b37-810a-ade233a9cd70-kube-api-access-rkpgt\") pod \"b152ff41-62e7-4b37-810a-ade233a9cd70\" (UID: \"b152ff41-62e7-4b37-810a-ade233a9cd70\") "
Jul 28 18:02:36.005822 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.005664    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b152ff41-62e7-4b37-810a-ade233a9cd70-tls-certs\") pod \"b152ff41-62e7-4b37-810a-ade233a9cd70\" (UID: \"b152ff41-62e7-4b37-810a-ade233a9cd70\") "
Jul 28 18:02:36.005822 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.005710    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b152ff41-62e7-4b37-810a-ade233a9cd70-cabundle-cert\") pod \"b152ff41-62e7-4b37-810a-ade233a9cd70\" (UID: \"b152ff41-62e7-4b37-810a-ade233a9cd70\") "
Jul 28 18:02:36.005822 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.005780    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b152ff41-62e7-4b37-810a-ade233a9cd70-kserve-provision-location\") pod \"b152ff41-62e7-4b37-810a-ade233a9cd70\" (UID: \"b152ff41-62e7-4b37-810a-ade233a9cd70\") "
Jul 28 18:02:36.005822 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.005801    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b152ff41-62e7-4b37-810a-ade233a9cd70-tmp-dir\") pod \"b152ff41-62e7-4b37-810a-ade233a9cd70\" (UID: \"b152ff41-62e7-4b37-810a-ade233a9cd70\") "
Jul 28 18:02:36.006099 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.005842    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b152ff41-62e7-4b37-810a-ade233a9cd70-home\") pod \"b152ff41-62e7-4b37-810a-ade233a9cd70\" (UID: \"b152ff41-62e7-4b37-810a-ade233a9cd70\") "
Jul 28 18:02:36.006099 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.005919    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b152ff41-62e7-4b37-810a-ade233a9cd70-model-cache\") pod \"b152ff41-62e7-4b37-810a-ade233a9cd70\" (UID: \"b152ff41-62e7-4b37-810a-ade233a9cd70\") "
Jul 28 18:02:36.006248 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.006203    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b152ff41-62e7-4b37-810a-ade233a9cd70-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b152ff41-62e7-4b37-810a-ade233a9cd70" (UID: "b152ff41-62e7-4b37-810a-ade233a9cd70"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:02:36.006307 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.006272    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b152ff41-62e7-4b37-810a-ade233a9cd70-home" (OuterVolumeSpecName: "home") pod "b152ff41-62e7-4b37-810a-ade233a9cd70" (UID: "b152ff41-62e7-4b37-810a-ade233a9cd70"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:02:36.006368 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.006334    2543 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b152ff41-62e7-4b37-810a-ade233a9cd70-cabundle-cert\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:02:36.006562 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.006539    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b152ff41-62e7-4b37-810a-ade233a9cd70-model-cache" (OuterVolumeSpecName: "model-cache") pod "b152ff41-62e7-4b37-810a-ade233a9cd70" (UID: "b152ff41-62e7-4b37-810a-ade233a9cd70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:02:36.008277 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.008248    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b152ff41-62e7-4b37-810a-ade233a9cd70-dshm" (OuterVolumeSpecName: "dshm") pod "b152ff41-62e7-4b37-810a-ade233a9cd70" (UID: "b152ff41-62e7-4b37-810a-ade233a9cd70"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:02:36.008404 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.008369    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b152ff41-62e7-4b37-810a-ade233a9cd70-kube-api-access-rkpgt" (OuterVolumeSpecName: "kube-api-access-rkpgt") pod "b152ff41-62e7-4b37-810a-ade233a9cd70" (UID: "b152ff41-62e7-4b37-810a-ade233a9cd70"). InnerVolumeSpecName "kube-api-access-rkpgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:02:36.008550 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.008525    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b152ff41-62e7-4b37-810a-ade233a9cd70-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b152ff41-62e7-4b37-810a-ade233a9cd70" (UID: "b152ff41-62e7-4b37-810a-ade233a9cd70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:02:36.018192 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.018154    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b152ff41-62e7-4b37-810a-ade233a9cd70-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b152ff41-62e7-4b37-810a-ade233a9cd70" (UID: "b152ff41-62e7-4b37-810a-ade233a9cd70"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:02:36.068195 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.068157    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b152ff41-62e7-4b37-810a-ade233a9cd70-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b152ff41-62e7-4b37-810a-ade233a9cd70" (UID: "b152ff41-62e7-4b37-810a-ade233a9cd70"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:02:36.107774 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.107746    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b152ff41-62e7-4b37-810a-ade233a9cd70-kserve-provision-location\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:02:36.107774 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.107772    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b152ff41-62e7-4b37-810a-ade233a9cd70-tmp-dir\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:02:36.107774 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.107782    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b152ff41-62e7-4b37-810a-ade233a9cd70-home\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:02:36.108002 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.107790    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b152ff41-62e7-4b37-810a-ade233a9cd70-model-cache\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:02:36.108002 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.107800    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b152ff41-62e7-4b37-810a-ade233a9cd70-dshm\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:02:36.108002 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.107807    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkpgt\" (UniqueName: \"kubernetes.io/projected/b152ff41-62e7-4b37-810a-ade233a9cd70-kube-api-access-rkpgt\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:02:36.108002 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.107815    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b152ff41-62e7-4b37-810a-ade233a9cd70-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:02:36.183935 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.183901    2543 generic.go:356] "Generic (PLEG): container finished" podID="b152ff41-62e7-4b37-810a-ade233a9cd70" containerID="68ef3aed34fc90261fb8a62a9fbcd168b097cb8c937b3cfcc860fe69dfe93ee7" exitCode=0
Jul 28 18:02:36.184122 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.183978    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"
Jul 28 18:02:36.184122 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.183993    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw" event={"ID":"b152ff41-62e7-4b37-810a-ade233a9cd70","Type":"ContainerDied","Data":"68ef3aed34fc90261fb8a62a9fbcd168b097cb8c937b3cfcc860fe69dfe93ee7"}
Jul 28 18:02:36.184122 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.184087    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw" event={"ID":"b152ff41-62e7-4b37-810a-ade233a9cd70","Type":"ContainerDied","Data":"676ff5cbc34f18e8918780e8d01b44f8906685749be3a529a6146bc73c66ee70"}
Jul 28 18:02:36.184122 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.184117    2543 scope.go:117] "RemoveContainer" containerID="68ef3aed34fc90261fb8a62a9fbcd168b097cb8c937b3cfcc860fe69dfe93ee7"
Jul 28 18:02:36.193822 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.193806    2543 scope.go:117] "RemoveContainer" containerID="2835d61367d752112bab3e596e4cf5224609066f1656d252d8872af2afa7f425"
Jul 28 18:02:36.201548 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.201532    2543 scope.go:117] "RemoveContainer" containerID="68ef3aed34fc90261fb8a62a9fbcd168b097cb8c937b3cfcc860fe69dfe93ee7"
Jul 28 18:02:36.201820 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:02:36.201799    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68ef3aed34fc90261fb8a62a9fbcd168b097cb8c937b3cfcc860fe69dfe93ee7\": container with ID starting with 68ef3aed34fc90261fb8a62a9fbcd168b097cb8c937b3cfcc860fe69dfe93ee7 not found: ID does not exist" containerID="68ef3aed34fc90261fb8a62a9fbcd168b097cb8c937b3cfcc860fe69dfe93ee7"
Jul 28 18:02:36.201886 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.201829    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68ef3aed34fc90261fb8a62a9fbcd168b097cb8c937b3cfcc860fe69dfe93ee7"} err="failed to get container status \"68ef3aed34fc90261fb8a62a9fbcd168b097cb8c937b3cfcc860fe69dfe93ee7\": rpc error: code = NotFound desc = could not find container \"68ef3aed34fc90261fb8a62a9fbcd168b097cb8c937b3cfcc860fe69dfe93ee7\": container with ID starting with 68ef3aed34fc90261fb8a62a9fbcd168b097cb8c937b3cfcc860fe69dfe93ee7 not found: ID does not exist"
Jul 28 18:02:36.201886 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.201846    2543 scope.go:117] "RemoveContainer" containerID="2835d61367d752112bab3e596e4cf5224609066f1656d252d8872af2afa7f425"
Jul 28 18:02:36.202057 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:02:36.202041    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2835d61367d752112bab3e596e4cf5224609066f1656d252d8872af2afa7f425\": container with ID starting with 2835d61367d752112bab3e596e4cf5224609066f1656d252d8872af2afa7f425 not found: ID does not exist" containerID="2835d61367d752112bab3e596e4cf5224609066f1656d252d8872af2afa7f425"
Jul 28 18:02:36.202103 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.202063    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2835d61367d752112bab3e596e4cf5224609066f1656d252d8872af2afa7f425"} err="failed to get container status \"2835d61367d752112bab3e596e4cf5224609066f1656d252d8872af2afa7f425\": rpc error: code = NotFound desc = could not find container \"2835d61367d752112bab3e596e4cf5224609066f1656d252d8872af2afa7f425\": container with ID starting with 2835d61367d752112bab3e596e4cf5224609066f1656d252d8872af2afa7f425 not found: ID does not exist"
Jul 28 18:02:36.208161 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.208118    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"]
Jul 28 18:02:36.211893 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:36.211874    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-f5dhw"]
Jul 28 18:02:37.152113 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:37.152081    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b152ff41-62e7-4b37-810a-ade233a9cd70" path="/var/lib/kubelet/pods/b152ff41-62e7-4b37-810a-ade233a9cd70/volumes"
Jul 28 18:02:51.921798 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:51.921762    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"]
Jul 28 18:02:51.922290 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:51.922069    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b152ff41-62e7-4b37-810a-ade233a9cd70" containerName="main"
Jul 28 18:02:51.922290 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:51.922080    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="b152ff41-62e7-4b37-810a-ade233a9cd70" containerName="main"
Jul 28 18:02:51.922290 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:51.922102    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b152ff41-62e7-4b37-810a-ade233a9cd70" containerName="storage-initializer"
Jul 28 18:02:51.922290 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:51.922108    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="b152ff41-62e7-4b37-810a-ade233a9cd70" containerName="storage-initializer"
Jul 28 18:02:51.922290 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:51.922174    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="b152ff41-62e7-4b37-810a-ade233a9cd70" containerName="main"
Jul 28 18:02:51.925401 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:51.925384    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"
Jul 28 18:02:51.951904 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:51.951872    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Jul 28 18:02:51.951904 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:51.951898    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:02:51.952109 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:51.951923    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Jul 28 18:02:51.952255 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:51.952239    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-nxtkr\""
Jul 28 18:02:51.953634 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:51.953613    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 28 18:02:51.954535 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:51.954519    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Jul 28 18:02:52.008739 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:52.008703    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"]
Jul 28 18:02:52.043479 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:52.043448    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d434cc-d03a-4ef4-9996-adbdf81e583e-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7\" (UID: \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"
Jul 28 18:02:52.043635 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:52.043483    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25hzm\" (UniqueName: \"kubernetes.io/projected/a2d434cc-d03a-4ef4-9996-adbdf81e583e-kube-api-access-25hzm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7\" (UID: \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"
Jul 28 18:02:52.043635 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:52.043535    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d434cc-d03a-4ef4-9996-adbdf81e583e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7\" (UID: \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"
Jul 28 18:02:52.043635 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:52.043552    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2d434cc-d03a-4ef4-9996-adbdf81e583e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7\" (UID: \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"
Jul 28 18:02:52.043635 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:52.043570    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d434cc-d03a-4ef4-9996-adbdf81e583e-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7\" (UID: \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"
Jul 28 18:02:52.043635 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:52.043589    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2d434cc-d03a-4ef4-9996-adbdf81e583e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7\" (UID: \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"
Jul 28 18:02:52.043635 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:52.043606    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d434cc-d03a-4ef4-9996-adbdf81e583e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7\" (UID: \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"
Jul 28 18:02:52.043844 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:52.043677    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d434cc-d03a-4ef4-9996-adbdf81e583e-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7\" (UID: \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"
Jul 28 18:02:52.144506 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:52.144474    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d434cc-d03a-4ef4-9996-adbdf81e583e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7\" (UID: \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"
Jul 28 18:02:52.144506 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:52.144508    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2d434cc-d03a-4ef4-9996-adbdf81e583e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7\" (UID: \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"
Jul 28 18:02:52.144737 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:52.144529    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d434cc-d03a-4ef4-9996-adbdf81e583e-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7\" (UID: \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"
Jul 28 18:02:52.144737 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:52.144549    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2d434cc-d03a-4ef4-9996-adbdf81e583e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7\" (UID: \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"
Jul 28 18:02:52.144737 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:52.144569    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d434cc-d03a-4ef4-9996-adbdf81e583e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7\" (UID: \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"
Jul 28 18:02:52.144737 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:52.144605    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d434cc-d03a-4ef4-9996-adbdf81e583e-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7\" (UID: \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"
Jul 28 18:02:52.144737 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:52.144629    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d434cc-d03a-4ef4-9996-adbdf81e583e-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7\" (UID: \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"
Jul 28 18:02:52.144737 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:52.144647    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25hzm\" (UniqueName: \"kubernetes.io/projected/a2d434cc-d03a-4ef4-9996-adbdf81e583e-kube-api-access-25hzm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7\" (UID: \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"
Jul 28 18:02:52.145065 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:52.144943    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d434cc-d03a-4ef4-9996-adbdf81e583e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7\" (UID: \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"
Jul 28 18:02:52.145065 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:52.144964    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2d434cc-d03a-4ef4-9996-adbdf81e583e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7\" (UID: \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"
Jul 28 18:02:52.145065 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:52.145043    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d434cc-d03a-4ef4-9996-adbdf81e583e-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7\" (UID: \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"
Jul 28 18:02:52.145275 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:52.145245    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d434cc-d03a-4ef4-9996-adbdf81e583e-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7\" (UID: \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"
Jul 28 18:02:52.145384 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:52.145365    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2d434cc-d03a-4ef4-9996-adbdf81e583e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7\" (UID: \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"
Jul 28 18:02:52.147120 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:52.147097    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d434cc-d03a-4ef4-9996-adbdf81e583e-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7\" (UID: \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"
Jul 28 18:02:52.147378 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:52.147357    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d434cc-d03a-4ef4-9996-adbdf81e583e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7\" (UID: \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"
Jul 28 18:02:52.154710 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:52.154673    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25hzm\" (UniqueName: \"kubernetes.io/projected/a2d434cc-d03a-4ef4-9996-adbdf81e583e-kube-api-access-25hzm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7\" (UID: \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"
Jul 28 18:02:52.235620 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:52.235537    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"
Jul 28 18:02:52.437940 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:52.437915    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"]
Jul 28 18:02:52.440198 ip-10-0-135-177 kubenswrapper[2543]: W0728 18:02:52.440164    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2d434cc_d03a_4ef4_9996_adbdf81e583e.slice/crio-fcb62e0cc56d0cc677c4caf4d0604c70d39801e367bfc583bef5ea8f358ef6fa WatchSource:0}: Error finding container fcb62e0cc56d0cc677c4caf4d0604c70d39801e367bfc583bef5ea8f358ef6fa: Status 404 returned error can't find the container with id fcb62e0cc56d0cc677c4caf4d0604c70d39801e367bfc583bef5ea8f358ef6fa
Jul 28 18:02:53.269321 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:53.269279    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7" event={"ID":"a2d434cc-d03a-4ef4-9996-adbdf81e583e","Type":"ContainerStarted","Data":"c14dac7c5fb8dc47edf9a2305d0b763800f27c703669543181a41c878408ad1f"}
Jul 28 18:02:53.269795 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:02:53.269334    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7" event={"ID":"a2d434cc-d03a-4ef4-9996-adbdf81e583e","Type":"ContainerStarted","Data":"fcb62e0cc56d0cc677c4caf4d0604c70d39801e367bfc583bef5ea8f358ef6fa"}
Jul 28 18:03:00.306383 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:03:00.306347    2543 generic.go:356] "Generic (PLEG): container finished" podID="a2d434cc-d03a-4ef4-9996-adbdf81e583e" containerID="c14dac7c5fb8dc47edf9a2305d0b763800f27c703669543181a41c878408ad1f" exitCode=0
Jul 28 18:03:00.306857 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:03:00.306421    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7" event={"ID":"a2d434cc-d03a-4ef4-9996-adbdf81e583e","Type":"ContainerDied","Data":"c14dac7c5fb8dc47edf9a2305d0b763800f27c703669543181a41c878408ad1f"}
Jul 28 18:03:01.313085 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:03:01.313052    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7" event={"ID":"a2d434cc-d03a-4ef4-9996-adbdf81e583e","Type":"ContainerStarted","Data":"c15744b09270f1ac1c1a5af0f05c8f2af15c6d0ab71b55ac8d0f230032ab423f"}
Jul 28 18:03:01.339463 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:03:01.339413    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7" podStartSLOduration=3.37778735 podStartE2EDuration="10.339396977s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.961609627s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:03:01.336864875 +0000 UTC m=+2384.743138618" watchObservedRunningTime="2026-07-28 18:03:01.339396977 +0000 UTC m=+2384.745670719"
Jul 28 18:03:02.235936 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:03:02.235892    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"
Jul 28 18:03:02.236127 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:03:02.235984    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"
Jul 28 18:03:02.237721 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:03:02.237674    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7" podUID="a2d434cc-d03a-4ef4-9996-adbdf81e583e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 28 18:03:12.236980 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:03:12.236939    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7" podUID="a2d434cc-d03a-4ef4-9996-adbdf81e583e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 28 18:03:22.236770 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:03:22.236728    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7" podUID="a2d434cc-d03a-4ef4-9996-adbdf81e583e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 28 18:03:32.236976 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:03:32.236921    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7" podUID="a2d434cc-d03a-4ef4-9996-adbdf81e583e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 28 18:03:42.236729 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:03:42.236636    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7" podUID="a2d434cc-d03a-4ef4-9996-adbdf81e583e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 28 18:03:52.236356 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:03:52.236312    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7" podUID="a2d434cc-d03a-4ef4-9996-adbdf81e583e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 28 18:04:02.236946 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:02.236906    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7" podUID="a2d434cc-d03a-4ef4-9996-adbdf81e583e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 28 18:04:12.236908 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:12.236846    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7" podUID="a2d434cc-d03a-4ef4-9996-adbdf81e583e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 28 18:04:22.246689 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:22.246644    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"
Jul 28 18:04:22.756626 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:22.756592    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"
Jul 28 18:04:24.591167 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:24.591115    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"]
Jul 28 18:04:39.781195 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:39.781090    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7" podUID="a2d434cc-d03a-4ef4-9996-adbdf81e583e" containerName="main" containerID="cri-o://c15744b09270f1ac1c1a5af0f05c8f2af15c6d0ab71b55ac8d0f230032ab423f" gracePeriod=45
Jul 28 18:04:40.340161 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.340119    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"
Jul 28 18:04:40.407062 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.407016    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d434cc-d03a-4ef4-9996-adbdf81e583e-tls-certs\") pod \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\" (UID: \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\") "
Jul 28 18:04:40.407264 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.407083    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d434cc-d03a-4ef4-9996-adbdf81e583e-home\") pod \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\" (UID: \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\") "
Jul 28 18:04:40.407264 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.407108    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d434cc-d03a-4ef4-9996-adbdf81e583e-dshm\") pod \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\" (UID: \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\") "
Jul 28 18:04:40.407264 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.407161    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25hzm\" (UniqueName: \"kubernetes.io/projected/a2d434cc-d03a-4ef4-9996-adbdf81e583e-kube-api-access-25hzm\") pod \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\" (UID: \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\") "
Jul 28 18:04:40.407264 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.407185    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2d434cc-d03a-4ef4-9996-adbdf81e583e-cabundle-cert\") pod \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\" (UID: \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\") "
Jul 28 18:04:40.407264 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.407217    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2d434cc-d03a-4ef4-9996-adbdf81e583e-kserve-provision-location\") pod \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\" (UID: \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\") "
Jul 28 18:04:40.407550 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.407288    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d434cc-d03a-4ef4-9996-adbdf81e583e-tmp-dir\") pod \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\" (UID: \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\") "
Jul 28 18:04:40.407550 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.407326    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d434cc-d03a-4ef4-9996-adbdf81e583e-model-cache\") pod \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\" (UID: \"a2d434cc-d03a-4ef4-9996-adbdf81e583e\") "
Jul 28 18:04:40.407730 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.407683    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d434cc-d03a-4ef4-9996-adbdf81e583e-home" (OuterVolumeSpecName: "home") pod "a2d434cc-d03a-4ef4-9996-adbdf81e583e" (UID: "a2d434cc-d03a-4ef4-9996-adbdf81e583e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:04:40.407858 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.407720    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2d434cc-d03a-4ef4-9996-adbdf81e583e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a2d434cc-d03a-4ef4-9996-adbdf81e583e" (UID: "a2d434cc-d03a-4ef4-9996-adbdf81e583e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:04:40.407858 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.407732    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d434cc-d03a-4ef4-9996-adbdf81e583e-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2d434cc-d03a-4ef4-9996-adbdf81e583e" (UID: "a2d434cc-d03a-4ef4-9996-adbdf81e583e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:04:40.407990 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.407891    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d434cc-d03a-4ef4-9996-adbdf81e583e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2d434cc-d03a-4ef4-9996-adbdf81e583e" (UID: "a2d434cc-d03a-4ef4-9996-adbdf81e583e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:04:40.409506 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.409474    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2d434cc-d03a-4ef4-9996-adbdf81e583e-kube-api-access-25hzm" (OuterVolumeSpecName: "kube-api-access-25hzm") pod "a2d434cc-d03a-4ef4-9996-adbdf81e583e" (UID: "a2d434cc-d03a-4ef4-9996-adbdf81e583e"). InnerVolumeSpecName "kube-api-access-25hzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:04:40.409616 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.409548    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2d434cc-d03a-4ef4-9996-adbdf81e583e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2d434cc-d03a-4ef4-9996-adbdf81e583e" (UID: "a2d434cc-d03a-4ef4-9996-adbdf81e583e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:04:40.409679 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.409653    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d434cc-d03a-4ef4-9996-adbdf81e583e-dshm" (OuterVolumeSpecName: "dshm") pod "a2d434cc-d03a-4ef4-9996-adbdf81e583e" (UID: "a2d434cc-d03a-4ef4-9996-adbdf81e583e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:04:40.470550 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.470471    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d434cc-d03a-4ef4-9996-adbdf81e583e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a2d434cc-d03a-4ef4-9996-adbdf81e583e" (UID: "a2d434cc-d03a-4ef4-9996-adbdf81e583e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:04:40.508222 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.508178    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2d434cc-d03a-4ef4-9996-adbdf81e583e-home\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:04:40.508222 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.508215    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2d434cc-d03a-4ef4-9996-adbdf81e583e-dshm\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:04:40.508222 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.508225    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25hzm\" (UniqueName: \"kubernetes.io/projected/a2d434cc-d03a-4ef4-9996-adbdf81e583e-kube-api-access-25hzm\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:04:40.508222 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.508235    2543 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2d434cc-d03a-4ef4-9996-adbdf81e583e-cabundle-cert\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:04:40.508476 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.508244    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2d434cc-d03a-4ef4-9996-adbdf81e583e-kserve-provision-location\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:04:40.508476 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.508253    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2d434cc-d03a-4ef4-9996-adbdf81e583e-tmp-dir\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:04:40.508476 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.508262    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2d434cc-d03a-4ef4-9996-adbdf81e583e-model-cache\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:04:40.508476 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.508269    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2d434cc-d03a-4ef4-9996-adbdf81e583e-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:04:40.845623 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.845593    2543 generic.go:356] "Generic (PLEG): container finished" podID="a2d434cc-d03a-4ef4-9996-adbdf81e583e" containerID="c15744b09270f1ac1c1a5af0f05c8f2af15c6d0ab71b55ac8d0f230032ab423f" exitCode=0
Jul 28 18:04:40.846043 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.845645    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"
Jul 28 18:04:40.846043 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.845683    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7" event={"ID":"a2d434cc-d03a-4ef4-9996-adbdf81e583e","Type":"ContainerDied","Data":"c15744b09270f1ac1c1a5af0f05c8f2af15c6d0ab71b55ac8d0f230032ab423f"}
Jul 28 18:04:40.846043 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.845765    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7" event={"ID":"a2d434cc-d03a-4ef4-9996-adbdf81e583e","Type":"ContainerDied","Data":"fcb62e0cc56d0cc677c4caf4d0604c70d39801e367bfc583bef5ea8f358ef6fa"}
Jul 28 18:04:40.846043 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.845790    2543 scope.go:117] "RemoveContainer" containerID="c15744b09270f1ac1c1a5af0f05c8f2af15c6d0ab71b55ac8d0f230032ab423f"
Jul 28 18:04:40.854830 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.854814    2543 scope.go:117] "RemoveContainer" containerID="c14dac7c5fb8dc47edf9a2305d0b763800f27c703669543181a41c878408ad1f"
Jul 28 18:04:40.862979 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.862962    2543 scope.go:117] "RemoveContainer" containerID="c15744b09270f1ac1c1a5af0f05c8f2af15c6d0ab71b55ac8d0f230032ab423f"
Jul 28 18:04:40.863221 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:04:40.863201    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c15744b09270f1ac1c1a5af0f05c8f2af15c6d0ab71b55ac8d0f230032ab423f\": container with ID starting with c15744b09270f1ac1c1a5af0f05c8f2af15c6d0ab71b55ac8d0f230032ab423f not found: ID does not exist" containerID="c15744b09270f1ac1c1a5af0f05c8f2af15c6d0ab71b55ac8d0f230032ab423f"
Jul 28 18:04:40.863291 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.863232    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c15744b09270f1ac1c1a5af0f05c8f2af15c6d0ab71b55ac8d0f230032ab423f"} err="failed to get container status \"c15744b09270f1ac1c1a5af0f05c8f2af15c6d0ab71b55ac8d0f230032ab423f\": rpc error: code = NotFound desc = could not find container \"c15744b09270f1ac1c1a5af0f05c8f2af15c6d0ab71b55ac8d0f230032ab423f\": container with ID starting with c15744b09270f1ac1c1a5af0f05c8f2af15c6d0ab71b55ac8d0f230032ab423f not found: ID does not exist"
Jul 28 18:04:40.863291 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.863255    2543 scope.go:117] "RemoveContainer" containerID="c14dac7c5fb8dc47edf9a2305d0b763800f27c703669543181a41c878408ad1f"
Jul 28 18:04:40.863469 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:04:40.863454    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c14dac7c5fb8dc47edf9a2305d0b763800f27c703669543181a41c878408ad1f\": container with ID starting with c14dac7c5fb8dc47edf9a2305d0b763800f27c703669543181a41c878408ad1f not found: ID does not exist" containerID="c14dac7c5fb8dc47edf9a2305d0b763800f27c703669543181a41c878408ad1f"
Jul 28 18:04:40.863514 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.863475    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c14dac7c5fb8dc47edf9a2305d0b763800f27c703669543181a41c878408ad1f"} err="failed to get container status \"c14dac7c5fb8dc47edf9a2305d0b763800f27c703669543181a41c878408ad1f\": rpc error: code = NotFound desc = could not find container \"c14dac7c5fb8dc47edf9a2305d0b763800f27c703669543181a41c878408ad1f\": container with ID starting with c14dac7c5fb8dc47edf9a2305d0b763800f27c703669543181a41c878408ad1f not found: ID does not exist"
Jul 28 18:04:40.873327 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.873306    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"]
Jul 28 18:04:40.879705 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:40.879685    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-vbwr7"]
Jul 28 18:04:41.153468 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:41.153393    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2d434cc-d03a-4ef4-9996-adbdf81e583e" path="/var/lib/kubelet/pods/a2d434cc-d03a-4ef4-9996-adbdf81e583e/volumes"
Jul 28 18:04:48.007685 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:48.007649    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-8zf46"]
Jul 28 18:04:48.008188 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:48.007978    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2d434cc-d03a-4ef4-9996-adbdf81e583e" containerName="storage-initializer"
Jul 28 18:04:48.008188 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:48.007989    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2d434cc-d03a-4ef4-9996-adbdf81e583e" containerName="storage-initializer"
Jul 28 18:04:48.008188 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:48.007997    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2d434cc-d03a-4ef4-9996-adbdf81e583e" containerName="main"
Jul 28 18:04:48.008188 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:48.008003    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2d434cc-d03a-4ef4-9996-adbdf81e583e" containerName="main"
Jul 28 18:04:48.008188 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:48.008050    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2d434cc-d03a-4ef4-9996-adbdf81e583e" containerName="main"
Jul 28 18:04:48.012496 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:48.012480    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-8zf46"
Jul 28 18:04:48.016825 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:48.016803    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"kube-root-ca.crt\""
Jul 28 18:04:48.016955 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:48.016938    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"default-dockercfg-6t8zk\""
Jul 28 18:04:48.017638 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:48.017619    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"openshift-service-ca.crt\""
Jul 28 18:04:48.047529 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:48.047502    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-8zf46"]
Jul 28 18:04:48.071386 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:48.071358    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr5df\" (UniqueName: \"kubernetes.io/projected/e5a36d21-8daa-4c48-9625-80865a269e00-kube-api-access-tr5df\") pod \"e2e-pvc-model-download-8zf46\" (UID: \"e5a36d21-8daa-4c48-9625-80865a269e00\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-8zf46"
Jul 28 18:04:48.172375 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:48.172339    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr5df\" (UniqueName: \"kubernetes.io/projected/e5a36d21-8daa-4c48-9625-80865a269e00-kube-api-access-tr5df\") pod \"e2e-pvc-model-download-8zf46\" (UID: \"e5a36d21-8daa-4c48-9625-80865a269e00\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-8zf46"
Jul 28 18:04:48.181734 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:48.181704    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr5df\" (UniqueName: \"kubernetes.io/projected/e5a36d21-8daa-4c48-9625-80865a269e00-kube-api-access-tr5df\") pod \"e2e-pvc-model-download-8zf46\" (UID: \"e5a36d21-8daa-4c48-9625-80865a269e00\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-8zf46"
Jul 28 18:04:50.899228 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:50.899193    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-0cd2a7f4-dab0-4dd1-afea-3be8a9762a2e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02faa6629b6a9a787\") pod \"e2e-pvc-model-download-8zf46\" (UID: \"e5a36d21-8daa-4c48-9625-80865a269e00\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-8zf46"
Jul 28 18:04:50.908011 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:50.907987    2543 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-0cd2a7f4-dab0-4dd1-afea-3be8a9762a2e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02faa6629b6a9a787\") pod \"e2e-pvc-model-download-8zf46\" (UID: \"e5a36d21-8daa-4c48-9625-80865a269e00\") device path: \"\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-8zf46"
Jul 28 18:04:51.000292 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:51.000261    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-0cd2a7f4-dab0-4dd1-afea-3be8a9762a2e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02faa6629b6a9a787\") pod \"e2e-pvc-model-download-8zf46\" (UID: \"e5a36d21-8daa-4c48-9625-80865a269e00\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-8zf46"
Jul 28 18:04:51.000415 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:51.000395    2543 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-0cd2a7f4-dab0-4dd1-afea-3be8a9762a2e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02faa6629b6a9a787\") pod \"e2e-pvc-model-download-8zf46\" (UID: \"e5a36d21-8daa-4c48-9625-80865a269e00\") DevicePath \"\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-8zf46"
Jul 28 18:04:51.004042 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:51.004024    2543 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-0cd2a7f4-dab0-4dd1-afea-3be8a9762a2e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02faa6629b6a9a787\") pod \"e2e-pvc-model-download-8zf46\" (UID: \"e5a36d21-8daa-4c48-9625-80865a269e00\") DevicePath \"csi-f9ba76048ba6b415b5b327393e0b9189fe10d1d3d3390d04e07484bb43e98473\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-8zf46"
Jul 28 18:04:51.218632 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:51.218600    2543 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-0cd2a7f4-dab0-4dd1-afea-3be8a9762a2e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02faa6629b6a9a787\") pod \"e2e-pvc-model-download-8zf46\" (UID: \"e5a36d21-8daa-4c48-9625-80865a269e00\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/4603f6215567b3d7a9973d53157cdef0c0cbd9e8b44a99d35c810edc3956bb7a/globalmount\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-8zf46"
Jul 28 18:04:51.233302 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:51.233277    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-0cd2a7f4-dab0-4dd1-afea-3be8a9762a2e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02faa6629b6a9a787\") pod \"e2e-pvc-model-download-8zf46\" (UID: \"e5a36d21-8daa-4c48-9625-80865a269e00\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-8zf46"
Jul 28 18:04:51.321251 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:51.321214    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-8zf46"
Jul 28 18:04:51.463529 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:51.463502    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-8zf46"]
Jul 28 18:04:51.466370 ip-10-0-135-177 kubenswrapper[2543]: W0728 18:04:51.466339    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5a36d21_8daa_4c48_9625_80865a269e00.slice/crio-04ca22cb52de58c0be4ab98be639aaa344248790e7100576c29903eed41d44cd WatchSource:0}: Error finding container 04ca22cb52de58c0be4ab98be639aaa344248790e7100576c29903eed41d44cd: Status 404 returned error can't find the container with id 04ca22cb52de58c0be4ab98be639aaa344248790e7100576c29903eed41d44cd
Jul 28 18:04:51.901970 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:51.901928    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-8zf46" event={"ID":"e5a36d21-8daa-4c48-9625-80865a269e00","Type":"ContainerStarted","Data":"8f6e4893bcb44bd5cb268436f8dbd50c77b29221d35c8d9fb19048356f4d0231"}
Jul 28 18:04:51.901970 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:51.901968    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-8zf46" event={"ID":"e5a36d21-8daa-4c48-9625-80865a269e00","Type":"ContainerStarted","Data":"04ca22cb52de58c0be4ab98be639aaa344248790e7100576c29903eed41d44cd"}
Jul 28 18:04:51.925063 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:51.925012    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-8zf46" podStartSLOduration=7.924995828 podStartE2EDuration="7.924995828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:04:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:04:51.923806493 +0000 UTC m=+2495.330080233" watchObservedRunningTime="2026-07-28 18:04:51.924995828 +0000 UTC m=+2495.331269568"
Jul 28 18:04:58.941859 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:58.941824    2543 generic.go:356] "Generic (PLEG): container finished" podID="e5a36d21-8daa-4c48-9625-80865a269e00" containerID="8f6e4893bcb44bd5cb268436f8dbd50c77b29221d35c8d9fb19048356f4d0231" exitCode=0
Jul 28 18:04:58.942333 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:04:58.941878    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-8zf46" event={"ID":"e5a36d21-8daa-4c48-9625-80865a269e00","Type":"ContainerDied","Data":"8f6e4893bcb44bd5cb268436f8dbd50c77b29221d35c8d9fb19048356f4d0231"}
Jul 28 18:05:00.097861 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:00.097835    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-8zf46"
Jul 28 18:05:00.184038 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:00.183998    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tr5df\" (UniqueName: \"kubernetes.io/projected/e5a36d21-8daa-4c48-9625-80865a269e00-kube-api-access-tr5df\") pod \"e5a36d21-8daa-4c48-9625-80865a269e00\" (UID: \"e5a36d21-8daa-4c48-9625-80865a269e00\") "
Jul 28 18:05:00.184251 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:00.184154    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02faa6629b6a9a787\") pod \"e5a36d21-8daa-4c48-9625-80865a269e00\" (UID: \"e5a36d21-8daa-4c48-9625-80865a269e00\") "
Jul 28 18:05:00.186262 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:00.186232    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5a36d21-8daa-4c48-9625-80865a269e00-kube-api-access-tr5df" (OuterVolumeSpecName: "kube-api-access-tr5df") pod "e5a36d21-8daa-4c48-9625-80865a269e00" (UID: "e5a36d21-8daa-4c48-9625-80865a269e00"). InnerVolumeSpecName "kube-api-access-tr5df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:05:00.188433 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:00.188409    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-02faa6629b6a9a787" (OuterVolumeSpecName: "model-storage") pod "e5a36d21-8daa-4c48-9625-80865a269e00" (UID: "e5a36d21-8daa-4c48-9625-80865a269e00"). InnerVolumeSpecName "pvc-0cd2a7f4-dab0-4dd1-afea-3be8a9762a2e". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 28 18:05:00.284773 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:00.284742    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr5df\" (UniqueName: \"kubernetes.io/projected/e5a36d21-8daa-4c48-9625-80865a269e00-kube-api-access-tr5df\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:05:00.284919 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:00.284786    2543 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-0cd2a7f4-dab0-4dd1-afea-3be8a9762a2e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02faa6629b6a9a787\") on node \"ip-10-0-135-177.ec2.internal\" "
Jul 28 18:05:00.397165 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:00.397124    2543 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-0cd2a7f4-dab0-4dd1-afea-3be8a9762a2e" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-02faa6629b6a9a787") on node "ip-10-0-135-177.ec2.internal"
Jul 28 18:05:00.486233 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:00.486204    2543 reconciler_common.go:299] "Volume detached for volume \"pvc-0cd2a7f4-dab0-4dd1-afea-3be8a9762a2e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02faa6629b6a9a787\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"csi-f9ba76048ba6b415b5b327393e0b9189fe10d1d3d3390d04e07484bb43e98473\""
Jul 28 18:05:00.952661 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:00.952632    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-8zf46" event={"ID":"e5a36d21-8daa-4c48-9625-80865a269e00","Type":"ContainerDied","Data":"04ca22cb52de58c0be4ab98be639aaa344248790e7100576c29903eed41d44cd"}
Jul 28 18:05:00.952839 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:00.952673    2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04ca22cb52de58c0be4ab98be639aaa344248790e7100576c29903eed41d44cd"
Jul 28 18:05:00.952839 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:00.952646    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-8zf46"
Jul 28 18:05:24.649153 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:24.649089    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"]
Jul 28 18:05:24.649661 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:24.649641    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5a36d21-8daa-4c48-9625-80865a269e00" containerName="storage-initializer"
Jul 28 18:05:24.649741 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:24.649664    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5a36d21-8daa-4c48-9625-80865a269e00" containerName="storage-initializer"
Jul 28 18:05:24.649798 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:24.649744    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5a36d21-8daa-4c48-9625-80865a269e00" containerName="storage-initializer"
Jul 28 18:05:24.652646 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:24.652625    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:05:24.656102 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:24.656085    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"default-dockercfg-6t8zk\""
Jul 28 18:05:24.656311 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:24.656297    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"openshift-service-ca.crt\""
Jul 28 18:05:24.657096 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:24.657083    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"kube-root-ca.crt\""
Jul 28 18:05:24.659965 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:24.659941    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 28 18:05:24.672590 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:24.672564    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"]
Jul 28 18:05:24.802316 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:24.802279    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03c6c570-438a-4b12-ae99-16b71c4a6ffc-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8\" (UID: \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:05:24.802495 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:24.802329    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03c6c570-438a-4b12-ae99-16b71c4a6ffc-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8\" (UID: \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:05:24.802495 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:24.802403    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03c6c570-438a-4b12-ae99-16b71c4a6ffc-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8\" (UID: \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:05:24.802495 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:24.802477    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clgvm\" (UniqueName: \"kubernetes.io/projected/03c6c570-438a-4b12-ae99-16b71c4a6ffc-kube-api-access-clgvm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8\" (UID: \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:05:24.802626 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:24.802511    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03c6c570-438a-4b12-ae99-16b71c4a6ffc-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8\" (UID: \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:05:24.802626 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:24.802530    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03c6c570-438a-4b12-ae99-16b71c4a6ffc-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8\" (UID: \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:05:24.903572 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:24.903498    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03c6c570-438a-4b12-ae99-16b71c4a6ffc-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8\" (UID: \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:05:24.903572 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:24.903541    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03c6c570-438a-4b12-ae99-16b71c4a6ffc-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8\" (UID: \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:05:24.903572 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:24.903565    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03c6c570-438a-4b12-ae99-16b71c4a6ffc-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8\" (UID: \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:05:24.903858 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:24.903624    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clgvm\" (UniqueName: \"kubernetes.io/projected/03c6c570-438a-4b12-ae99-16b71c4a6ffc-kube-api-access-clgvm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8\" (UID: \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:05:24.903858 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:24.903654    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03c6c570-438a-4b12-ae99-16b71c4a6ffc-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8\" (UID: \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:05:24.903858 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:24.903703    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03c6c570-438a-4b12-ae99-16b71c4a6ffc-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8\" (UID: \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:05:24.904019 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:24.903984    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03c6c570-438a-4b12-ae99-16b71c4a6ffc-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8\" (UID: \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:05:24.904019 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:24.904012    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03c6c570-438a-4b12-ae99-16b71c4a6ffc-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8\" (UID: \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:05:24.904156 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:24.904069    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03c6c570-438a-4b12-ae99-16b71c4a6ffc-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8\" (UID: \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:05:24.905853 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:24.905835    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03c6c570-438a-4b12-ae99-16b71c4a6ffc-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8\" (UID: \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:05:24.906124 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:24.906106    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03c6c570-438a-4b12-ae99-16b71c4a6ffc-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8\" (UID: \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:05:24.940229 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:24.940203    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clgvm\" (UniqueName: \"kubernetes.io/projected/03c6c570-438a-4b12-ae99-16b71c4a6ffc-kube-api-access-clgvm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8\" (UID: \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:05:31.767903 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:31.767868    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-0cd2a7f4-dab0-4dd1-afea-3be8a9762a2e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02faa6629b6a9a787\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8\" (UID: \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:05:31.781422 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:31.781392    2543 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-0cd2a7f4-dab0-4dd1-afea-3be8a9762a2e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02faa6629b6a9a787\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8\" (UID: \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\") device path: \"\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:05:31.869148 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:31.869098    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-0cd2a7f4-dab0-4dd1-afea-3be8a9762a2e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02faa6629b6a9a787\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8\" (UID: \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:05:31.869297 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:31.869231    2543 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-0cd2a7f4-dab0-4dd1-afea-3be8a9762a2e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02faa6629b6a9a787\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8\" (UID: \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\") DevicePath \"\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:05:31.878382 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:31.878362    2543 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-0cd2a7f4-dab0-4dd1-afea-3be8a9762a2e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02faa6629b6a9a787\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8\" (UID: \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\") DevicePath \"csi-f9ba76048ba6b415b5b327393e0b9189fe10d1d3d3390d04e07484bb43e98473\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:05:31.975429 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:31.975376    2543 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-0cd2a7f4-dab0-4dd1-afea-3be8a9762a2e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02faa6629b6a9a787\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8\" (UID: \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/4603f6215567b3d7a9973d53157cdef0c0cbd9e8b44a99d35c810edc3956bb7a/globalmount\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:05:31.991198 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:31.991173    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-0cd2a7f4-dab0-4dd1-afea-3be8a9762a2e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02faa6629b6a9a787\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8\" (UID: \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:05:32.162511 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:32.162443    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:05:32.315831 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:32.315778    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"]
Jul 28 18:05:32.320015 ip-10-0-135-177 kubenswrapper[2543]: W0728 18:05:32.319987    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c6c570_438a_4b12_ae99_16b71c4a6ffc.slice/crio-b9eec035e671ddd75a8ae2e05cf052a037e6b859b73e38b0f1c5bd94aa549be2 WatchSource:0}: Error finding container b9eec035e671ddd75a8ae2e05cf052a037e6b859b73e38b0f1c5bd94aa549be2: Status 404 returned error can't find the container with id b9eec035e671ddd75a8ae2e05cf052a037e6b859b73e38b0f1c5bd94aa549be2
Jul 28 18:05:33.111976 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:33.111947    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8" event={"ID":"03c6c570-438a-4b12-ae99-16b71c4a6ffc","Type":"ContainerStarted","Data":"5c6e6e47a6363ecf1e6002a47df25e230836be77d00e5b6a130e1e5f2110e2cc"}
Jul 28 18:05:33.112380 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:33.111992    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8" event={"ID":"03c6c570-438a-4b12-ae99-16b71c4a6ffc","Type":"ContainerStarted","Data":"b9eec035e671ddd75a8ae2e05cf052a037e6b859b73e38b0f1c5bd94aa549be2"}
Jul 28 18:05:33.141309 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:33.141259    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8" podStartSLOduration=9.141244055 podStartE2EDuration="9.141244055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:05:33.141240735 +0000 UTC m=+2536.547514476" watchObservedRunningTime="2026-07-28 18:05:33.141244055 +0000 UTC m=+2536.547517795"
Jul 28 18:05:33.162603 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:33.162575    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:05:42.163346 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:42.163299    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:05:42.164935 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:42.164906    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8" podUID="03c6c570-438a-4b12-ae99-16b71c4a6ffc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 28 18:05:52.163896 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:05:52.163854    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8" podUID="03c6c570-438a-4b12-ae99-16b71c4a6ffc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 28 18:06:02.163548 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:06:02.163507    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8" podUID="03c6c570-438a-4b12-ae99-16b71c4a6ffc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 28 18:06:12.163065 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:06:12.163023    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8" podUID="03c6c570-438a-4b12-ae99-16b71c4a6ffc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 28 18:06:22.163726 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:06:22.163672    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8" podUID="03c6c570-438a-4b12-ae99-16b71c4a6ffc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 28 18:06:32.163555 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:06:32.163508    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8" podUID="03c6c570-438a-4b12-ae99-16b71c4a6ffc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 28 18:06:42.163418 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:06:42.163316    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8" podUID="03c6c570-438a-4b12-ae99-16b71c4a6ffc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 28 18:06:52.163566 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:06:52.163513    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8" podUID="03c6c570-438a-4b12-ae99-16b71c4a6ffc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 28 18:07:02.163573 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:02.163517    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8" podUID="03c6c570-438a-4b12-ae99-16b71c4a6ffc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 28 18:07:12.163645 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:12.163564    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8" podUID="03c6c570-438a-4b12-ae99-16b71c4a6ffc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 28 18:07:22.163635 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:22.163585    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8" podUID="03c6c570-438a-4b12-ae99-16b71c4a6ffc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 28 18:07:32.173066 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:32.173035    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:07:32.180870 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:32.180851    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:07:36.990725 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:36.990685    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"]
Jul 28 18:07:52.013019 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:52.012957    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8" podUID="03c6c570-438a-4b12-ae99-16b71c4a6ffc" containerName="main" containerID="cri-o://5c6e6e47a6363ecf1e6002a47df25e230836be77d00e5b6a130e1e5f2110e2cc" gracePeriod=45
Jul 28 18:07:52.174182 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:52.174127    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8" podUID="03c6c570-438a-4b12-ae99-16b71c4a6ffc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Jul 28 18:07:52.752198 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:52.752175    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:07:52.817496 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:52.817408    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clgvm\" (UniqueName: \"kubernetes.io/projected/03c6c570-438a-4b12-ae99-16b71c4a6ffc-kube-api-access-clgvm\") pod \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\" (UID: \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\") "
Jul 28 18:07:52.817496 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:52.817468    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03c6c570-438a-4b12-ae99-16b71c4a6ffc-model-cache\") pod \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\" (UID: \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\") "
Jul 28 18:07:52.817726 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:52.817572    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02faa6629b6a9a787\") pod \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\" (UID: \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\") "
Jul 28 18:07:52.817726 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:52.817598    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03c6c570-438a-4b12-ae99-16b71c4a6ffc-tls-certs\") pod \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\" (UID: \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\") "
Jul 28 18:07:52.817726 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:52.817640    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03c6c570-438a-4b12-ae99-16b71c4a6ffc-home\") pod \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\" (UID: \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\") "
Jul 28 18:07:52.817726 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:52.817657    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03c6c570-438a-4b12-ae99-16b71c4a6ffc-tmp-dir\") pod \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\" (UID: \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\") "
Jul 28 18:07:52.817726 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:52.817713    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03c6c570-438a-4b12-ae99-16b71c4a6ffc-dshm\") pod \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\" (UID: \"03c6c570-438a-4b12-ae99-16b71c4a6ffc\") "
Jul 28 18:07:52.818579 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:52.817747    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c6c570-438a-4b12-ae99-16b71c4a6ffc-model-cache" (OuterVolumeSpecName: "model-cache") pod "03c6c570-438a-4b12-ae99-16b71c4a6ffc" (UID: "03c6c570-438a-4b12-ae99-16b71c4a6ffc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:07:52.818579 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:52.817957    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03c6c570-438a-4b12-ae99-16b71c4a6ffc-model-cache\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:07:52.818579 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:52.818271    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c6c570-438a-4b12-ae99-16b71c4a6ffc-home" (OuterVolumeSpecName: "home") pod "03c6c570-438a-4b12-ae99-16b71c4a6ffc" (UID: "03c6c570-438a-4b12-ae99-16b71c4a6ffc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:07:52.820268 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:52.820229    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03c6c570-438a-4b12-ae99-16b71c4a6ffc-kube-api-access-clgvm" (OuterVolumeSpecName: "kube-api-access-clgvm") pod "03c6c570-438a-4b12-ae99-16b71c4a6ffc" (UID: "03c6c570-438a-4b12-ae99-16b71c4a6ffc"). InnerVolumeSpecName "kube-api-access-clgvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:07:52.820268 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:52.820230    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03c6c570-438a-4b12-ae99-16b71c4a6ffc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "03c6c570-438a-4b12-ae99-16b71c4a6ffc" (UID: "03c6c570-438a-4b12-ae99-16b71c4a6ffc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:07:52.820452 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:52.820437    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c6c570-438a-4b12-ae99-16b71c4a6ffc-dshm" (OuterVolumeSpecName: "dshm") pod "03c6c570-438a-4b12-ae99-16b71c4a6ffc" (UID: "03c6c570-438a-4b12-ae99-16b71c4a6ffc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:07:52.820806 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:52.820784    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-02faa6629b6a9a787" (OuterVolumeSpecName: "kserve-pvc-source") pod "03c6c570-438a-4b12-ae99-16b71c4a6ffc" (UID: "03c6c570-438a-4b12-ae99-16b71c4a6ffc"). InnerVolumeSpecName "pvc-0cd2a7f4-dab0-4dd1-afea-3be8a9762a2e". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 28 18:07:52.826680 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:52.826612    2543 generic.go:356] "Generic (PLEG): container finished" podID="03c6c570-438a-4b12-ae99-16b71c4a6ffc" containerID="5c6e6e47a6363ecf1e6002a47df25e230836be77d00e5b6a130e1e5f2110e2cc" exitCode=0
Jul 28 18:07:52.826680 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:52.826659    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8" event={"ID":"03c6c570-438a-4b12-ae99-16b71c4a6ffc","Type":"ContainerDied","Data":"5c6e6e47a6363ecf1e6002a47df25e230836be77d00e5b6a130e1e5f2110e2cc"}
Jul 28 18:07:52.826680 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:52.826670    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"
Jul 28 18:07:52.826902 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:52.826710    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8" event={"ID":"03c6c570-438a-4b12-ae99-16b71c4a6ffc","Type":"ContainerDied","Data":"b9eec035e671ddd75a8ae2e05cf052a037e6b859b73e38b0f1c5bd94aa549be2"}
Jul 28 18:07:52.826902 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:52.826740    2543 scope.go:117] "RemoveContainer" containerID="5c6e6e47a6363ecf1e6002a47df25e230836be77d00e5b6a130e1e5f2110e2cc"
Jul 28 18:07:52.830414 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:52.830388    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c6c570-438a-4b12-ae99-16b71c4a6ffc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "03c6c570-438a-4b12-ae99-16b71c4a6ffc" (UID: "03c6c570-438a-4b12-ae99-16b71c4a6ffc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:07:52.838310 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:52.838292    2543 scope.go:117] "RemoveContainer" containerID="5c6e6e47a6363ecf1e6002a47df25e230836be77d00e5b6a130e1e5f2110e2cc"
Jul 28 18:07:52.838553 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:07:52.838535    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c6e6e47a6363ecf1e6002a47df25e230836be77d00e5b6a130e1e5f2110e2cc\": container with ID starting with 5c6e6e47a6363ecf1e6002a47df25e230836be77d00e5b6a130e1e5f2110e2cc not found: ID does not exist" containerID="5c6e6e47a6363ecf1e6002a47df25e230836be77d00e5b6a130e1e5f2110e2cc"
Jul 28 18:07:52.838595 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:52.838562    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c6e6e47a6363ecf1e6002a47df25e230836be77d00e5b6a130e1e5f2110e2cc"} err="failed to get container status \"5c6e6e47a6363ecf1e6002a47df25e230836be77d00e5b6a130e1e5f2110e2cc\": rpc error: code = NotFound desc = could not find container \"5c6e6e47a6363ecf1e6002a47df25e230836be77d00e5b6a130e1e5f2110e2cc\": container with ID starting with 5c6e6e47a6363ecf1e6002a47df25e230836be77d00e5b6a130e1e5f2110e2cc not found: ID does not exist"
Jul 28 18:07:52.918995 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:52.918948    2543 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-0cd2a7f4-dab0-4dd1-afea-3be8a9762a2e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02faa6629b6a9a787\") on node \"ip-10-0-135-177.ec2.internal\" "
Jul 28 18:07:52.918995 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:52.918991    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03c6c570-438a-4b12-ae99-16b71c4a6ffc-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:07:52.918995 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:52.919001    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03c6c570-438a-4b12-ae99-16b71c4a6ffc-home\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:07:52.918995 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:52.919012    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03c6c570-438a-4b12-ae99-16b71c4a6ffc-tmp-dir\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:07:52.919268 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:52.919022    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03c6c570-438a-4b12-ae99-16b71c4a6ffc-dshm\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:07:52.919268 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:52.919035    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clgvm\" (UniqueName: \"kubernetes.io/projected/03c6c570-438a-4b12-ae99-16b71c4a6ffc-kube-api-access-clgvm\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:07:52.949581 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:52.949558    2543 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-0cd2a7f4-dab0-4dd1-afea-3be8a9762a2e" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-02faa6629b6a9a787") on node "ip-10-0-135-177.ec2.internal"
Jul 28 18:07:53.019664 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:53.019638    2543 reconciler_common.go:299] "Volume detached for volume \"pvc-0cd2a7f4-dab0-4dd1-afea-3be8a9762a2e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02faa6629b6a9a787\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"csi-f9ba76048ba6b415b5b327393e0b9189fe10d1d3d3390d04e07484bb43e98473\""
Jul 28 18:07:53.154771 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:53.154742    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"]
Jul 28 18:07:53.165627 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:53.165604    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclvfh8"]
Jul 28 18:07:55.152392 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:07:55.152358    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03c6c570-438a-4b12-ae99-16b71c4a6ffc" path="/var/lib/kubelet/pods/03c6c570-438a-4b12-ae99-16b71c4a6ffc/volumes"
Jul 28 18:08:00.283827 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:00.283792    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rwbfs"]
Jul 28 18:08:00.284219 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:00.284101    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03c6c570-438a-4b12-ae99-16b71c4a6ffc" containerName="main"
Jul 28 18:08:00.284219 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:00.284112    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="03c6c570-438a-4b12-ae99-16b71c4a6ffc" containerName="main"
Jul 28 18:08:00.284219 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:00.284196    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="03c6c570-438a-4b12-ae99-16b71c4a6ffc" containerName="main"
Jul 28 18:08:00.286963 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:00.286946    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rwbfs"
Jul 28 18:08:00.289968 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:00.289943    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"openshift-service-ca.crt\""
Jul 28 18:08:00.290128 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:00.290113    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"kube-root-ca.crt\""
Jul 28 18:08:00.290858 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:00.290841    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-xjjgk\""
Jul 28 18:08:00.299522 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:00.299501    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rwbfs"]
Jul 28 18:08:00.383673 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:00.383648    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t6fc\" (UniqueName: \"kubernetes.io/projected/7d7b3007-7142-4a10-b762-0ba3c9e9e4c8-kube-api-access-4t6fc\") pod \"e2e-pvc-model-download-rwbfs\" (UID: \"7d7b3007-7142-4a10-b762-0ba3c9e9e4c8\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rwbfs"
Jul 28 18:08:00.484465 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:00.484432    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t6fc\" (UniqueName: \"kubernetes.io/projected/7d7b3007-7142-4a10-b762-0ba3c9e9e4c8-kube-api-access-4t6fc\") pod \"e2e-pvc-model-download-rwbfs\" (UID: \"7d7b3007-7142-4a10-b762-0ba3c9e9e4c8\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rwbfs"
Jul 28 18:08:00.493725 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:00.493703    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t6fc\" (UniqueName: \"kubernetes.io/projected/7d7b3007-7142-4a10-b762-0ba3c9e9e4c8-kube-api-access-4t6fc\") pod \"e2e-pvc-model-download-rwbfs\" (UID: \"7d7b3007-7142-4a10-b762-0ba3c9e9e4c8\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rwbfs"
Jul 28 18:08:01.900367 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:01.900331    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-8zf46"]
Jul 28 18:08:01.909209 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:01.909180    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-8zf46"]
Jul 28 18:08:03.152459 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:03.152419    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5a36d21-8daa-4c48-9625-80865a269e00" path="/var/lib/kubelet/pods/e5a36d21-8daa-4c48-9625-80865a269e00/volumes"
Jul 28 18:08:04.718575 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:04.718540    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-7baa4492-9403-4ad1-a2ce-101d15aeeaeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0408b107de8cef29c\") pod \"e2e-pvc-model-download-rwbfs\" (UID: \"7d7b3007-7142-4a10-b762-0ba3c9e9e4c8\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rwbfs"
Jul 28 18:08:04.725246 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:04.725220    2543 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-7baa4492-9403-4ad1-a2ce-101d15aeeaeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0408b107de8cef29c\") pod \"e2e-pvc-model-download-rwbfs\" (UID: \"7d7b3007-7142-4a10-b762-0ba3c9e9e4c8\") device path: \"\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rwbfs"
Jul 28 18:08:04.819795 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:04.819763    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-7baa4492-9403-4ad1-a2ce-101d15aeeaeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0408b107de8cef29c\") pod \"e2e-pvc-model-download-rwbfs\" (UID: \"7d7b3007-7142-4a10-b762-0ba3c9e9e4c8\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rwbfs"
Jul 28 18:08:04.819930 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:04.819901    2543 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-7baa4492-9403-4ad1-a2ce-101d15aeeaeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0408b107de8cef29c\") pod \"e2e-pvc-model-download-rwbfs\" (UID: \"7d7b3007-7142-4a10-b762-0ba3c9e9e4c8\") DevicePath \"\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rwbfs"
Jul 28 18:08:04.822612 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:04.822590    2543 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-7baa4492-9403-4ad1-a2ce-101d15aeeaeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0408b107de8cef29c\") pod \"e2e-pvc-model-download-rwbfs\" (UID: \"7d7b3007-7142-4a10-b762-0ba3c9e9e4c8\") DevicePath \"csi-0ee262add9c934129ef01d5178d9b2aa0f688c02ef865c88cf07bdc882882bdd\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rwbfs"
Jul 28 18:08:05.032679 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:05.032651    2543 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-7baa4492-9403-4ad1-a2ce-101d15aeeaeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0408b107de8cef29c\") pod \"e2e-pvc-model-download-rwbfs\" (UID: \"7d7b3007-7142-4a10-b762-0ba3c9e9e4c8\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a2b1b41a2708640572ef6a0f5ba7fe5be71361ae6201f7c3e4b363026141d9c9/globalmount\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rwbfs"
Jul 28 18:08:05.050342 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:05.050319    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-7baa4492-9403-4ad1-a2ce-101d15aeeaeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0408b107de8cef29c\") pod \"e2e-pvc-model-download-rwbfs\" (UID: \"7d7b3007-7142-4a10-b762-0ba3c9e9e4c8\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rwbfs"
Jul 28 18:08:05.095916 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:05.095893    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rwbfs"
Jul 28 18:08:05.241355 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:05.241331    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rwbfs"]
Jul 28 18:08:05.243472 ip-10-0-135-177 kubenswrapper[2543]: W0728 18:08:05.243444    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d7b3007_7142_4a10_b762_0ba3c9e9e4c8.slice/crio-619b055795402c3ac3cda9665a1657923255e5299299d7fee6d2be17734d9b2e WatchSource:0}: Error finding container 619b055795402c3ac3cda9665a1657923255e5299299d7fee6d2be17734d9b2e: Status 404 returned error can't find the container with id 619b055795402c3ac3cda9665a1657923255e5299299d7fee6d2be17734d9b2e
Jul 28 18:08:05.894327 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:05.894294    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rwbfs" event={"ID":"7d7b3007-7142-4a10-b762-0ba3c9e9e4c8","Type":"ContainerStarted","Data":"8f7debd060371503730da39bb17175c418536c3ffe0059810cf3215358fc028c"}
Jul 28 18:08:05.894770 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:05.894337    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rwbfs" event={"ID":"7d7b3007-7142-4a10-b762-0ba3c9e9e4c8","Type":"ContainerStarted","Data":"619b055795402c3ac3cda9665a1657923255e5299299d7fee6d2be17734d9b2e"}
Jul 28 18:08:05.916597 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:05.916539    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rwbfs" podStartSLOduration=8.916520181 podStartE2EDuration="8.916520181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:07:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:08:05.913437781 +0000 UTC m=+2689.319711523" watchObservedRunningTime="2026-07-28 18:08:05.916520181 +0000 UTC m=+2689.322793922"
Jul 28 18:08:21.978006 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:21.977974    2543 generic.go:356] "Generic (PLEG): container finished" podID="7d7b3007-7142-4a10-b762-0ba3c9e9e4c8" containerID="8f7debd060371503730da39bb17175c418536c3ffe0059810cf3215358fc028c" exitCode=0
Jul 28 18:08:21.978465 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:21.978044    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rwbfs" event={"ID":"7d7b3007-7142-4a10-b762-0ba3c9e9e4c8","Type":"ContainerDied","Data":"8f7debd060371503730da39bb17175c418536c3ffe0059810cf3215358fc028c"}
Jul 28 18:08:23.133453 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:23.133426    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rwbfs"
Jul 28 18:08:23.283744 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:23.283704    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4t6fc\" (UniqueName: \"kubernetes.io/projected/7d7b3007-7142-4a10-b762-0ba3c9e9e4c8-kube-api-access-4t6fc\") pod \"7d7b3007-7142-4a10-b762-0ba3c9e9e4c8\" (UID: \"7d7b3007-7142-4a10-b762-0ba3c9e9e4c8\") "
Jul 28 18:08:23.283941 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:23.283856    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0408b107de8cef29c\") pod \"7d7b3007-7142-4a10-b762-0ba3c9e9e4c8\" (UID: \"7d7b3007-7142-4a10-b762-0ba3c9e9e4c8\") "
Jul 28 18:08:23.286100 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:23.286075    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d7b3007-7142-4a10-b762-0ba3c9e9e4c8-kube-api-access-4t6fc" (OuterVolumeSpecName: "kube-api-access-4t6fc") pod "7d7b3007-7142-4a10-b762-0ba3c9e9e4c8" (UID: "7d7b3007-7142-4a10-b762-0ba3c9e9e4c8"). InnerVolumeSpecName "kube-api-access-4t6fc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:08:23.286563 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:23.286544    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0408b107de8cef29c" (OuterVolumeSpecName: "model-storage") pod "7d7b3007-7142-4a10-b762-0ba3c9e9e4c8" (UID: "7d7b3007-7142-4a10-b762-0ba3c9e9e4c8"). InnerVolumeSpecName "pvc-7baa4492-9403-4ad1-a2ce-101d15aeeaeb". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 28 18:08:23.385213 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:23.385172    2543 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-7baa4492-9403-4ad1-a2ce-101d15aeeaeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0408b107de8cef29c\") on node \"ip-10-0-135-177.ec2.internal\" "
Jul 28 18:08:23.385213 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:23.385210    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t6fc\" (UniqueName: \"kubernetes.io/projected/7d7b3007-7142-4a10-b762-0ba3c9e9e4c8-kube-api-access-4t6fc\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:08:23.498860 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:23.498841    2543 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-7baa4492-9403-4ad1-a2ce-101d15aeeaeb" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0408b107de8cef29c") on node "ip-10-0-135-177.ec2.internal"
Jul 28 18:08:23.587602 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:23.587522    2543 reconciler_common.go:299] "Volume detached for volume \"pvc-7baa4492-9403-4ad1-a2ce-101d15aeeaeb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0408b107de8cef29c\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"csi-0ee262add9c934129ef01d5178d9b2aa0f688c02ef865c88cf07bdc882882bdd\""
Jul 28 18:08:23.990830 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:23.990802    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rwbfs"
Jul 28 18:08:23.990996 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:23.990846    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rwbfs" event={"ID":"7d7b3007-7142-4a10-b762-0ba3c9e9e4c8","Type":"ContainerDied","Data":"619b055795402c3ac3cda9665a1657923255e5299299d7fee6d2be17734d9b2e"}
Jul 28 18:08:23.990996 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:23.990879    2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="619b055795402c3ac3cda9665a1657923255e5299299d7fee6d2be17734d9b2e"
Jul 28 18:08:36.447357 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.447319    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"]
Jul 28 18:08:36.447828 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.447811    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d7b3007-7142-4a10-b762-0ba3c9e9e4c8" containerName="storage-initializer"
Jul 28 18:08:36.447872 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.447831    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d7b3007-7142-4a10-b762-0ba3c9e9e4c8" containerName="storage-initializer"
Jul 28 18:08:36.447907 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.447892    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d7b3007-7142-4a10-b762-0ba3c9e9e4c8" containerName="storage-initializer"
Jul 28 18:08:36.451071 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.451055    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:08:36.454770 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.454712    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"kube-root-ca.crt\""
Jul 28 18:08:36.454770 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.454766    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-2x7tv\""
Jul 28 18:08:36.455375 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.455345    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-xjjgk\""
Jul 28 18:08:36.455510 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.455387    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 28 18:08:36.455699 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.455684    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"openshift-service-ca.crt\""
Jul 28 18:08:36.469504 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.469474    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"]
Jul 28 18:08:36.593556 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.593522    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4t8s\" (UniqueName: \"kubernetes.io/projected/1c409c83-3d4a-40a6-9d31-4982e37030c4-kube-api-access-l4t8s\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8\" (UID: \"1c409c83-3d4a-40a6-9d31-4982e37030c4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:08:36.593556 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.593564    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c409c83-3d4a-40a6-9d31-4982e37030c4-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8\" (UID: \"1c409c83-3d4a-40a6-9d31-4982e37030c4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:08:36.593791 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.593683    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c409c83-3d4a-40a6-9d31-4982e37030c4-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8\" (UID: \"1c409c83-3d4a-40a6-9d31-4982e37030c4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:08:36.593791 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.593703    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c409c83-3d4a-40a6-9d31-4982e37030c4-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8\" (UID: \"1c409c83-3d4a-40a6-9d31-4982e37030c4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:08:36.593791 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.593737    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c409c83-3d4a-40a6-9d31-4982e37030c4-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8\" (UID: \"1c409c83-3d4a-40a6-9d31-4982e37030c4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:08:36.593791 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.593767    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c409c83-3d4a-40a6-9d31-4982e37030c4-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8\" (UID: \"1c409c83-3d4a-40a6-9d31-4982e37030c4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:08:36.694324 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.694290    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c409c83-3d4a-40a6-9d31-4982e37030c4-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8\" (UID: \"1c409c83-3d4a-40a6-9d31-4982e37030c4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:08:36.694324 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.694324    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c409c83-3d4a-40a6-9d31-4982e37030c4-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8\" (UID: \"1c409c83-3d4a-40a6-9d31-4982e37030c4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:08:36.694541 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.694357    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c409c83-3d4a-40a6-9d31-4982e37030c4-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8\" (UID: \"1c409c83-3d4a-40a6-9d31-4982e37030c4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:08:36.694541 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.694470    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c409c83-3d4a-40a6-9d31-4982e37030c4-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8\" (UID: \"1c409c83-3d4a-40a6-9d31-4982e37030c4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:08:36.694541 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.694512    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4t8s\" (UniqueName: \"kubernetes.io/projected/1c409c83-3d4a-40a6-9d31-4982e37030c4-kube-api-access-l4t8s\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8\" (UID: \"1c409c83-3d4a-40a6-9d31-4982e37030c4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:08:36.694707 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.694542    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c409c83-3d4a-40a6-9d31-4982e37030c4-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8\" (UID: \"1c409c83-3d4a-40a6-9d31-4982e37030c4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:08:36.694768 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.694704    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c409c83-3d4a-40a6-9d31-4982e37030c4-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8\" (UID: \"1c409c83-3d4a-40a6-9d31-4982e37030c4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:08:36.694879 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.694849    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c409c83-3d4a-40a6-9d31-4982e37030c4-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8\" (UID: \"1c409c83-3d4a-40a6-9d31-4982e37030c4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:08:36.694941 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.694872    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c409c83-3d4a-40a6-9d31-4982e37030c4-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8\" (UID: \"1c409c83-3d4a-40a6-9d31-4982e37030c4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:08:36.696716 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.696699    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c409c83-3d4a-40a6-9d31-4982e37030c4-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8\" (UID: \"1c409c83-3d4a-40a6-9d31-4982e37030c4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:08:36.697010 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.696989    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c409c83-3d4a-40a6-9d31-4982e37030c4-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8\" (UID: \"1c409c83-3d4a-40a6-9d31-4982e37030c4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:08:36.709383 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.709316    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4t8s\" (UniqueName: \"kubernetes.io/projected/1c409c83-3d4a-40a6-9d31-4982e37030c4-kube-api-access-l4t8s\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8\" (UID: \"1c409c83-3d4a-40a6-9d31-4982e37030c4\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:08:36.790247 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.790214    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqzcxr"]
Jul 28 18:08:36.793837 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.793820    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqzcxr"
Jul 28 18:08:36.796339 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.796314    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-bkcw5\""
Jul 28 18:08:36.806550 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.806524    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqzcxr"]
Jul 28 18:08:36.896650 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.896608    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a175d464-b6e6-41ee-a9d8-c5d400230273-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqzcxr\" (UID: \"a175d464-b6e6-41ee-a9d8-c5d400230273\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqzcxr"
Jul 28 18:08:36.896840 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.896720    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8hvk\" (UniqueName: \"kubernetes.io/projected/a175d464-b6e6-41ee-a9d8-c5d400230273-kube-api-access-m8hvk\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqzcxr\" (UID: \"a175d464-b6e6-41ee-a9d8-c5d400230273\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqzcxr"
Jul 28 18:08:36.997598 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.997567    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8hvk\" (UniqueName: \"kubernetes.io/projected/a175d464-b6e6-41ee-a9d8-c5d400230273-kube-api-access-m8hvk\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqzcxr\" (UID: \"a175d464-b6e6-41ee-a9d8-c5d400230273\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqzcxr"
Jul 28 18:08:36.997782 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:36.997633    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a175d464-b6e6-41ee-a9d8-c5d400230273-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqzcxr\" (UID: \"a175d464-b6e6-41ee-a9d8-c5d400230273\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqzcxr"
Jul 28 18:08:37.000080 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:37.000061    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a175d464-b6e6-41ee-a9d8-c5d400230273-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqzcxr\" (UID: \"a175d464-b6e6-41ee-a9d8-c5d400230273\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqzcxr"
Jul 28 18:08:37.008530 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:37.008508    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8hvk\" (UniqueName: \"kubernetes.io/projected/a175d464-b6e6-41ee-a9d8-c5d400230273-kube-api-access-m8hvk\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqzcxr\" (UID: \"a175d464-b6e6-41ee-a9d8-c5d400230273\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqzcxr"
Jul 28 18:08:37.104800 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:37.104763    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqzcxr"
Jul 28 18:08:37.247249 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:37.247220    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqzcxr"]
Jul 28 18:08:37.249338 ip-10-0-135-177 kubenswrapper[2543]: W0728 18:08:37.249276    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda175d464_b6e6_41ee_a9d8_c5d400230273.slice/crio-7e0c42f17bf9d279b9be071629664bd72628a5c9f471196490bc9068d47eb8f5 WatchSource:0}: Error finding container 7e0c42f17bf9d279b9be071629664bd72628a5c9f471196490bc9068d47eb8f5: Status 404 returned error can't find the container with id 7e0c42f17bf9d279b9be071629664bd72628a5c9f471196490bc9068d47eb8f5
Jul 28 18:08:38.066498 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:38.066461    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqzcxr" event={"ID":"a175d464-b6e6-41ee-a9d8-c5d400230273","Type":"ContainerStarted","Data":"2c2f1fa1ef8f2bfc1c08ce1ad84b5b3959a777c334e89e0c93e2595ebc0cfdb2"}
Jul 28 18:08:38.066952 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:38.066523    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqzcxr" event={"ID":"a175d464-b6e6-41ee-a9d8-c5d400230273","Type":"ContainerStarted","Data":"7e0c42f17bf9d279b9be071629664bd72628a5c9f471196490bc9068d47eb8f5"}
Jul 28 18:08:38.066952 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:38.066551    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqzcxr"
Jul 28 18:08:38.088117 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:08:38.088055    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqzcxr" podStartSLOduration=2.088033218 podStartE2EDuration="2.088033218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:08:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:08:38.084593848 +0000 UTC m=+2721.490867590" watchObservedRunningTime="2026-07-28 18:08:38.088033218 +0000 UTC m=+2721.494306960"
Jul 28 18:09:09.073716 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:09:09.073686    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqzcxr"
Jul 28 18:10:39.463383 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:10:39.463342    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8" podUID="1c409c83-3d4a-40a6-9d31-4982e37030c4"
Jul 28 18:10:39.664309 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:10:39.664274    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:11:17.729497 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:11:17.729421    2543 scope.go:117] "RemoveContainer" containerID="8f6e4893bcb44bd5cb268436f8dbd50c77b29221d35c8d9fb19048356f4d0231"
Jul 28 18:12:42.665792 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:12:42.665704    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8" podUID="1c409c83-3d4a-40a6-9d31-4982e37030c4"
Jul 28 18:12:43.272275 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:12:43.272242    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:14:46.273280 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:14:46.273237    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8" podUID="1c409c83-3d4a-40a6-9d31-4982e37030c4"
Jul 28 18:14:46.882165 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:14:46.882116    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:16:49.883214 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:16:49.883169    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8" podUID="1c409c83-3d4a-40a6-9d31-4982e37030c4"
Jul 28 18:16:50.499356 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:16:50.499321    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:18:53.500064 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:18:53.500016    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8" podUID="1c409c83-3d4a-40a6-9d31-4982e37030c4"
Jul 28 18:18:54.128269 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:18:54.128239    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:19:22.615436 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:19:22.615403    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 28 18:19:22.619519 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:19:22.619498    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:22.623307 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:19:22.623286    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-kserve-mn-dockercfg-k2mvb\""
Jul 28 18:19:22.623734 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:19:22.623720    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"kube-root-ca.crt\""
Jul 28 18:19:22.624250 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:19:22.624221    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-kserve-self-signed-certs\""
Jul 28 18:19:22.624738 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:19:22.624264    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"default-dockercfg-v865s\""
Jul 28 18:19:22.626765 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:19:22.626737    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"openshift-service-ca.crt\""
Jul 28 18:19:22.644177 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:19:22.644110    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 28 18:19:22.691697 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:19:22.691666    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4337ce4-acb9-459b-b984-ee75b0aa4149-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"f4337ce4-acb9-459b-b984-ee75b0aa4149\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:22.691814 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:19:22.691703    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4337ce4-acb9-459b-b984-ee75b0aa4149-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"f4337ce4-acb9-459b-b984-ee75b0aa4149\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:22.691814 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:19:22.691721    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4337ce4-acb9-459b-b984-ee75b0aa4149-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"f4337ce4-acb9-459b-b984-ee75b0aa4149\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:22.691814 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:19:22.691802    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr74s\" (UniqueName: \"kubernetes.io/projected/f4337ce4-acb9-459b-b984-ee75b0aa4149-kube-api-access-wr74s\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"f4337ce4-acb9-459b-b984-ee75b0aa4149\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:22.691928 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:19:22.691854    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4337ce4-acb9-459b-b984-ee75b0aa4149-home\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"f4337ce4-acb9-459b-b984-ee75b0aa4149\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:22.691928 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:19:22.691893    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4337ce4-acb9-459b-b984-ee75b0aa4149-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"f4337ce4-acb9-459b-b984-ee75b0aa4149\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:22.792457 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:19:22.792424    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4337ce4-acb9-459b-b984-ee75b0aa4149-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"f4337ce4-acb9-459b-b984-ee75b0aa4149\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:22.792616 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:19:22.792466    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4337ce4-acb9-459b-b984-ee75b0aa4149-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"f4337ce4-acb9-459b-b984-ee75b0aa4149\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:22.792616 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:19:22.792496    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4337ce4-acb9-459b-b984-ee75b0aa4149-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"f4337ce4-acb9-459b-b984-ee75b0aa4149\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:22.792616 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:19:22.792567    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr74s\" (UniqueName: \"kubernetes.io/projected/f4337ce4-acb9-459b-b984-ee75b0aa4149-kube-api-access-wr74s\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"f4337ce4-acb9-459b-b984-ee75b0aa4149\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:22.792616 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:19:22.792613    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4337ce4-acb9-459b-b984-ee75b0aa4149-home\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"f4337ce4-acb9-459b-b984-ee75b0aa4149\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:22.792848 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:19:22.792652    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4337ce4-acb9-459b-b984-ee75b0aa4149-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"f4337ce4-acb9-459b-b984-ee75b0aa4149\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:22.792934 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:19:22.792912    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4337ce4-acb9-459b-b984-ee75b0aa4149-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"f4337ce4-acb9-459b-b984-ee75b0aa4149\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:22.792992 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:19:22.792944    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4337ce4-acb9-459b-b984-ee75b0aa4149-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"f4337ce4-acb9-459b-b984-ee75b0aa4149\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:22.792992 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:19:22.792959    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4337ce4-acb9-459b-b984-ee75b0aa4149-home\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"f4337ce4-acb9-459b-b984-ee75b0aa4149\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:22.794696 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:19:22.794672    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4337ce4-acb9-459b-b984-ee75b0aa4149-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"f4337ce4-acb9-459b-b984-ee75b0aa4149\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:22.794895 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:19:22.794879    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4337ce4-acb9-459b-b984-ee75b0aa4149-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"f4337ce4-acb9-459b-b984-ee75b0aa4149\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:22.812036 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:19:22.812008    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr74s\" (UniqueName: \"kubernetes.io/projected/f4337ce4-acb9-459b-b984-ee75b0aa4149-kube-api-access-wr74s\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"f4337ce4-acb9-459b-b984-ee75b0aa4149\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:22.930117 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:19:22.930044    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:19:23.067314 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:19:23.067282    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 28 18:19:23.067508 ip-10-0-135-177 kubenswrapper[2543]: W0728 18:19:23.067484    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4337ce4_acb9_459b_b984_ee75b0aa4149.slice/crio-6ed3319902cbaf2e59fd5d968f78acc1ca3ea1d21f1364fea91d4a422f538c60 WatchSource:0}: Error finding container 6ed3319902cbaf2e59fd5d968f78acc1ca3ea1d21f1364fea91d4a422f538c60: Status 404 returned error can't find the container with id 6ed3319902cbaf2e59fd5d968f78acc1ca3ea1d21f1364fea91d4a422f538c60
Jul 28 18:19:23.276280 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:19:23.276242    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" event={"ID":"f4337ce4-acb9-459b-b984-ee75b0aa4149","Type":"ContainerStarted","Data":"8bae7d1c6b7f416383eefe5f7e1a0ac833a29f15399269e8ff3b681faaddcd6c"}
Jul 28 18:19:23.276471 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:19:23.276300    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" event={"ID":"f4337ce4-acb9-459b-b984-ee75b0aa4149","Type":"ContainerStarted","Data":"6ed3319902cbaf2e59fd5d968f78acc1ca3ea1d21f1364fea91d4a422f538c60"}
Jul 28 18:19:23.297825 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:19:23.297774    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" podStartSLOduration=1.297754157 podStartE2EDuration="1.297754157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:19:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:19:23.296298296 +0000 UTC m=+3366.702572036" watchObservedRunningTime="2026-07-28 18:19:23.297754157 +0000 UTC m=+3366.704027900"
Jul 28 18:20:57.128871 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:20:57.128833    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8" podUID="1c409c83-3d4a-40a6-9d31-4982e37030c4"
Jul 28 18:20:57.744933 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:20:57.744901    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:23:00.746222 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:23:00.746181    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8" podUID="1c409c83-3d4a-40a6-9d31-4982e37030c4"
Jul 28 18:23:01.385167 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:23:01.380555    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:23:28.727257 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:23:28.727230    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-48639af5_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheqzcxr_a175d464-b6e6-41ee-a9d8-c5d400230273/main/0.log"
Jul 28 18:24:25.460586 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:24:25.460547    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schecjgww"]
Jul 28 18:24:25.463979 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:24:25.463962    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schecjgww"
Jul 28 18:24:25.468657 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:24:25.468637    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 28 18:24:25.468765 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:24:25.468724    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"kube-root-ca.crt\""
Jul 28 18:24:25.468812 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:24:25.468778    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-xgkv6\""
Jul 28 18:24:25.468850 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:24:25.468816    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-s7ngj\""
Jul 28 18:24:25.480006 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:24:25.479990    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"openshift-service-ca.crt\""
Jul 28 18:24:25.487763 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:24:25.487742    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schecjgww"]
Jul 28 18:24:25.515360 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:24:25.515339    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04e17f16-b4ef-4f55-a41e-20da7fcc90d2-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schecjgww\" (UID: \"04e17f16-b4ef-4f55-a41e-20da7fcc90d2\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schecjgww"
Jul 28 18:24:25.515464 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:24:25.515402    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkg2l\" (UniqueName: \"kubernetes.io/projected/04e17f16-b4ef-4f55-a41e-20da7fcc90d2-kube-api-access-qkg2l\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schecjgww\" (UID: \"04e17f16-b4ef-4f55-a41e-20da7fcc90d2\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schecjgww"
Jul 28 18:24:25.615826 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:24:25.615795    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04e17f16-b4ef-4f55-a41e-20da7fcc90d2-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schecjgww\" (UID: \"04e17f16-b4ef-4f55-a41e-20da7fcc90d2\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schecjgww"
Jul 28 18:24:25.615964 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:24:25.615877    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkg2l\" (UniqueName: \"kubernetes.io/projected/04e17f16-b4ef-4f55-a41e-20da7fcc90d2-kube-api-access-qkg2l\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schecjgww\" (UID: \"04e17f16-b4ef-4f55-a41e-20da7fcc90d2\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schecjgww"
Jul 28 18:24:25.618346 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:24:25.618326    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04e17f16-b4ef-4f55-a41e-20da7fcc90d2-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schecjgww\" (UID: \"04e17f16-b4ef-4f55-a41e-20da7fcc90d2\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schecjgww"
Jul 28 18:24:25.624897 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:24:25.624873    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkg2l\" (UniqueName: \"kubernetes.io/projected/04e17f16-b4ef-4f55-a41e-20da7fcc90d2-kube-api-access-qkg2l\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schecjgww\" (UID: \"04e17f16-b4ef-4f55-a41e-20da7fcc90d2\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schecjgww"
Jul 28 18:24:25.775073 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:24:25.775040    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schecjgww"
Jul 28 18:24:25.910840 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:24:25.910815    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schecjgww"]
Jul 28 18:24:25.912678 ip-10-0-135-177 kubenswrapper[2543]: W0728 18:24:25.912652    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e17f16_b4ef_4f55_a41e_20da7fcc90d2.slice/crio-2e044060b6fbd4935f3ce2010082a6feb9572077fb49d31395c2ddbbac2d1056 WatchSource:0}: Error finding container 2e044060b6fbd4935f3ce2010082a6feb9572077fb49d31395c2ddbbac2d1056: Status 404 returned error can't find the container with id 2e044060b6fbd4935f3ce2010082a6feb9572077fb49d31395c2ddbbac2d1056
Jul 28 18:24:26.810081 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:24:26.810045    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schecjgww" event={"ID":"04e17f16-b4ef-4f55-a41e-20da7fcc90d2","Type":"ContainerStarted","Data":"87caf377cd49d362beefbea39b34a988add1871fe4b33e4ad4b4a4d9504adebd"}
Jul 28 18:24:26.810525 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:24:26.810106    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schecjgww" event={"ID":"04e17f16-b4ef-4f55-a41e-20da7fcc90d2","Type":"ContainerStarted","Data":"2e044060b6fbd4935f3ce2010082a6feb9572077fb49d31395c2ddbbac2d1056"}
Jul 28 18:24:26.810525 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:24:26.810147    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schecjgww"
Jul 28 18:24:26.830109 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:24:26.830061    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schecjgww" podStartSLOduration=1.8300451789999999 podStartE2EDuration="1.830045179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:24:26.828587534 +0000 UTC m=+3670.234861275" watchObservedRunningTime="2026-07-28 18:24:26.830045179 +0000 UTC m=+3670.236318919"
Jul 28 18:24:57.817947 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:24:57.817911    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schecjgww"
Jul 28 18:25:04.385849 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:25:04.385804    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8" podUID="1c409c83-3d4a-40a6-9d31-4982e37030c4"
Jul 28 18:25:04.997735 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:25:04.997704    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:26:38.061988 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:26:38.061948    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schecjgww"]
Jul 28 18:26:47.817754 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:26:47.817712    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schecjgww" podUID="04e17f16-b4ef-4f55-a41e-20da7fcc90d2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:26:53.084182 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:26:53.084096    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schecjgww" podUID="04e17f16-b4ef-4f55-a41e-20da7fcc90d2" containerName="main" containerID="cri-o://87caf377cd49d362beefbea39b34a988add1871fe4b33e4ad4b4a4d9504adebd" gracePeriod=45
Jul 28 18:26:53.324793 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:26:53.324769    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schecjgww"
Jul 28 18:26:53.429463 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:26:53.429378    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04e17f16-b4ef-4f55-a41e-20da7fcc90d2-tls-certs\") pod \"04e17f16-b4ef-4f55-a41e-20da7fcc90d2\" (UID: \"04e17f16-b4ef-4f55-a41e-20da7fcc90d2\") "
Jul 28 18:26:53.429463 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:26:53.429415    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkg2l\" (UniqueName: \"kubernetes.io/projected/04e17f16-b4ef-4f55-a41e-20da7fcc90d2-kube-api-access-qkg2l\") pod \"04e17f16-b4ef-4f55-a41e-20da7fcc90d2\" (UID: \"04e17f16-b4ef-4f55-a41e-20da7fcc90d2\") "
Jul 28 18:26:53.431593 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:26:53.431561    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04e17f16-b4ef-4f55-a41e-20da7fcc90d2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "04e17f16-b4ef-4f55-a41e-20da7fcc90d2" (UID: "04e17f16-b4ef-4f55-a41e-20da7fcc90d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:26:53.431711 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:26:53.431680    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e17f16-b4ef-4f55-a41e-20da7fcc90d2-kube-api-access-qkg2l" (OuterVolumeSpecName: "kube-api-access-qkg2l") pod "04e17f16-b4ef-4f55-a41e-20da7fcc90d2" (UID: "04e17f16-b4ef-4f55-a41e-20da7fcc90d2"). InnerVolumeSpecName "kube-api-access-qkg2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:26:53.530535 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:26:53.530506    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04e17f16-b4ef-4f55-a41e-20da7fcc90d2-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:26:53.530535 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:26:53.530531    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkg2l\" (UniqueName: \"kubernetes.io/projected/04e17f16-b4ef-4f55-a41e-20da7fcc90d2-kube-api-access-qkg2l\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:26:53.535917 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:26:53.535892    2543 generic.go:356] "Generic (PLEG): container finished" podID="04e17f16-b4ef-4f55-a41e-20da7fcc90d2" containerID="87caf377cd49d362beefbea39b34a988add1871fe4b33e4ad4b4a4d9504adebd" exitCode=0
Jul 28 18:26:53.536044 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:26:53.535938    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schecjgww"
Jul 28 18:26:53.536044 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:26:53.535980    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schecjgww" event={"ID":"04e17f16-b4ef-4f55-a41e-20da7fcc90d2","Type":"ContainerDied","Data":"87caf377cd49d362beefbea39b34a988add1871fe4b33e4ad4b4a4d9504adebd"}
Jul 28 18:26:53.536044 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:26:53.536035    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schecjgww" event={"ID":"04e17f16-b4ef-4f55-a41e-20da7fcc90d2","Type":"ContainerDied","Data":"2e044060b6fbd4935f3ce2010082a6feb9572077fb49d31395c2ddbbac2d1056"}
Jul 28 18:26:53.536197 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:26:53.536058    2543 scope.go:117] "RemoveContainer" containerID="87caf377cd49d362beefbea39b34a988add1871fe4b33e4ad4b4a4d9504adebd"
Jul 28 18:26:53.545711 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:26:53.545694    2543 scope.go:117] "RemoveContainer" containerID="87caf377cd49d362beefbea39b34a988add1871fe4b33e4ad4b4a4d9504adebd"
Jul 28 18:26:53.545950 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:26:53.545931    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87caf377cd49d362beefbea39b34a988add1871fe4b33e4ad4b4a4d9504adebd\": container with ID starting with 87caf377cd49d362beefbea39b34a988add1871fe4b33e4ad4b4a4d9504adebd not found: ID does not exist" containerID="87caf377cd49d362beefbea39b34a988add1871fe4b33e4ad4b4a4d9504adebd"
Jul 28 18:26:53.546003 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:26:53.545957    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87caf377cd49d362beefbea39b34a988add1871fe4b33e4ad4b4a4d9504adebd"} err="failed to get container status \"87caf377cd49d362beefbea39b34a988add1871fe4b33e4ad4b4a4d9504adebd\": rpc error: code = NotFound desc = could not find container \"87caf377cd49d362beefbea39b34a988add1871fe4b33e4ad4b4a4d9504adebd\": container with ID starting with 87caf377cd49d362beefbea39b34a988add1871fe4b33e4ad4b4a4d9504adebd not found: ID does not exist"
Jul 28 18:26:53.558408 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:26:53.558374    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schecjgww"]
Jul 28 18:26:53.561253 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:26:53.561232    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schecjgww"]
Jul 28 18:26:55.152551 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:26:55.152520    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04e17f16-b4ef-4f55-a41e-20da7fcc90d2" path="/var/lib/kubelet/pods/04e17f16-b4ef-4f55-a41e-20da7fcc90d2/volumes"
Jul 28 18:27:07.998800 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:27:07.998758    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8" podUID="1c409c83-3d4a-40a6-9d31-4982e37030c4"
Jul 28 18:27:08.621823 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:27:08.621794    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:29:11.623318 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:29:11.623222    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8" podUID="1c409c83-3d4a-40a6-9d31-4982e37030c4"
Jul 28 18:29:12.244221 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:29:12.244190    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:31:15.244841 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:31:15.244794    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8" podUID="1c409c83-3d4a-40a6-9d31-4982e37030c4"
Jul 28 18:31:15.879416 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:31:15.879383    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:33:18.880217 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:33:18.880173    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8" podUID="1c409c83-3d4a-40a6-9d31-4982e37030c4"
Jul 28 18:33:19.485742 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:33:19.485707    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:34:14.003856 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:14.003769    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 28 18:34:14.012734 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:34:14.012692    2543 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 18:34:14.012734 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:34:14.012734 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 18:34:14.012734 ip-10-0-135-177 kubenswrapper[2543]:  > containerID="8bae7d1c6b7f416383eefe5f7e1a0ac833a29f15399269e8ff3b681faaddcd6c" cmd=["/bin/sleep","15"]
Jul 28 18:34:14.012995 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:14.012746    2543 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" message="" err=<
Jul 28 18:34:14.012995 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:34:14.012995 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 18:34:14.012995 ip-10-0-135-177 kubenswrapper[2543]:  >
Jul 28 18:34:14.012995 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:34:14.012758    2543 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 18:34:14.012995 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:34:14.012995 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 18:34:14.012995 ip-10-0-135-177 kubenswrapper[2543]:  > pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" podUID="f4337ce4-acb9-459b-b984-ee75b0aa4149" containerName="main" containerID="cri-o://8bae7d1c6b7f416383eefe5f7e1a0ac833a29f15399269e8ff3b681faaddcd6c"
Jul 28 18:34:14.012995 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:14.012800    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" podUID="f4337ce4-acb9-459b-b984-ee75b0aa4149" containerName="main" containerID="cri-o://8bae7d1c6b7f416383eefe5f7e1a0ac833a29f15399269e8ff3b681faaddcd6c" gracePeriod=60
Jul 28 18:34:14.366917 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:14.366892    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:34:14.367256 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:14.367239    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4337ce4-acb9-459b-b984-ee75b0aa4149-tls-certs\") pod \"f4337ce4-acb9-459b-b984-ee75b0aa4149\" (UID: \"f4337ce4-acb9-459b-b984-ee75b0aa4149\") "
Jul 28 18:34:14.367318 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:14.367278    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4337ce4-acb9-459b-b984-ee75b0aa4149-model-cache\") pod \"f4337ce4-acb9-459b-b984-ee75b0aa4149\" (UID: \"f4337ce4-acb9-459b-b984-ee75b0aa4149\") "
Jul 28 18:34:14.367318 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:14.367302    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4337ce4-acb9-459b-b984-ee75b0aa4149-dshm\") pod \"f4337ce4-acb9-459b-b984-ee75b0aa4149\" (UID: \"f4337ce4-acb9-459b-b984-ee75b0aa4149\") "
Jul 28 18:34:14.367404 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:14.367356    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wr74s\" (UniqueName: \"kubernetes.io/projected/f4337ce4-acb9-459b-b984-ee75b0aa4149-kube-api-access-wr74s\") pod \"f4337ce4-acb9-459b-b984-ee75b0aa4149\" (UID: \"f4337ce4-acb9-459b-b984-ee75b0aa4149\") "
Jul 28 18:34:14.367404 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:14.367388    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4337ce4-acb9-459b-b984-ee75b0aa4149-home\") pod \"f4337ce4-acb9-459b-b984-ee75b0aa4149\" (UID: \"f4337ce4-acb9-459b-b984-ee75b0aa4149\") "
Jul 28 18:34:14.367512 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:14.367421    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4337ce4-acb9-459b-b984-ee75b0aa4149-tmp-dir\") pod \"f4337ce4-acb9-459b-b984-ee75b0aa4149\" (UID: \"f4337ce4-acb9-459b-b984-ee75b0aa4149\") "
Jul 28 18:34:14.367573 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:14.367518    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4337ce4-acb9-459b-b984-ee75b0aa4149-model-cache" (OuterVolumeSpecName: "model-cache") pod "f4337ce4-acb9-459b-b984-ee75b0aa4149" (UID: "f4337ce4-acb9-459b-b984-ee75b0aa4149"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:34:14.367734 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:14.367714    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4337ce4-acb9-459b-b984-ee75b0aa4149-model-cache\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:34:14.367801 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:14.367717    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4337ce4-acb9-459b-b984-ee75b0aa4149-home" (OuterVolumeSpecName: "home") pod "f4337ce4-acb9-459b-b984-ee75b0aa4149" (UID: "f4337ce4-acb9-459b-b984-ee75b0aa4149"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:34:14.367801 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:14.367727    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4337ce4-acb9-459b-b984-ee75b0aa4149-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f4337ce4-acb9-459b-b984-ee75b0aa4149" (UID: "f4337ce4-acb9-459b-b984-ee75b0aa4149"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:34:14.369493 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:14.369464    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4337ce4-acb9-459b-b984-ee75b0aa4149-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f4337ce4-acb9-459b-b984-ee75b0aa4149" (UID: "f4337ce4-acb9-459b-b984-ee75b0aa4149"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:34:14.369594 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:14.369505    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4337ce4-acb9-459b-b984-ee75b0aa4149-kube-api-access-wr74s" (OuterVolumeSpecName: "kube-api-access-wr74s") pod "f4337ce4-acb9-459b-b984-ee75b0aa4149" (UID: "f4337ce4-acb9-459b-b984-ee75b0aa4149"). InnerVolumeSpecName "kube-api-access-wr74s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:34:14.369594 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:14.369550    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4337ce4-acb9-459b-b984-ee75b0aa4149-dshm" (OuterVolumeSpecName: "dshm") pod "f4337ce4-acb9-459b-b984-ee75b0aa4149" (UID: "f4337ce4-acb9-459b-b984-ee75b0aa4149"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:34:14.468916 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:14.468884    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4337ce4-acb9-459b-b984-ee75b0aa4149-dshm\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:34:14.468916 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:14.468915    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr74s\" (UniqueName: \"kubernetes.io/projected/f4337ce4-acb9-459b-b984-ee75b0aa4149-kube-api-access-wr74s\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:34:14.469107 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:14.468928    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4337ce4-acb9-459b-b984-ee75b0aa4149-home\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:34:14.469107 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:14.468940    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4337ce4-acb9-459b-b984-ee75b0aa4149-tmp-dir\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:34:14.469107 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:14.468951    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4337ce4-acb9-459b-b984-ee75b0aa4149-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:34:14.773835 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:14.773798    2543 generic.go:356] "Generic (PLEG): container finished" podID="f4337ce4-acb9-459b-b984-ee75b0aa4149" containerID="8bae7d1c6b7f416383eefe5f7e1a0ac833a29f15399269e8ff3b681faaddcd6c" exitCode=0
Jul 28 18:34:14.774018 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:14.773842    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 28 18:34:14.774018 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:14.773881    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" event={"ID":"f4337ce4-acb9-459b-b984-ee75b0aa4149","Type":"ContainerDied","Data":"8bae7d1c6b7f416383eefe5f7e1a0ac833a29f15399269e8ff3b681faaddcd6c"}
Jul 28 18:34:14.774018 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:14.773930    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" event={"ID":"f4337ce4-acb9-459b-b984-ee75b0aa4149","Type":"ContainerDied","Data":"6ed3319902cbaf2e59fd5d968f78acc1ca3ea1d21f1364fea91d4a422f538c60"}
Jul 28 18:34:14.774018 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:14.773955    2543 scope.go:117] "RemoveContainer" containerID="8bae7d1c6b7f416383eefe5f7e1a0ac833a29f15399269e8ff3b681faaddcd6c"
Jul 28 18:34:14.783811 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:14.783794    2543 scope.go:117] "RemoveContainer" containerID="8bae7d1c6b7f416383eefe5f7e1a0ac833a29f15399269e8ff3b681faaddcd6c"
Jul 28 18:34:14.784072 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:34:14.784052    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bae7d1c6b7f416383eefe5f7e1a0ac833a29f15399269e8ff3b681faaddcd6c\": container with ID starting with 8bae7d1c6b7f416383eefe5f7e1a0ac833a29f15399269e8ff3b681faaddcd6c not found: ID does not exist" containerID="8bae7d1c6b7f416383eefe5f7e1a0ac833a29f15399269e8ff3b681faaddcd6c"
Jul 28 18:34:14.784125 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:14.784081    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bae7d1c6b7f416383eefe5f7e1a0ac833a29f15399269e8ff3b681faaddcd6c"} err="failed to get container status \"8bae7d1c6b7f416383eefe5f7e1a0ac833a29f15399269e8ff3b681faaddcd6c\": rpc error: code = NotFound desc = could not find container \"8bae7d1c6b7f416383eefe5f7e1a0ac833a29f15399269e8ff3b681faaddcd6c\": container with ID starting with 8bae7d1c6b7f416383eefe5f7e1a0ac833a29f15399269e8ff3b681faaddcd6c not found: ID does not exist"
Jul 28 18:34:14.798088 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:14.798065    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 28 18:34:14.803008 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:14.802987    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 28 18:34:15.152301 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:15.152225    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4337ce4-acb9-459b-b984-ee75b0aa4149" path="/var/lib/kubelet/pods/f4337ce4-acb9-459b-b984-ee75b0aa4149/volumes"
Jul 28 18:34:47.028584 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.028550    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 28 18:34:47.029261 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.028895    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4337ce4-acb9-459b-b984-ee75b0aa4149" containerName="main"
Jul 28 18:34:47.029261 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.028905    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4337ce4-acb9-459b-b984-ee75b0aa4149" containerName="main"
Jul 28 18:34:47.029261 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.028926    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04e17f16-b4ef-4f55-a41e-20da7fcc90d2" containerName="main"
Jul 28 18:34:47.029261 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.028931    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="04e17f16-b4ef-4f55-a41e-20da7fcc90d2" containerName="main"
Jul 28 18:34:47.029261 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.028982    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="04e17f16-b4ef-4f55-a41e-20da7fcc90d2" containerName="main"
Jul 28 18:34:47.029261 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.028991    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4337ce4-acb9-459b-b984-ee75b0aa4149" containerName="main"
Jul 28 18:34:47.031973 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.031957    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:34:47.035052 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.035031    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-kserve-mn-dockercfg-pfdqg\""
Jul 28 18:34:47.036102 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.036080    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"kube-root-ca.crt\""
Jul 28 18:34:47.036247 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.036080    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-kserve-self-signed-certs\""
Jul 28 18:34:47.036247 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.036182    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"default-dockercfg-qr8kw\""
Jul 28 18:34:47.036247 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.036205    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"openshift-service-ca.crt\""
Jul 28 18:34:47.046215 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.046192    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 28 18:34:47.157449 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.157418    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d8a975f-9633-46bf-b87f-e97956c18a68-dshm\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"5d8a975f-9633-46bf-b87f-e97956c18a68\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:34:47.157617 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.157471    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d8a975f-9633-46bf-b87f-e97956c18a68-home\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"5d8a975f-9633-46bf-b87f-e97956c18a68\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:34:47.157617 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.157494    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d8a975f-9633-46bf-b87f-e97956c18a68-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"5d8a975f-9633-46bf-b87f-e97956c18a68\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:34:47.157617 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.157550    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d8a975f-9633-46bf-b87f-e97956c18a68-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"5d8a975f-9633-46bf-b87f-e97956c18a68\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:34:47.157739 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.157613    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v5nr\" (UniqueName: \"kubernetes.io/projected/5d8a975f-9633-46bf-b87f-e97956c18a68-kube-api-access-5v5nr\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"5d8a975f-9633-46bf-b87f-e97956c18a68\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:34:47.157739 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.157648    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d8a975f-9633-46bf-b87f-e97956c18a68-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"5d8a975f-9633-46bf-b87f-e97956c18a68\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:34:47.258769 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.258732    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d8a975f-9633-46bf-b87f-e97956c18a68-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"5d8a975f-9633-46bf-b87f-e97956c18a68\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:34:47.258939 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.258817    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d8a975f-9633-46bf-b87f-e97956c18a68-dshm\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"5d8a975f-9633-46bf-b87f-e97956c18a68\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:34:47.258939 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.258876    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d8a975f-9633-46bf-b87f-e97956c18a68-home\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"5d8a975f-9633-46bf-b87f-e97956c18a68\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:34:47.258939 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.258908    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d8a975f-9633-46bf-b87f-e97956c18a68-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"5d8a975f-9633-46bf-b87f-e97956c18a68\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:34:47.259070 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.258939    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d8a975f-9633-46bf-b87f-e97956c18a68-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"5d8a975f-9633-46bf-b87f-e97956c18a68\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:34:47.259070 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.258992    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v5nr\" (UniqueName: \"kubernetes.io/projected/5d8a975f-9633-46bf-b87f-e97956c18a68-kube-api-access-5v5nr\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"5d8a975f-9633-46bf-b87f-e97956c18a68\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:34:47.259333 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.259311    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d8a975f-9633-46bf-b87f-e97956c18a68-home\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"5d8a975f-9633-46bf-b87f-e97956c18a68\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:34:47.259333 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.259324    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d8a975f-9633-46bf-b87f-e97956c18a68-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"5d8a975f-9633-46bf-b87f-e97956c18a68\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:34:47.259431 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.259376    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d8a975f-9633-46bf-b87f-e97956c18a68-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"5d8a975f-9633-46bf-b87f-e97956c18a68\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:34:47.261207 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.261188    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d8a975f-9633-46bf-b87f-e97956c18a68-dshm\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"5d8a975f-9633-46bf-b87f-e97956c18a68\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:34:47.261275 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.261259    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d8a975f-9633-46bf-b87f-e97956c18a68-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"5d8a975f-9633-46bf-b87f-e97956c18a68\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:34:47.270028 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.269999    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v5nr\" (UniqueName: \"kubernetes.io/projected/5d8a975f-9633-46bf-b87f-e97956c18a68-kube-api-access-5v5nr\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"5d8a975f-9633-46bf-b87f-e97956c18a68\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:34:47.342339 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.342264    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:34:47.479077 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.479045    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 28 18:34:47.482153 ip-10-0-135-177 kubenswrapper[2543]: W0728 18:34:47.482111    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d8a975f_9633_46bf_b87f_e97956c18a68.slice/crio-dfa493c2d7bd4da7521fb0635847410d5a6dc96661c0ba79c672087afd184d0e WatchSource:0}: Error finding container dfa493c2d7bd4da7521fb0635847410d5a6dc96661c0ba79c672087afd184d0e: Status 404 returned error can't find the container with id dfa493c2d7bd4da7521fb0635847410d5a6dc96661c0ba79c672087afd184d0e
Jul 28 18:34:47.938988 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.938952    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" event={"ID":"5d8a975f-9633-46bf-b87f-e97956c18a68","Type":"ContainerStarted","Data":"ca1cb8cf17fb5f57d80be7ab13d2306fdac2645634a7512e695210d389b50dc4"}
Jul 28 18:34:47.939186 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.939004    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" event={"ID":"5d8a975f-9633-46bf-b87f-e97956c18a68","Type":"ContainerStarted","Data":"dfa493c2d7bd4da7521fb0635847410d5a6dc96661c0ba79c672087afd184d0e"}
Jul 28 18:34:47.961360 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:34:47.961299    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" podStartSLOduration=1.961280149 podStartE2EDuration="1.961280149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:34:47.96083051 +0000 UTC m=+4291.367104251" watchObservedRunningTime="2026-07-28 18:34:47.961280149 +0000 UTC m=+4291.367553890"
Jul 28 18:35:22.486857 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:35:22.486755    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8" podUID="1c409c83-3d4a-40a6-9d31-4982e37030c4"
Jul 28 18:35:23.121070 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:35:23.121039    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:37:26.121979 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:37:26.121921    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8" podUID="1c409c83-3d4a-40a6-9d31-4982e37030c4"
Jul 28 18:37:26.752306 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:37:26.752277    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:39:29.753529 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:39:29.753478    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8" podUID="1c409c83-3d4a-40a6-9d31-4982e37030c4"
Jul 28 18:39:30.375075 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:39:30.375041    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:41:33.376537 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:41:33.376452    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8" podUID="1c409c83-3d4a-40a6-9d31-4982e37030c4"
Jul 28 18:41:33.995844 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:41:33.995811    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:42:48.226554 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.226473    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"]
Jul 28 18:42:48.229903 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.229885    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"
Jul 28 18:42:48.232423 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.232387    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Jul 28 18:42:48.232423 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.232387    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-9jqlj\""
Jul 28 18:42:48.232580 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.232434    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Jul 28 18:42:48.232580 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.232499    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:42:48.232580 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.232577    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Jul 28 18:42:48.232743 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.232689    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Jul 28 18:42:48.249251 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.249226    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"]
Jul 28 18:42:48.319357 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.319316    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh8p8\" (UniqueName: \"kubernetes.io/projected/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-kube-api-access-nh8p8\") pod \"lora-single-adapter-test-kserve-75845869bf-56bvz\" (UID: \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"
Jul 28 18:42:48.319528 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.319372    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-tls-certs\") pod \"lora-single-adapter-test-kserve-75845869bf-56bvz\" (UID: \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"
Jul 28 18:42:48.319528 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.319408    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-75845869bf-56bvz\" (UID: \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"
Jul 28 18:42:48.319528 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.319441    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-cabundle-cert\") pod \"lora-single-adapter-test-kserve-75845869bf-56bvz\" (UID: \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"
Jul 28 18:42:48.319528 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.319475    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-home\") pod \"lora-single-adapter-test-kserve-75845869bf-56bvz\" (UID: \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"
Jul 28 18:42:48.319528 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.319509    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-dshm\") pod \"lora-single-adapter-test-kserve-75845869bf-56bvz\" (UID: \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"
Jul 28 18:42:48.319701 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.319578    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-model-cache\") pod \"lora-single-adapter-test-kserve-75845869bf-56bvz\" (UID: \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"
Jul 28 18:42:48.319701 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.319602    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-tmp-dir\") pod \"lora-single-adapter-test-kserve-75845869bf-56bvz\" (UID: \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"
Jul 28 18:42:48.420821 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.420789    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-model-cache\") pod \"lora-single-adapter-test-kserve-75845869bf-56bvz\" (UID: \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"
Jul 28 18:42:48.420821 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.420819    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-tmp-dir\") pod \"lora-single-adapter-test-kserve-75845869bf-56bvz\" (UID: \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"
Jul 28 18:42:48.421073 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.420842    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh8p8\" (UniqueName: \"kubernetes.io/projected/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-kube-api-access-nh8p8\") pod \"lora-single-adapter-test-kserve-75845869bf-56bvz\" (UID: \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"
Jul 28 18:42:48.421073 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.420973    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-tls-certs\") pod \"lora-single-adapter-test-kserve-75845869bf-56bvz\" (UID: \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"
Jul 28 18:42:48.421073 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.421017    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-75845869bf-56bvz\" (UID: \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"
Jul 28 18:42:48.421073 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.421067    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-cabundle-cert\") pod \"lora-single-adapter-test-kserve-75845869bf-56bvz\" (UID: \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"
Jul 28 18:42:48.421321 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.421157    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-home\") pod \"lora-single-adapter-test-kserve-75845869bf-56bvz\" (UID: \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"
Jul 28 18:42:48.421321 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.421216    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-dshm\") pod \"lora-single-adapter-test-kserve-75845869bf-56bvz\" (UID: \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"
Jul 28 18:42:48.421321 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.421291    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-tmp-dir\") pod \"lora-single-adapter-test-kserve-75845869bf-56bvz\" (UID: \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"
Jul 28 18:42:48.421458 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.421352    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-model-cache\") pod \"lora-single-adapter-test-kserve-75845869bf-56bvz\" (UID: \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"
Jul 28 18:42:48.421458 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.421421    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-75845869bf-56bvz\" (UID: \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"
Jul 28 18:42:48.421541 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.421520    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-home\") pod \"lora-single-adapter-test-kserve-75845869bf-56bvz\" (UID: \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"
Jul 28 18:42:48.421770 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.421753    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-cabundle-cert\") pod \"lora-single-adapter-test-kserve-75845869bf-56bvz\" (UID: \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"
Jul 28 18:42:48.423400 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.423383    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-dshm\") pod \"lora-single-adapter-test-kserve-75845869bf-56bvz\" (UID: \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"
Jul 28 18:42:48.423570 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.423555    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-tls-certs\") pod \"lora-single-adapter-test-kserve-75845869bf-56bvz\" (UID: \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"
Jul 28 18:42:48.429146 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.429109    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh8p8\" (UniqueName: \"kubernetes.io/projected/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-kube-api-access-nh8p8\") pod \"lora-single-adapter-test-kserve-75845869bf-56bvz\" (UID: \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"
Jul 28 18:42:48.540438 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.540404    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"
Jul 28 18:42:48.679183 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:48.679155    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"]
Jul 28 18:42:48.680655 ip-10-0-135-177 kubenswrapper[2543]: W0728 18:42:48.680631    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e50a62a_da2f_40a2_a45f_1b5ebf7b13c6.slice/crio-bb8e7c6208e086c405f249d2a16a39b1896d6b5822d5b2431cb028848a65e92c WatchSource:0}: Error finding container bb8e7c6208e086c405f249d2a16a39b1896d6b5822d5b2431cb028848a65e92c: Status 404 returned error can't find the container with id bb8e7c6208e086c405f249d2a16a39b1896d6b5822d5b2431cb028848a65e92c
Jul 28 18:42:49.373117 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:49.373078    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz" event={"ID":"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6","Type":"ContainerStarted","Data":"1e3944df21e66845e29ce65648817bc257c30ebda25146fb201432fd37e13613"}
Jul 28 18:42:49.373631 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:49.373143    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz" event={"ID":"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6","Type":"ContainerStarted","Data":"bb8e7c6208e086c405f249d2a16a39b1896d6b5822d5b2431cb028848a65e92c"}
Jul 28 18:42:59.427048 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:59.426955    2543 generic.go:356] "Generic (PLEG): container finished" podID="9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6" containerID="1e3944df21e66845e29ce65648817bc257c30ebda25146fb201432fd37e13613" exitCode=0
Jul 28 18:42:59.427048 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:42:59.427033    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz" event={"ID":"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6","Type":"ContainerDied","Data":"1e3944df21e66845e29ce65648817bc257c30ebda25146fb201432fd37e13613"}
Jul 28 18:43:00.434124 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:43:00.434090    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz" event={"ID":"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6","Type":"ContainerStarted","Data":"7a1b6c038ba9d236b7c02e7f5de01383bf3d1747766378c773ddfea112658e69"}
Jul 28 18:43:00.456392 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:43:00.456343    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz" podStartSLOduration=2.184580124 podStartE2EDuration="12.456327343s" totalImagesPullingTime="0s" totalInitContainerRuntime="10.271747219s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:43:00.45466739 +0000 UTC m=+4783.860941131" watchObservedRunningTime="2026-07-28 18:43:00.456327343 +0000 UTC m=+4783.862601084"
Jul 28 18:43:00.541286 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:43:00.541251    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"
Jul 28 18:43:08.540593 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:43:08.540562    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"
Jul 28 18:43:08.542332 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:43:08.542303    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz" podUID="9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 28 18:43:18.541244 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:43:18.541179    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz" podUID="9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 28 18:43:28.541070 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:43:28.540999    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz" podUID="9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 28 18:43:36.997127 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:43:36.997078    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8" podUID="1c409c83-3d4a-40a6-9d31-4982e37030c4"
Jul 28 18:43:37.638870 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:43:37.638836    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:43:38.541391 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:43:38.541345    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz" podUID="9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 28 18:43:48.541265 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:43:48.541206    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz" podUID="9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 28 18:43:58.541069 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:43:58.540947    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz" podUID="9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 28 18:44:08.540925 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:44:08.540875    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz" podUID="9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 28 18:44:18.554597 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:44:18.554563    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"
Jul 28 18:44:18.564985 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:44:18.564950    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"
Jul 28 18:44:58.305182 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:44:58.305149    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-75845869bf-56bvz_9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6/storage-initializer/0.log"
Jul 28 18:44:58.314765 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:44:58.314743    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-75845869bf-56bvz_9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6/main/0.log"
Jul 28 18:45:06.086462 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:06.086427    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"]
Jul 28 18:45:21.109000 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:21.108918    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz" podUID="9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6" containerName="main" containerID="cri-o://7a1b6c038ba9d236b7c02e7f5de01383bf3d1747766378c773ddfea112658e69" gracePeriod=45
Jul 28 18:45:21.555127 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:21.555094    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz" podUID="9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": dial tcp 10.134.0.76:8000: connect: connection refused"
Jul 28 18:45:21.763730 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:21.763705    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"
Jul 28 18:45:21.867433 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:21.867347    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-cabundle-cert\") pod \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\" (UID: \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\") "
Jul 28 18:45:21.867433 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:21.867393    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-tmp-dir\") pod \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\" (UID: \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\") "
Jul 28 18:45:21.867658 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:21.867439    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-tls-certs\") pod \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\" (UID: \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\") "
Jul 28 18:45:21.867658 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:21.867510    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-kserve-provision-location\") pod \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\" (UID: \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\") "
Jul 28 18:45:21.867658 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:21.867556    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-dshm\") pod \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\" (UID: \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\") "
Jul 28 18:45:21.867658 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:21.867584    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-model-cache\") pod \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\" (UID: \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\") "
Jul 28 18:45:21.867658 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:21.867615    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nh8p8\" (UniqueName: \"kubernetes.io/projected/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-kube-api-access-nh8p8\") pod \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\" (UID: \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\") "
Jul 28 18:45:21.867658 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:21.867648    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-home\") pod \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\" (UID: \"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6\") "
Jul 28 18:45:21.867978 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:21.867785    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6" (UID: "9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:45:21.867978 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:21.867819    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-model-cache" (OuterVolumeSpecName: "model-cache") pod "9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6" (UID: "9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:45:21.868114 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:21.868019    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-model-cache\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:45:21.868114 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:21.868041    2543 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-cabundle-cert\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:45:21.868114 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:21.868103    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-home" (OuterVolumeSpecName: "home") pod "9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6" (UID: "9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:45:21.869884 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:21.869859    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6" (UID: "9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:45:21.870012 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:21.869950    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-dshm" (OuterVolumeSpecName: "dshm") pod "9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6" (UID: "9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:45:21.870081 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:21.870026    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-kube-api-access-nh8p8" (OuterVolumeSpecName: "kube-api-access-nh8p8") pod "9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6" (UID: "9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6"). InnerVolumeSpecName "kube-api-access-nh8p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:45:21.884835 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:21.884785    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6" (UID: "9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:45:21.936431 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:21.936389    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6" (UID: "9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:45:21.968716 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:21.968681    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-kserve-provision-location\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:45:21.968716 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:21.968706    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-dshm\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:45:21.968716 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:21.968717    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh8p8\" (UniqueName: \"kubernetes.io/projected/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-kube-api-access-nh8p8\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:45:21.968918 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:21.968727    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-home\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:45:21.968918 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:21.968735    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-tmp-dir\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:45:21.968918 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:21.968744    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:45:22.217947 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:22.217853    2543 generic.go:356] "Generic (PLEG): container finished" podID="9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6" containerID="7a1b6c038ba9d236b7c02e7f5de01383bf3d1747766378c773ddfea112658e69" exitCode=0
Jul 28 18:45:22.217947 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:22.217912    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"
Jul 28 18:45:22.217947 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:22.217940    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz" event={"ID":"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6","Type":"ContainerDied","Data":"7a1b6c038ba9d236b7c02e7f5de01383bf3d1747766378c773ddfea112658e69"}
Jul 28 18:45:22.218573 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:22.217995    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz" event={"ID":"9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6","Type":"ContainerDied","Data":"bb8e7c6208e086c405f249d2a16a39b1896d6b5822d5b2431cb028848a65e92c"}
Jul 28 18:45:22.218573 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:22.218019    2543 scope.go:117] "RemoveContainer" containerID="7a1b6c038ba9d236b7c02e7f5de01383bf3d1747766378c773ddfea112658e69"
Jul 28 18:45:22.231856 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:22.229217    2543 scope.go:117] "RemoveContainer" containerID="1e3944df21e66845e29ce65648817bc257c30ebda25146fb201432fd37e13613"
Jul 28 18:45:22.239490 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:22.239460    2543 scope.go:117] "RemoveContainer" containerID="7a1b6c038ba9d236b7c02e7f5de01383bf3d1747766378c773ddfea112658e69"
Jul 28 18:45:22.239864 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:45:22.239837    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a1b6c038ba9d236b7c02e7f5de01383bf3d1747766378c773ddfea112658e69\": container with ID starting with 7a1b6c038ba9d236b7c02e7f5de01383bf3d1747766378c773ddfea112658e69 not found: ID does not exist" containerID="7a1b6c038ba9d236b7c02e7f5de01383bf3d1747766378c773ddfea112658e69"
Jul 28 18:45:22.239918 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:22.239881    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a1b6c038ba9d236b7c02e7f5de01383bf3d1747766378c773ddfea112658e69"} err="failed to get container status \"7a1b6c038ba9d236b7c02e7f5de01383bf3d1747766378c773ddfea112658e69\": rpc error: code = NotFound desc = could not find container \"7a1b6c038ba9d236b7c02e7f5de01383bf3d1747766378c773ddfea112658e69\": container with ID starting with 7a1b6c038ba9d236b7c02e7f5de01383bf3d1747766378c773ddfea112658e69 not found: ID does not exist"
Jul 28 18:45:22.239918 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:22.239911    2543 scope.go:117] "RemoveContainer" containerID="1e3944df21e66845e29ce65648817bc257c30ebda25146fb201432fd37e13613"
Jul 28 18:45:22.240239 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:45:22.240215    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e3944df21e66845e29ce65648817bc257c30ebda25146fb201432fd37e13613\": container with ID starting with 1e3944df21e66845e29ce65648817bc257c30ebda25146fb201432fd37e13613 not found: ID does not exist" containerID="1e3944df21e66845e29ce65648817bc257c30ebda25146fb201432fd37e13613"
Jul 28 18:45:22.240349 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:22.240241    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e3944df21e66845e29ce65648817bc257c30ebda25146fb201432fd37e13613"} err="failed to get container status \"1e3944df21e66845e29ce65648817bc257c30ebda25146fb201432fd37e13613\": rpc error: code = NotFound desc = could not find container \"1e3944df21e66845e29ce65648817bc257c30ebda25146fb201432fd37e13613\": container with ID starting with 1e3944df21e66845e29ce65648817bc257c30ebda25146fb201432fd37e13613 not found: ID does not exist"
Jul 28 18:45:22.247880 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:22.247840    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"]
Jul 28 18:45:22.250677 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:22.250655    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-75845869bf-56bvz"]
Jul 28 18:45:23.152176 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:23.152123    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6" path="/var/lib/kubelet/pods/9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6/volumes"
Jul 28 18:45:32.327989 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.327954    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"]
Jul 28 18:45:32.328436 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.328328    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6" containerName="main"
Jul 28 18:45:32.328436 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.328339    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6" containerName="main"
Jul 28 18:45:32.328436 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.328348    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6" containerName="storage-initializer"
Jul 28 18:45:32.328436 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.328354    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6" containerName="storage-initializer"
Jul 28 18:45:32.328436 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.328404    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e50a62a-da2f-40a2-a45f-1b5ebf7b13c6" containerName="main"
Jul 28 18:45:32.333164 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.333125    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"
Jul 28 18:45:32.335722 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.335696    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-hqvvj\""
Jul 28 18:45:32.338670 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.338653    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Jul 28 18:45:32.338849 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.338823    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Jul 28 18:45:32.338966 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.338831    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:45:32.338966 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.338878    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Jul 28 18:45:32.338966 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.338890    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Jul 28 18:45:32.347959 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.347937    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"]
Jul 28 18:45:32.464863 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.464832    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-k7q5w\" (UID: \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"
Jul 28 18:45:32.465021 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.464948    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-k7q5w\" (UID: \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"
Jul 28 18:45:32.465021 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.464977    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-dshm\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-k7q5w\" (UID: \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"
Jul 28 18:45:32.465021 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.465004    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qmss\" (UniqueName: \"kubernetes.io/projected/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-kube-api-access-9qmss\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-k7q5w\" (UID: \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"
Jul 28 18:45:32.465236 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.465024    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-home\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-k7q5w\" (UID: \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"
Jul 28 18:45:32.465236 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.465049    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-k7q5w\" (UID: \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"
Jul 28 18:45:32.465236 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.465157    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-model-cache\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-k7q5w\" (UID: \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"
Jul 28 18:45:32.465236 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.465227    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-k7q5w\" (UID: \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"
Jul 28 18:45:32.565931 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.565893    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-k7q5w\" (UID: \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"
Jul 28 18:45:32.565931 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.565931    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-dshm\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-k7q5w\" (UID: \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"
Jul 28 18:45:32.566206 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.565959    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qmss\" (UniqueName: \"kubernetes.io/projected/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-kube-api-access-9qmss\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-k7q5w\" (UID: \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"
Jul 28 18:45:32.566206 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.565985    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-home\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-k7q5w\" (UID: \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"
Jul 28 18:45:32.566206 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.566017    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-k7q5w\" (UID: \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"
Jul 28 18:45:32.566206 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.566044    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-model-cache\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-k7q5w\" (UID: \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"
Jul 28 18:45:32.566206 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.566090    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-k7q5w\" (UID: \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"
Jul 28 18:45:32.566206 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.566121    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-k7q5w\" (UID: \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"
Jul 28 18:45:32.566549 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.566453    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-home\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-k7q5w\" (UID: \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"
Jul 28 18:45:32.566640 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.566617    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-model-cache\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-k7q5w\" (UID: \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"
Jul 28 18:45:32.566711 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.566626    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-k7q5w\" (UID: \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"
Jul 28 18:45:32.566711 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.566700    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-k7q5w\" (UID: \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"
Jul 28 18:45:32.567023 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.567001    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-k7q5w\" (UID: \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"
Jul 28 18:45:32.568761 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.568733    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-dshm\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-k7q5w\" (UID: \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"
Jul 28 18:45:32.569002 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.568984    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-k7q5w\" (UID: \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"
Jul 28 18:45:32.574103 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.574085    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qmss\" (UniqueName: \"kubernetes.io/projected/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-kube-api-access-9qmss\") pod \"lora-multiple-adapters-test-kserve-6d8b575478-k7q5w\" (UID: \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"
Jul 28 18:45:32.644000 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.643911    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"
Jul 28 18:45:32.783768 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:32.783742    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"]
Jul 28 18:45:32.785324 ip-10-0-135-177 kubenswrapper[2543]: W0728 18:45:32.785293    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea22d8d5_744d_4f36_a295_e1b70e3a60c5.slice/crio-efbf4aa555ac0a0b2e8de03a856511ed988362fc02fd5011bafaf6f5fb7a970c WatchSource:0}: Error finding container efbf4aa555ac0a0b2e8de03a856511ed988362fc02fd5011bafaf6f5fb7a970c: Status 404 returned error can't find the container with id efbf4aa555ac0a0b2e8de03a856511ed988362fc02fd5011bafaf6f5fb7a970c
Jul 28 18:45:33.277213 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:33.277170    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w" event={"ID":"ea22d8d5-744d-4f36-a295-e1b70e3a60c5","Type":"ContainerStarted","Data":"08c4e98d27c8a275a1c14ef52ef1063d8d77f5939817d5420975de2e48052c83"}
Jul 28 18:45:33.277431 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:33.277244    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w" event={"ID":"ea22d8d5-744d-4f36-a295-e1b70e3a60c5","Type":"ContainerStarted","Data":"efbf4aa555ac0a0b2e8de03a856511ed988362fc02fd5011bafaf6f5fb7a970c"}
Jul 28 18:45:40.640026 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:45:40.639933    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8" podUID="1c409c83-3d4a-40a6-9d31-4982e37030c4"
Jul 28 18:45:41.317211 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:41.317178    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:45:56.392548 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:56.392462    2543 generic.go:356] "Generic (PLEG): container finished" podID="ea22d8d5-744d-4f36-a295-e1b70e3a60c5" containerID="08c4e98d27c8a275a1c14ef52ef1063d8d77f5939817d5420975de2e48052c83" exitCode=0
Jul 28 18:45:56.393019 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:56.392541    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w" event={"ID":"ea22d8d5-744d-4f36-a295-e1b70e3a60c5","Type":"ContainerDied","Data":"08c4e98d27c8a275a1c14ef52ef1063d8d77f5939817d5420975de2e48052c83"}
Jul 28 18:45:57.400173 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:57.400123    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w" event={"ID":"ea22d8d5-744d-4f36-a295-e1b70e3a60c5","Type":"ContainerStarted","Data":"ef79e319bade851c1fdfa8d495fbe483a9ed6e9ca05253033865389e2d040312"}
Jul 28 18:45:57.427090 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:57.427034    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w" podStartSLOduration=2.181446489 podStartE2EDuration="25.427019715s" totalImagesPullingTime="0s" totalInitContainerRuntime="23.245573226s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:45:57.424206922 +0000 UTC m=+4960.830480664" watchObservedRunningTime="2026-07-28 18:45:57.427019715 +0000 UTC m=+4960.833293455"
Jul 28 18:45:57.645103 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:45:57.645069    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"
Jul 28 18:46:02.644379 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:46:02.644342    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"
Jul 28 18:46:02.646268 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:46:02.646234    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w" podUID="ea22d8d5-744d-4f36-a295-e1b70e3a60c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Jul 28 18:46:12.644762 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:46:12.644714    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w" podUID="ea22d8d5-744d-4f36-a295-e1b70e3a60c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Jul 28 18:46:22.645191 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:46:22.645147    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w" podUID="ea22d8d5-744d-4f36-a295-e1b70e3a60c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Jul 28 18:46:32.644994 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:46:32.644949    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w" podUID="ea22d8d5-744d-4f36-a295-e1b70e3a60c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Jul 28 18:46:42.645084 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:46:42.645025    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w" podUID="ea22d8d5-744d-4f36-a295-e1b70e3a60c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Jul 28 18:46:52.645297 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:46:52.645247    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w" podUID="ea22d8d5-744d-4f36-a295-e1b70e3a60c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Jul 28 18:47:02.645153 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:47:02.645093    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w" podUID="ea22d8d5-744d-4f36-a295-e1b70e3a60c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Jul 28 18:47:12.655426 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:47:12.655344    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"
Jul 28 18:47:12.663539 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:47:12.663511    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"
Jul 28 18:47:44.318147 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:47:44.318095    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8" podUID="1c409c83-3d4a-40a6-9d31-4982e37030c4"
Jul 28 18:47:45.011681 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:47:45.011650    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:48:04.681051 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:04.681020    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-6d8b575478-k7q5w_ea22d8d5-744d-4f36-a295-e1b70e3a60c5/storage-initializer/0.log"
Jul 28 18:48:04.690280 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:04.690257    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-6d8b575478-k7q5w_ea22d8d5-744d-4f36-a295-e1b70e3a60c5/main/0.log"
Jul 28 18:48:21.997196 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:21.997161    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"]
Jul 28 18:48:37.020083 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:37.020030    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w" podUID="ea22d8d5-744d-4f36-a295-e1b70e3a60c5" containerName="main" containerID="cri-o://ef79e319bade851c1fdfa8d495fbe483a9ed6e9ca05253033865389e2d040312" gracePeriod=45
Jul 28 18:48:37.678364 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:37.678341    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"
Jul 28 18:48:37.799242 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:37.799208    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-tls-certs\") pod \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\" (UID: \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\") "
Jul 28 18:48:37.799411 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:37.799254    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-tmp-dir\") pod \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\" (UID: \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\") "
Jul 28 18:48:37.799411 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:37.799312    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-home\") pod \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\" (UID: \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\") "
Jul 28 18:48:37.799411 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:37.799338    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qmss\" (UniqueName: \"kubernetes.io/projected/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-kube-api-access-9qmss\") pod \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\" (UID: \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\") "
Jul 28 18:48:37.799411 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:37.799377    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-cabundle-cert\") pod \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\" (UID: \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\") "
Jul 28 18:48:37.799569 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:37.799411    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-kserve-provision-location\") pod \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\" (UID: \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\") "
Jul 28 18:48:37.799569 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:37.799448    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-dshm\") pod \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\" (UID: \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\") "
Jul 28 18:48:37.799569 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:37.799477    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-model-cache\") pod \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\" (UID: \"ea22d8d5-744d-4f36-a295-e1b70e3a60c5\") "
Jul 28 18:48:37.799835 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:37.799800    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ea22d8d5-744d-4f36-a295-e1b70e3a60c5" (UID: "ea22d8d5-744d-4f36-a295-e1b70e3a60c5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:48:37.799990 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:37.799808    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-home" (OuterVolumeSpecName: "home") pod "ea22d8d5-744d-4f36-a295-e1b70e3a60c5" (UID: "ea22d8d5-744d-4f36-a295-e1b70e3a60c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:48:37.799990 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:37.799980    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-model-cache" (OuterVolumeSpecName: "model-cache") pod "ea22d8d5-744d-4f36-a295-e1b70e3a60c5" (UID: "ea22d8d5-744d-4f36-a295-e1b70e3a60c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:48:37.802230 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:37.802195    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-dshm" (OuterVolumeSpecName: "dshm") pod "ea22d8d5-744d-4f36-a295-e1b70e3a60c5" (UID: "ea22d8d5-744d-4f36-a295-e1b70e3a60c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:48:37.802230 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:37.802212    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea22d8d5-744d-4f36-a295-e1b70e3a60c5" (UID: "ea22d8d5-744d-4f36-a295-e1b70e3a60c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:48:37.802573 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:37.802544    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-kube-api-access-9qmss" (OuterVolumeSpecName: "kube-api-access-9qmss") pod "ea22d8d5-744d-4f36-a295-e1b70e3a60c5" (UID: "ea22d8d5-744d-4f36-a295-e1b70e3a60c5"). InnerVolumeSpecName "kube-api-access-9qmss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:48:37.811618 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:37.811587    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ea22d8d5-744d-4f36-a295-e1b70e3a60c5" (UID: "ea22d8d5-744d-4f36-a295-e1b70e3a60c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:48:37.866384 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:37.866347    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ea22d8d5-744d-4f36-a295-e1b70e3a60c5" (UID: "ea22d8d5-744d-4f36-a295-e1b70e3a60c5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:48:37.901119 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:37.901093    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-dshm\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:48:37.901119 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:37.901118    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-model-cache\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:48:37.901312 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:37.901128    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:48:37.901312 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:37.901157    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-tmp-dir\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:48:37.901312 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:37.901169    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-home\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:48:37.901312 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:37.901181    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qmss\" (UniqueName: \"kubernetes.io/projected/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-kube-api-access-9qmss\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:48:37.901312 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:37.901190    2543 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-cabundle-cert\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:48:37.901312 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:37.901198    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea22d8d5-744d-4f36-a295-e1b70e3a60c5-kserve-provision-location\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:48:38.308505 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:38.308467    2543 generic.go:356] "Generic (PLEG): container finished" podID="ea22d8d5-744d-4f36-a295-e1b70e3a60c5" containerID="ef79e319bade851c1fdfa8d495fbe483a9ed6e9ca05253033865389e2d040312" exitCode=0
Jul 28 18:48:38.308929 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:38.308537    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"
Jul 28 18:48:38.308929 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:38.308551    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w" event={"ID":"ea22d8d5-744d-4f36-a295-e1b70e3a60c5","Type":"ContainerDied","Data":"ef79e319bade851c1fdfa8d495fbe483a9ed6e9ca05253033865389e2d040312"}
Jul 28 18:48:38.308929 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:38.308620    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w" event={"ID":"ea22d8d5-744d-4f36-a295-e1b70e3a60c5","Type":"ContainerDied","Data":"efbf4aa555ac0a0b2e8de03a856511ed988362fc02fd5011bafaf6f5fb7a970c"}
Jul 28 18:48:38.308929 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:38.308653    2543 scope.go:117] "RemoveContainer" containerID="ef79e319bade851c1fdfa8d495fbe483a9ed6e9ca05253033865389e2d040312"
Jul 28 18:48:38.318192 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:38.318168    2543 scope.go:117] "RemoveContainer" containerID="08c4e98d27c8a275a1c14ef52ef1063d8d77f5939817d5420975de2e48052c83"
Jul 28 18:48:38.326537 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:38.326519    2543 scope.go:117] "RemoveContainer" containerID="ef79e319bade851c1fdfa8d495fbe483a9ed6e9ca05253033865389e2d040312"
Jul 28 18:48:38.326818 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:48:38.326795    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef79e319bade851c1fdfa8d495fbe483a9ed6e9ca05253033865389e2d040312\": container with ID starting with ef79e319bade851c1fdfa8d495fbe483a9ed6e9ca05253033865389e2d040312 not found: ID does not exist" containerID="ef79e319bade851c1fdfa8d495fbe483a9ed6e9ca05253033865389e2d040312"
Jul 28 18:48:38.326872 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:38.326827    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef79e319bade851c1fdfa8d495fbe483a9ed6e9ca05253033865389e2d040312"} err="failed to get container status \"ef79e319bade851c1fdfa8d495fbe483a9ed6e9ca05253033865389e2d040312\": rpc error: code = NotFound desc = could not find container \"ef79e319bade851c1fdfa8d495fbe483a9ed6e9ca05253033865389e2d040312\": container with ID starting with ef79e319bade851c1fdfa8d495fbe483a9ed6e9ca05253033865389e2d040312 not found: ID does not exist"
Jul 28 18:48:38.326872 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:38.326844    2543 scope.go:117] "RemoveContainer" containerID="08c4e98d27c8a275a1c14ef52ef1063d8d77f5939817d5420975de2e48052c83"
Jul 28 18:48:38.327049 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:48:38.327032    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08c4e98d27c8a275a1c14ef52ef1063d8d77f5939817d5420975de2e48052c83\": container with ID starting with 08c4e98d27c8a275a1c14ef52ef1063d8d77f5939817d5420975de2e48052c83 not found: ID does not exist" containerID="08c4e98d27c8a275a1c14ef52ef1063d8d77f5939817d5420975de2e48052c83"
Jul 28 18:48:38.327108 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:38.327058    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08c4e98d27c8a275a1c14ef52ef1063d8d77f5939817d5420975de2e48052c83"} err="failed to get container status \"08c4e98d27c8a275a1c14ef52ef1063d8d77f5939817d5420975de2e48052c83\": rpc error: code = NotFound desc = could not find container \"08c4e98d27c8a275a1c14ef52ef1063d8d77f5939817d5420975de2e48052c83\": container with ID starting with 08c4e98d27c8a275a1c14ef52ef1063d8d77f5939817d5420975de2e48052c83 not found: ID does not exist"
Jul 28 18:48:38.332292 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:38.332268    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"]
Jul 28 18:48:38.335987 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:38.335965    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w"]
Jul 28 18:48:38.657310 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:38.657192    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6d8b575478-k7q5w" podUID="ea22d8d5-744d-4f36-a295-e1b70e3a60c5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": context deadline exceeded"
Jul 28 18:48:39.152597 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:48:39.152565    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea22d8d5-744d-4f36-a295-e1b70e3a60c5" path="/var/lib/kubelet/pods/ea22d8d5-744d-4f36-a295-e1b70e3a60c5/volumes"
Jul 28 18:49:42.689321 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:42.689287    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 28 18:49:42.697979 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:49:42.697943    2543 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 18:49:42.697979 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:49:42.697979 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 18:49:42.697979 ip-10-0-135-177 kubenswrapper[2543]:  > containerID="ca1cb8cf17fb5f57d80be7ab13d2306fdac2645634a7512e695210d389b50dc4" cmd=["/bin/sleep","15"]
Jul 28 18:49:42.698247 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:42.697995    2543 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" message="" err=<
Jul 28 18:49:42.698247 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:49:42.698247 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 18:49:42.698247 ip-10-0-135-177 kubenswrapper[2543]:  >
Jul 28 18:49:42.698247 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:49:42.698007    2543 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 18:49:42.698247 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:49:42.698247 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 18:49:42.698247 ip-10-0-135-177 kubenswrapper[2543]:  > pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" podUID="5d8a975f-9633-46bf-b87f-e97956c18a68" containerName="main" containerID="cri-o://ca1cb8cf17fb5f57d80be7ab13d2306fdac2645634a7512e695210d389b50dc4"
Jul 28 18:49:42.698247 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:42.698052    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" podUID="5d8a975f-9633-46bf-b87f-e97956c18a68" containerName="main" containerID="cri-o://ca1cb8cf17fb5f57d80be7ab13d2306fdac2645634a7512e695210d389b50dc4" gracePeriod=60
Jul 28 18:49:42.951954 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:42.951897    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:49:43.083781 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:43.083731    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d8a975f-9633-46bf-b87f-e97956c18a68-tls-certs\") pod \"5d8a975f-9633-46bf-b87f-e97956c18a68\" (UID: \"5d8a975f-9633-46bf-b87f-e97956c18a68\") "
Jul 28 18:49:43.083781 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:43.083790    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d8a975f-9633-46bf-b87f-e97956c18a68-home\") pod \"5d8a975f-9633-46bf-b87f-e97956c18a68\" (UID: \"5d8a975f-9633-46bf-b87f-e97956c18a68\") "
Jul 28 18:49:43.084052 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:43.083834    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d8a975f-9633-46bf-b87f-e97956c18a68-dshm\") pod \"5d8a975f-9633-46bf-b87f-e97956c18a68\" (UID: \"5d8a975f-9633-46bf-b87f-e97956c18a68\") "
Jul 28 18:49:43.084052 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:43.083931    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d8a975f-9633-46bf-b87f-e97956c18a68-tmp-dir\") pod \"5d8a975f-9633-46bf-b87f-e97956c18a68\" (UID: \"5d8a975f-9633-46bf-b87f-e97956c18a68\") "
Jul 28 18:49:43.084052 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:43.083966    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d8a975f-9633-46bf-b87f-e97956c18a68-model-cache\") pod \"5d8a975f-9633-46bf-b87f-e97956c18a68\" (UID: \"5d8a975f-9633-46bf-b87f-e97956c18a68\") "
Jul 28 18:49:43.084052 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:43.084015    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5v5nr\" (UniqueName: \"kubernetes.io/projected/5d8a975f-9633-46bf-b87f-e97956c18a68-kube-api-access-5v5nr\") pod \"5d8a975f-9633-46bf-b87f-e97956c18a68\" (UID: \"5d8a975f-9633-46bf-b87f-e97956c18a68\") "
Jul 28 18:49:43.084296 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:43.084058    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8a975f-9633-46bf-b87f-e97956c18a68-home" (OuterVolumeSpecName: "home") pod "5d8a975f-9633-46bf-b87f-e97956c18a68" (UID: "5d8a975f-9633-46bf-b87f-e97956c18a68"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:49:43.084296 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:43.084189    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8a975f-9633-46bf-b87f-e97956c18a68-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5d8a975f-9633-46bf-b87f-e97956c18a68" (UID: "5d8a975f-9633-46bf-b87f-e97956c18a68"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:49:43.084373 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:43.084295    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8a975f-9633-46bf-b87f-e97956c18a68-model-cache" (OuterVolumeSpecName: "model-cache") pod "5d8a975f-9633-46bf-b87f-e97956c18a68" (UID: "5d8a975f-9633-46bf-b87f-e97956c18a68"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:49:43.084421 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:43.084377    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d8a975f-9633-46bf-b87f-e97956c18a68-home\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:49:43.084421 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:43.084396    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d8a975f-9633-46bf-b87f-e97956c18a68-tmp-dir\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:49:43.084421 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:43.084408    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d8a975f-9633-46bf-b87f-e97956c18a68-model-cache\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:49:43.086255 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:43.086230    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d8a975f-9633-46bf-b87f-e97956c18a68-kube-api-access-5v5nr" (OuterVolumeSpecName: "kube-api-access-5v5nr") pod "5d8a975f-9633-46bf-b87f-e97956c18a68" (UID: "5d8a975f-9633-46bf-b87f-e97956c18a68"). InnerVolumeSpecName "kube-api-access-5v5nr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:49:43.086418 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:43.086397    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8a975f-9633-46bf-b87f-e97956c18a68-dshm" (OuterVolumeSpecName: "dshm") pod "5d8a975f-9633-46bf-b87f-e97956c18a68" (UID: "5d8a975f-9633-46bf-b87f-e97956c18a68"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:49:43.086520 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:43.086500    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d8a975f-9633-46bf-b87f-e97956c18a68-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5d8a975f-9633-46bf-b87f-e97956c18a68" (UID: "5d8a975f-9633-46bf-b87f-e97956c18a68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:49:43.185768 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:43.185737    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d8a975f-9633-46bf-b87f-e97956c18a68-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:49:43.185768 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:43.185761    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d8a975f-9633-46bf-b87f-e97956c18a68-dshm\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:49:43.185768 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:43.185770    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v5nr\" (UniqueName: \"kubernetes.io/projected/5d8a975f-9633-46bf-b87f-e97956c18a68-kube-api-access-5v5nr\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:49:43.641938 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:43.641896    2543 generic.go:356] "Generic (PLEG): container finished" podID="5d8a975f-9633-46bf-b87f-e97956c18a68" containerID="ca1cb8cf17fb5f57d80be7ab13d2306fdac2645634a7512e695210d389b50dc4" exitCode=0
Jul 28 18:49:43.641938 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:43.641938    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 28 18:49:43.642241 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:43.641962    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" event={"ID":"5d8a975f-9633-46bf-b87f-e97956c18a68","Type":"ContainerDied","Data":"ca1cb8cf17fb5f57d80be7ab13d2306fdac2645634a7512e695210d389b50dc4"}
Jul 28 18:49:43.642241 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:43.642023    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" event={"ID":"5d8a975f-9633-46bf-b87f-e97956c18a68","Type":"ContainerDied","Data":"dfa493c2d7bd4da7521fb0635847410d5a6dc96661c0ba79c672087afd184d0e"}
Jul 28 18:49:43.642241 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:43.642046    2543 scope.go:117] "RemoveContainer" containerID="ca1cb8cf17fb5f57d80be7ab13d2306fdac2645634a7512e695210d389b50dc4"
Jul 28 18:49:43.651173 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:43.651155    2543 scope.go:117] "RemoveContainer" containerID="ca1cb8cf17fb5f57d80be7ab13d2306fdac2645634a7512e695210d389b50dc4"
Jul 28 18:49:43.651421 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:49:43.651403    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca1cb8cf17fb5f57d80be7ab13d2306fdac2645634a7512e695210d389b50dc4\": container with ID starting with ca1cb8cf17fb5f57d80be7ab13d2306fdac2645634a7512e695210d389b50dc4 not found: ID does not exist" containerID="ca1cb8cf17fb5f57d80be7ab13d2306fdac2645634a7512e695210d389b50dc4"
Jul 28 18:49:43.651502 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:43.651428    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca1cb8cf17fb5f57d80be7ab13d2306fdac2645634a7512e695210d389b50dc4"} err="failed to get container status \"ca1cb8cf17fb5f57d80be7ab13d2306fdac2645634a7512e695210d389b50dc4\": rpc error: code = NotFound desc = could not find container \"ca1cb8cf17fb5f57d80be7ab13d2306fdac2645634a7512e695210d389b50dc4\": container with ID starting with ca1cb8cf17fb5f57d80be7ab13d2306fdac2645634a7512e695210d389b50dc4 not found: ID does not exist"
Jul 28 18:49:43.663682 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:43.662895    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 28 18:49:43.671899 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:43.671875    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 28 18:49:45.152576 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:45.152542    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d8a975f-9633-46bf-b87f-e97956c18a68" path="/var/lib/kubelet/pods/5d8a975f-9633-46bf-b87f-e97956c18a68/volumes"
Jul 28 18:49:48.012202 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:49:48.012158    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8" podUID="1c409c83-3d4a-40a6-9d31-4982e37030c4"
Jul 28 18:49:48.668407 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:49:48.668368    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:50:48.310600 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.310567    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx"]
Jul 28 18:50:48.311012 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.310939    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d8a975f-9633-46bf-b87f-e97956c18a68" containerName="main"
Jul 28 18:50:48.311012 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.310951    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d8a975f-9633-46bf-b87f-e97956c18a68" containerName="main"
Jul 28 18:50:48.311012 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.310967    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea22d8d5-744d-4f36-a295-e1b70e3a60c5" containerName="main"
Jul 28 18:50:48.311012 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.310972    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea22d8d5-744d-4f36-a295-e1b70e3a60c5" containerName="main"
Jul 28 18:50:48.311012 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.310981    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea22d8d5-744d-4f36-a295-e1b70e3a60c5" containerName="storage-initializer"
Jul 28 18:50:48.311012 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.310991    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea22d8d5-744d-4f36-a295-e1b70e3a60c5" containerName="storage-initializer"
Jul 28 18:50:48.311228 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.311052    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d8a975f-9633-46bf-b87f-e97956c18a68" containerName="main"
Jul 28 18:50:48.311228 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.311062    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea22d8d5-744d-4f36-a295-e1b70e3a60c5" containerName="main"
Jul 28 18:50:48.314014 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.313998    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx"
Jul 28 18:50:48.317318 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.317293    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40\"/\"autoscale-cleanup-hpa-kserve-self-signed-certs\""
Jul 28 18:50:48.317318 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.317310    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40\"/\"openshift-service-ca.crt\""
Jul 28 18:50:48.317543 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.317294    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40\"/\"kube-root-ca.crt\""
Jul 28 18:50:48.317543 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.317355    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40\"/\"default-dockercfg-frqjv\""
Jul 28 18:50:48.323774 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.323750    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx"]
Jul 28 18:50:48.443384 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.443346    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49991427-8664-4d2a-ac37-31467363ecde-dshm\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx\" (UID: \"49991427-8664-4d2a-ac37-31467363ecde\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx"
Jul 28 18:50:48.443563 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.443388    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrqnb\" (UniqueName: \"kubernetes.io/projected/49991427-8664-4d2a-ac37-31467363ecde-kube-api-access-qrqnb\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx\" (UID: \"49991427-8664-4d2a-ac37-31467363ecde\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx"
Jul 28 18:50:48.443563 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.443458    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49991427-8664-4d2a-ac37-31467363ecde-tls-certs\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx\" (UID: \"49991427-8664-4d2a-ac37-31467363ecde\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx"
Jul 28 18:50:48.443563 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.443498    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49991427-8664-4d2a-ac37-31467363ecde-model-cache\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx\" (UID: \"49991427-8664-4d2a-ac37-31467363ecde\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx"
Jul 28 18:50:48.443563 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.443536    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49991427-8664-4d2a-ac37-31467363ecde-home\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx\" (UID: \"49991427-8664-4d2a-ac37-31467363ecde\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx"
Jul 28 18:50:48.443709 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.443617    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49991427-8664-4d2a-ac37-31467363ecde-tmp-dir\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx\" (UID: \"49991427-8664-4d2a-ac37-31467363ecde\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx"
Jul 28 18:50:48.544603 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.544566    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49991427-8664-4d2a-ac37-31467363ecde-home\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx\" (UID: \"49991427-8664-4d2a-ac37-31467363ecde\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx"
Jul 28 18:50:48.544811 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.544632    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49991427-8664-4d2a-ac37-31467363ecde-tmp-dir\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx\" (UID: \"49991427-8664-4d2a-ac37-31467363ecde\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx"
Jul 28 18:50:48.544811 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.544708    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49991427-8664-4d2a-ac37-31467363ecde-dshm\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx\" (UID: \"49991427-8664-4d2a-ac37-31467363ecde\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx"
Jul 28 18:50:48.544811 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.544741    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrqnb\" (UniqueName: \"kubernetes.io/projected/49991427-8664-4d2a-ac37-31467363ecde-kube-api-access-qrqnb\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx\" (UID: \"49991427-8664-4d2a-ac37-31467363ecde\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx"
Jul 28 18:50:48.544811 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.544778    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49991427-8664-4d2a-ac37-31467363ecde-tls-certs\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx\" (UID: \"49991427-8664-4d2a-ac37-31467363ecde\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx"
Jul 28 18:50:48.544811 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.544810    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49991427-8664-4d2a-ac37-31467363ecde-model-cache\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx\" (UID: \"49991427-8664-4d2a-ac37-31467363ecde\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx"
Jul 28 18:50:48.545127 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.545023    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49991427-8664-4d2a-ac37-31467363ecde-home\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx\" (UID: \"49991427-8664-4d2a-ac37-31467363ecde\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx"
Jul 28 18:50:48.545127 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.545037    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49991427-8664-4d2a-ac37-31467363ecde-tmp-dir\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx\" (UID: \"49991427-8664-4d2a-ac37-31467363ecde\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx"
Jul 28 18:50:48.545292 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.545151    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49991427-8664-4d2a-ac37-31467363ecde-model-cache\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx\" (UID: \"49991427-8664-4d2a-ac37-31467363ecde\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx"
Jul 28 18:50:48.547388 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.547359    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49991427-8664-4d2a-ac37-31467363ecde-dshm\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx\" (UID: \"49991427-8664-4d2a-ac37-31467363ecde\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx"
Jul 28 18:50:48.547736 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.547719    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49991427-8664-4d2a-ac37-31467363ecde-tls-certs\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx\" (UID: \"49991427-8664-4d2a-ac37-31467363ecde\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx"
Jul 28 18:50:48.553588 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.553565    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrqnb\" (UniqueName: \"kubernetes.io/projected/49991427-8664-4d2a-ac37-31467363ecde-kube-api-access-qrqnb\") pod \"autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx\" (UID: \"49991427-8664-4d2a-ac37-31467363ecde\") " pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx"
Jul 28 18:50:48.624432 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.624346    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx"
Jul 28 18:50:48.759321 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.759289    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx"]
Jul 28 18:50:48.762369 ip-10-0-135-177 kubenswrapper[2543]: W0728 18:50:48.762343    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49991427_8664_4d2a_ac37_31467363ecde.slice/crio-085f4c0b72efe3fbbfdbff4b046790d33b2ddfe9ae7c84374df6c65b7d0b8e96 WatchSource:0}: Error finding container 085f4c0b72efe3fbbfdbff4b046790d33b2ddfe9ae7c84374df6c65b7d0b8e96: Status 404 returned error can't find the container with id 085f4c0b72efe3fbbfdbff4b046790d33b2ddfe9ae7c84374df6c65b7d0b8e96
Jul 28 18:50:48.958592 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.958496    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx" event={"ID":"49991427-8664-4d2a-ac37-31467363ecde","Type":"ContainerStarted","Data":"8311bd7e9dec2959eb3a7650a558c899606c3d192104be257dddb8d321451396"}
Jul 28 18:50:48.958592 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.958565    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx" event={"ID":"49991427-8664-4d2a-ac37-31467363ecde","Type":"ContainerStarted","Data":"085f4c0b72efe3fbbfdbff4b046790d33b2ddfe9ae7c84374df6c65b7d0b8e96"}
Jul 28 18:50:48.977461 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:48.977409    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx" podStartSLOduration=0.977394962 podStartE2EDuration="977.394962ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:50:48.976179006 +0000 UTC m=+5252.382452845" watchObservedRunningTime="2026-07-28 18:50:48.977394962 +0000 UTC m=+5252.383668702"
Jul 28 18:50:49.624791 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:49.624754    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx"
Jul 28 18:50:58.624962 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:58.624921    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx"
Jul 28 18:50:58.637269 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:58.637238    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx"
Jul 28 18:50:59.017977 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:50:59.017947    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx"
Jul 28 18:51:51.669417 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:51:51.669332    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8" podUID="1c409c83-3d4a-40a6-9d31-4982e37030c4"
Jul 28 18:51:52.263686 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:51:52.263657    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:53:55.264699 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:53:55.264651    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8" podUID="1c409c83-3d4a-40a6-9d31-4982e37030c4"
Jul 28 18:53:55.884914 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:53:55.884885    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:55:06.429297 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:06.429217    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"]
Jul 28 18:55:06.432759 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:06.432742    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"
Jul 28 18:55:06.435548 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:06.435528    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Jul 28 18:55:06.435649 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:06.435609    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Jul 28 18:55:06.435825 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:06.435809    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-4nwj4\""
Jul 28 18:55:06.435877 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:06.435814    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Jul 28 18:55:06.436343 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:06.436330    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 28 18:55:06.453788 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:06.453762    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"]
Jul 28 18:55:06.562077 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:06.562040    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-model-cache\") pod \"prestop-hook-test-kserve-68f6bd7b79-j2mdw\" (UID: \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"
Jul 28 18:55:06.562267 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:06.562105    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-tmp-dir\") pod \"prestop-hook-test-kserve-68f6bd7b79-j2mdw\" (UID: \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"
Jul 28 18:55:06.562267 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:06.562223    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-home\") pod \"prestop-hook-test-kserve-68f6bd7b79-j2mdw\" (UID: \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"
Jul 28 18:55:06.562267 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:06.562251    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-kserve-provision-location\") pod \"prestop-hook-test-kserve-68f6bd7b79-j2mdw\" (UID: \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"
Jul 28 18:55:06.562379 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:06.562283    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-dshm\") pod \"prestop-hook-test-kserve-68f6bd7b79-j2mdw\" (UID: \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"
Jul 28 18:55:06.562379 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:06.562345    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvqwr\" (UniqueName: \"kubernetes.io/projected/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-kube-api-access-rvqwr\") pod \"prestop-hook-test-kserve-68f6bd7b79-j2mdw\" (UID: \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"
Jul 28 18:55:06.562447 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:06.562375    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-tls-certs\") pod \"prestop-hook-test-kserve-68f6bd7b79-j2mdw\" (UID: \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"
Jul 28 18:55:06.663080 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:06.663039    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-tmp-dir\") pod \"prestop-hook-test-kserve-68f6bd7b79-j2mdw\" (UID: \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"
Jul 28 18:55:06.663277 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:06.663104    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-home\") pod \"prestop-hook-test-kserve-68f6bd7b79-j2mdw\" (UID: \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"
Jul 28 18:55:06.663277 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:06.663152    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-kserve-provision-location\") pod \"prestop-hook-test-kserve-68f6bd7b79-j2mdw\" (UID: \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"
Jul 28 18:55:06.663277 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:06.663182    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-dshm\") pod \"prestop-hook-test-kserve-68f6bd7b79-j2mdw\" (UID: \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"
Jul 28 18:55:06.663277 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:06.663216    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvqwr\" (UniqueName: \"kubernetes.io/projected/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-kube-api-access-rvqwr\") pod \"prestop-hook-test-kserve-68f6bd7b79-j2mdw\" (UID: \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"
Jul 28 18:55:06.663277 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:06.663242    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-tls-certs\") pod \"prestop-hook-test-kserve-68f6bd7b79-j2mdw\" (UID: \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"
Jul 28 18:55:06.663532 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:06.663287    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-model-cache\") pod \"prestop-hook-test-kserve-68f6bd7b79-j2mdw\" (UID: \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"
Jul 28 18:55:06.663592 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:06.663551    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-tmp-dir\") pod \"prestop-hook-test-kserve-68f6bd7b79-j2mdw\" (UID: \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"
Jul 28 18:55:06.663650 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:06.663615    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-kserve-provision-location\") pod \"prestop-hook-test-kserve-68f6bd7b79-j2mdw\" (UID: \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"
Jul 28 18:55:06.663713 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:06.663653    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-home\") pod \"prestop-hook-test-kserve-68f6bd7b79-j2mdw\" (UID: \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"
Jul 28 18:55:06.663713 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:06.663687    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-model-cache\") pod \"prestop-hook-test-kserve-68f6bd7b79-j2mdw\" (UID: \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"
Jul 28 18:55:06.665367 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:06.665341    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-dshm\") pod \"prestop-hook-test-kserve-68f6bd7b79-j2mdw\" (UID: \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"
Jul 28 18:55:06.665684 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:06.665666    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-tls-certs\") pod \"prestop-hook-test-kserve-68f6bd7b79-j2mdw\" (UID: \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"
Jul 28 18:55:06.672979 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:06.672952    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvqwr\" (UniqueName: \"kubernetes.io/projected/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-kube-api-access-rvqwr\") pod \"prestop-hook-test-kserve-68f6bd7b79-j2mdw\" (UID: \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"
Jul 28 18:55:06.743049 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:06.743015    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"
Jul 28 18:55:06.893634 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:06.893610    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"]
Jul 28 18:55:06.895518 ip-10-0-135-177 kubenswrapper[2543]: W0728 18:55:06.895489    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod401fbd3b_2c44_4d82_8df5_5f38fb64fedc.slice/crio-92e604131669a4318b5377599f9a078ccc082c20ba583faac70435bb6a747c93 WatchSource:0}: Error finding container 92e604131669a4318b5377599f9a078ccc082c20ba583faac70435bb6a747c93: Status 404 returned error can't find the container with id 92e604131669a4318b5377599f9a078ccc082c20ba583faac70435bb6a747c93
Jul 28 18:55:07.250488 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:07.250452    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw" event={"ID":"401fbd3b-2c44-4d82-8df5-5f38fb64fedc","Type":"ContainerStarted","Data":"78f422d8a4b03f193339abba994b2569e128887c00d335271127a6aff244c897"}
Jul 28 18:55:07.250683 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:07.250508    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw" event={"ID":"401fbd3b-2c44-4d82-8df5-5f38fb64fedc","Type":"ContainerStarted","Data":"92e604131669a4318b5377599f9a078ccc082c20ba583faac70435bb6a747c93"}
Jul 28 18:55:15.294963 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:15.294923    2543 generic.go:356] "Generic (PLEG): container finished" podID="401fbd3b-2c44-4d82-8df5-5f38fb64fedc" containerID="78f422d8a4b03f193339abba994b2569e128887c00d335271127a6aff244c897" exitCode=0
Jul 28 18:55:15.295453 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:15.294989    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw" event={"ID":"401fbd3b-2c44-4d82-8df5-5f38fb64fedc","Type":"ContainerDied","Data":"78f422d8a4b03f193339abba994b2569e128887c00d335271127a6aff244c897"}
Jul 28 18:55:16.305038 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:16.305002    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw" event={"ID":"401fbd3b-2c44-4d82-8df5-5f38fb64fedc","Type":"ContainerStarted","Data":"2d337a8be52beace1aed3780b8beb6c1649f43121f9dcd712ef0b075ec940335"}
Jul 28 18:55:16.334074 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:16.334024    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw" podStartSLOduration=2.767846959 podStartE2EDuration="10.334008843s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.566161884s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:55:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:55:16.330927436 +0000 UTC m=+5519.737201176" watchObservedRunningTime="2026-07-28 18:55:16.334008843 +0000 UTC m=+5519.740282584"
Jul 28 18:55:16.743734 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:16.743702    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"
Jul 28 18:55:16.743905 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:16.743768    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"
Jul 28 18:55:16.745217 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:16.745185    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw" podUID="401fbd3b-2c44-4d82-8df5-5f38fb64fedc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Jul 28 18:55:26.744470 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:26.744424    2543 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw" podUID="401fbd3b-2c44-4d82-8df5-5f38fb64fedc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": dial tcp 10.134.0.79:8000: connect: connection refused"
Jul 28 18:55:28.043067 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:28.043032    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"]
Jul 28 18:55:43.067039 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:43.066974    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw" podUID="401fbd3b-2c44-4d82-8df5-5f38fb64fedc" containerName="main" containerID="cri-o://2d337a8be52beace1aed3780b8beb6c1649f43121f9dcd712ef0b075ec940335" gracePeriod=45
Jul 28 18:55:58.886431 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:55:58.886387    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8" podUID="1c409c83-3d4a-40a6-9d31-4982e37030c4"
Jul 28 18:55:59.538963 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:55:59.538932    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:56:28.696168 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:28.696125    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-68f6bd7b79-j2mdw_401fbd3b-2c44-4d82-8df5-5f38fb64fedc/main/0.log"
Jul 28 18:56:28.696599 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:28.696545    2543 generic.go:356] "Generic (PLEG): container finished" podID="401fbd3b-2c44-4d82-8df5-5f38fb64fedc" containerID="2d337a8be52beace1aed3780b8beb6c1649f43121f9dcd712ef0b075ec940335" exitCode=137
Jul 28 18:56:28.696599 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:28.696579    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw" event={"ID":"401fbd3b-2c44-4d82-8df5-5f38fb64fedc","Type":"ContainerDied","Data":"2d337a8be52beace1aed3780b8beb6c1649f43121f9dcd712ef0b075ec940335"}
Jul 28 18:56:28.721246 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:28.721222    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-68f6bd7b79-j2mdw_401fbd3b-2c44-4d82-8df5-5f38fb64fedc/main/0.log"
Jul 28 18:56:28.723772 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:28.723751    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"
Jul 28 18:56:28.849498 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:28.849399    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-model-cache\") pod \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\" (UID: \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\") "
Jul 28 18:56:28.849498 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:28.849448    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-tmp-dir\") pod \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\" (UID: \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\") "
Jul 28 18:56:28.849498 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:28.849480    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvqwr\" (UniqueName: \"kubernetes.io/projected/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-kube-api-access-rvqwr\") pod \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\" (UID: \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\") "
Jul 28 18:56:28.849790 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:28.849561    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-dshm\") pod \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\" (UID: \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\") "
Jul 28 18:56:28.849790 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:28.849593    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-home\") pod \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\" (UID: \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\") "
Jul 28 18:56:28.849790 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:28.849643    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-kserve-provision-location\") pod \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\" (UID: \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\") "
Jul 28 18:56:28.849790 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:28.849671    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-tls-certs\") pod \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\" (UID: \"401fbd3b-2c44-4d82-8df5-5f38fb64fedc\") "
Jul 28 18:56:28.849790 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:28.849729    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-model-cache" (OuterVolumeSpecName: "model-cache") pod "401fbd3b-2c44-4d82-8df5-5f38fb64fedc" (UID: "401fbd3b-2c44-4d82-8df5-5f38fb64fedc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:56:28.850071 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:28.850024    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-model-cache\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:56:28.850071 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:28.850030    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "401fbd3b-2c44-4d82-8df5-5f38fb64fedc" (UID: "401fbd3b-2c44-4d82-8df5-5f38fb64fedc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:56:28.850278 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:28.850256    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-home" (OuterVolumeSpecName: "home") pod "401fbd3b-2c44-4d82-8df5-5f38fb64fedc" (UID: "401fbd3b-2c44-4d82-8df5-5f38fb64fedc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:56:28.851938 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:28.851911    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-dshm" (OuterVolumeSpecName: "dshm") pod "401fbd3b-2c44-4d82-8df5-5f38fb64fedc" (UID: "401fbd3b-2c44-4d82-8df5-5f38fb64fedc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:56:28.852047 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:28.851915    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "401fbd3b-2c44-4d82-8df5-5f38fb64fedc" (UID: "401fbd3b-2c44-4d82-8df5-5f38fb64fedc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:56:28.852047 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:28.851951    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-kube-api-access-rvqwr" (OuterVolumeSpecName: "kube-api-access-rvqwr") pod "401fbd3b-2c44-4d82-8df5-5f38fb64fedc" (UID: "401fbd3b-2c44-4d82-8df5-5f38fb64fedc"). InnerVolumeSpecName "kube-api-access-rvqwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:56:28.916301 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:28.916254    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "401fbd3b-2c44-4d82-8df5-5f38fb64fedc" (UID: "401fbd3b-2c44-4d82-8df5-5f38fb64fedc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:56:28.951100 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:28.951073    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-dshm\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:56:28.951100 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:28.951096    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-home\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:56:28.951285 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:28.951105    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-kserve-provision-location\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:56:28.951285 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:28.951117    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:56:28.951285 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:28.951126    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-tmp-dir\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:56:28.951285 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:28.951149    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvqwr\" (UniqueName: \"kubernetes.io/projected/401fbd3b-2c44-4d82-8df5-5f38fb64fedc-kube-api-access-rvqwr\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:56:29.702584 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:29.702555    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-68f6bd7b79-j2mdw_401fbd3b-2c44-4d82-8df5-5f38fb64fedc/main/0.log"
Jul 28 18:56:29.703009 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:29.702946    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"
Jul 28 18:56:29.703009 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:29.702976    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw" event={"ID":"401fbd3b-2c44-4d82-8df5-5f38fb64fedc","Type":"ContainerDied","Data":"92e604131669a4318b5377599f9a078ccc082c20ba583faac70435bb6a747c93"}
Jul 28 18:56:29.703088 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:29.703039    2543 scope.go:117] "RemoveContainer" containerID="2d337a8be52beace1aed3780b8beb6c1649f43121f9dcd712ef0b075ec940335"
Jul 28 18:56:29.712113 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:29.712083    2543 scope.go:117] "RemoveContainer" containerID="78f422d8a4b03f193339abba994b2569e128887c00d335271127a6aff244c897"
Jul 28 18:56:29.729879 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:29.729854    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"]
Jul 28 18:56:29.736984 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:29.736962    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-j2mdw"]
Jul 28 18:56:31.152699 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:56:31.152664    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="401fbd3b-2c44-4d82-8df5-5f38fb64fedc" path="/var/lib/kubelet/pods/401fbd3b-2c44-4d82-8df5-5f38fb64fedc/volumes"
Jul 28 18:57:38.619453 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:38.619417    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf"]
Jul 28 18:57:38.619900 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:38.619784    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="401fbd3b-2c44-4d82-8df5-5f38fb64fedc" containerName="main"
Jul 28 18:57:38.619900 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:38.619797    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="401fbd3b-2c44-4d82-8df5-5f38fb64fedc" containerName="main"
Jul 28 18:57:38.619900 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:38.619811    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="401fbd3b-2c44-4d82-8df5-5f38fb64fedc" containerName="storage-initializer"
Jul 28 18:57:38.619900 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:38.619817    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="401fbd3b-2c44-4d82-8df5-5f38fb64fedc" containerName="storage-initializer"
Jul 28 18:57:38.619900 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:38.619885    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="401fbd3b-2c44-4d82-8df5-5f38fb64fedc" containerName="main"
Jul 28 18:57:38.623999 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:38.623979    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf"
Jul 28 18:57:38.628670 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:38.628644    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Jul 28 18:57:38.629514 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:38.629480    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-hsm69\""
Jul 28 18:57:38.629841 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:38.629549    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Jul 28 18:57:38.629955 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:38.629554    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 28 18:57:38.640253 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:38.640226    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf"]
Jul 28 18:57:38.653337 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:38.653312    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zqcmf\" (UID: \"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf"
Jul 28 18:57:38.653496 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:38.653356    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zqcmf\" (UID: \"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf"
Jul 28 18:57:38.653496 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:38.653383    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zqcmf\" (UID: \"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf"
Jul 28 18:57:38.653589 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:38.653494    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zqcmf\" (UID: \"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf"
Jul 28 18:57:38.653589 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:38.653540    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hppdt\" (UniqueName: \"kubernetes.io/projected/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-kube-api-access-hppdt\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zqcmf\" (UID: \"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf"
Jul 28 18:57:38.653589 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:38.653574    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zqcmf\" (UID: \"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf"
Jul 28 18:57:38.754350 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:38.754281    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zqcmf\" (UID: \"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf"
Jul 28 18:57:38.754350 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:38.754326    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zqcmf\" (UID: \"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf"
Jul 28 18:57:38.754589 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:38.754356    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zqcmf\" (UID: \"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf"
Jul 28 18:57:38.754589 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:38.754382    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hppdt\" (UniqueName: \"kubernetes.io/projected/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-kube-api-access-hppdt\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zqcmf\" (UID: \"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf"
Jul 28 18:57:38.754589 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:38.754402    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zqcmf\" (UID: \"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf"
Jul 28 18:57:38.754752 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:38.754612    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zqcmf\" (UID: \"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf"
Jul 28 18:57:38.754752 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:38.754715    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zqcmf\" (UID: \"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf"
Jul 28 18:57:38.754867 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:38.754767    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zqcmf\" (UID: \"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf"
Jul 28 18:57:38.754946 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:38.754926    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zqcmf\" (UID: \"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf"
Jul 28 18:57:38.756621 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:38.756604    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zqcmf\" (UID: \"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf"
Jul 28 18:57:38.757234 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:38.757211    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zqcmf\" (UID: \"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf"
Jul 28 18:57:38.768889 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:38.768865    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hppdt\" (UniqueName: \"kubernetes.io/projected/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-kube-api-access-hppdt\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-zqcmf\" (UID: \"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf"
Jul 28 18:57:38.937023 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:38.936946    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf"
Jul 28 18:57:39.073291 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:39.073175    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf"]
Jul 28 18:57:39.075541 ip-10-0-135-177 kubenswrapper[2543]: W0728 18:57:39.075506    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1156c8f2_7183_4b71_9cc5_1ec2ebc04f06.slice/crio-26438fb8ef5b15be235283e68758a7560f1ee5a1519cefdbea010e968c4e37f8 WatchSource:0}: Error finding container 26438fb8ef5b15be235283e68758a7560f1ee5a1519cefdbea010e968c4e37f8: Status 404 returned error can't find the container with id 26438fb8ef5b15be235283e68758a7560f1ee5a1519cefdbea010e968c4e37f8
Jul 28 18:57:40.059603 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:40.059557    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf" event={"ID":"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06","Type":"ContainerStarted","Data":"ac7690a6c58c38d57958855a453b1878300d6931beba9d4348595600e3de7de9"}
Jul 28 18:57:40.060030 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:40.059641    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf" event={"ID":"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06","Type":"ContainerStarted","Data":"26438fb8ef5b15be235283e68758a7560f1ee5a1519cefdbea010e968c4e37f8"}
Jul 28 18:57:40.082205 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:40.082159    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf" podStartSLOduration=2.082145154 podStartE2EDuration="2.082145154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:57:40.078774177 +0000 UTC m=+5663.485047918" watchObservedRunningTime="2026-07-28 18:57:40.082145154 +0000 UTC m=+5663.488418895"
Jul 28 18:57:40.937334 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:40.937297    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf"
Jul 28 18:57:48.937593 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:48.937552    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf"
Jul 28 18:57:48.954327 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:48.954301    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf"
Jul 28 18:57:49.114374 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:49.114345    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf"
Jul 28 18:57:59.387576 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:59.387541    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf"]
Jul 28 18:57:59.396634 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:57:59.396594    2543 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 18:57:59.396634 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:57:59.396634 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 18:57:59.396634 ip-10-0-135-177 kubenswrapper[2543]:  > containerID="ac7690a6c58c38d57958855a453b1878300d6931beba9d4348595600e3de7de9" cmd=["/bin/sleep","15"]
Jul 28 18:57:59.396897 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:59.396649    2543 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf" message="" err=<
Jul 28 18:57:59.396897 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:57:59.396897 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 18:57:59.396897 ip-10-0-135-177 kubenswrapper[2543]:  >
Jul 28 18:57:59.396897 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:57:59.396666    2543 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 18:57:59.396897 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:57:59.396897 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 18:57:59.396897 ip-10-0-135-177 kubenswrapper[2543]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf" podUID="1156c8f2-7183-4b71-9cc5-1ec2ebc04f06" containerName="main" containerID="cri-o://ac7690a6c58c38d57958855a453b1878300d6931beba9d4348595600e3de7de9"
Jul 28 18:57:59.396897 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:59.396711    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf" podUID="1156c8f2-7183-4b71-9cc5-1ec2ebc04f06" containerName="main" containerID="cri-o://ac7690a6c58c38d57958855a453b1878300d6931beba9d4348595600e3de7de9" gracePeriod=60
Jul 28 18:57:59.751101 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:59.751076    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf"
Jul 28 18:57:59.838308 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:59.838274    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-dshm\") pod \"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06\" (UID: \"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06\") "
Jul 28 18:57:59.838498 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:59.838343    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-tls-certs\") pod \"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06\" (UID: \"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06\") "
Jul 28 18:57:59.838498 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:59.838385    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-tmp-dir\") pod \"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06\" (UID: \"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06\") "
Jul 28 18:57:59.838498 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:59.838410    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-model-cache\") pod \"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06\" (UID: \"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06\") "
Jul 28 18:57:59.838498 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:59.838441    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-home\") pod \"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06\" (UID: \"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06\") "
Jul 28 18:57:59.838498 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:59.838486    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hppdt\" (UniqueName: \"kubernetes.io/projected/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-kube-api-access-hppdt\") pod \"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06\" (UID: \"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06\") "
Jul 28 18:57:59.838741 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:59.838661    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1156c8f2-7183-4b71-9cc5-1ec2ebc04f06" (UID: "1156c8f2-7183-4b71-9cc5-1ec2ebc04f06"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:57:59.838741 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:59.838707    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-model-cache" (OuterVolumeSpecName: "model-cache") pod "1156c8f2-7183-4b71-9cc5-1ec2ebc04f06" (UID: "1156c8f2-7183-4b71-9cc5-1ec2ebc04f06"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:57:59.838875 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:59.838854    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-tmp-dir\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:57:59.838875 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:59.838873    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-model-cache\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:57:59.838980 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:59.838907    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-home" (OuterVolumeSpecName: "home") pod "1156c8f2-7183-4b71-9cc5-1ec2ebc04f06" (UID: "1156c8f2-7183-4b71-9cc5-1ec2ebc04f06"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:57:59.840677 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:59.840650    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-dshm" (OuterVolumeSpecName: "dshm") pod "1156c8f2-7183-4b71-9cc5-1ec2ebc04f06" (UID: "1156c8f2-7183-4b71-9cc5-1ec2ebc04f06"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:57:59.840798 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:59.840750    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1156c8f2-7183-4b71-9cc5-1ec2ebc04f06" (UID: "1156c8f2-7183-4b71-9cc5-1ec2ebc04f06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:57:59.841062 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:59.841040    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-kube-api-access-hppdt" (OuterVolumeSpecName: "kube-api-access-hppdt") pod "1156c8f2-7183-4b71-9cc5-1ec2ebc04f06" (UID: "1156c8f2-7183-4b71-9cc5-1ec2ebc04f06"). InnerVolumeSpecName "kube-api-access-hppdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:57:59.939997 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:59.939912    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-home\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:57:59.939997 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:59.939947    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hppdt\" (UniqueName: \"kubernetes.io/projected/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-kube-api-access-hppdt\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:57:59.939997 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:59.939961    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-dshm\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:57:59.939997 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:57:59.939976    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:58:00.160886 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:58:00.160851    2543 generic.go:356] "Generic (PLEG): container finished" podID="1156c8f2-7183-4b71-9cc5-1ec2ebc04f06" containerID="ac7690a6c58c38d57958855a453b1878300d6931beba9d4348595600e3de7de9" exitCode=0
Jul 28 18:58:00.161056 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:58:00.160905    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf"
Jul 28 18:58:00.161056 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:58:00.160935    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf" event={"ID":"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06","Type":"ContainerDied","Data":"ac7690a6c58c38d57958855a453b1878300d6931beba9d4348595600e3de7de9"}
Jul 28 18:58:00.161056 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:58:00.160987    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf" event={"ID":"1156c8f2-7183-4b71-9cc5-1ec2ebc04f06","Type":"ContainerDied","Data":"26438fb8ef5b15be235283e68758a7560f1ee5a1519cefdbea010e968c4e37f8"}
Jul 28 18:58:00.161056 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:58:00.161013    2543 scope.go:117] "RemoveContainer" containerID="ac7690a6c58c38d57958855a453b1878300d6931beba9d4348595600e3de7de9"
Jul 28 18:58:00.171411 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:58:00.171393    2543 scope.go:117] "RemoveContainer" containerID="ac7690a6c58c38d57958855a453b1878300d6931beba9d4348595600e3de7de9"
Jul 28 18:58:00.171659 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:58:00.171639    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac7690a6c58c38d57958855a453b1878300d6931beba9d4348595600e3de7de9\": container with ID starting with ac7690a6c58c38d57958855a453b1878300d6931beba9d4348595600e3de7de9 not found: ID does not exist" containerID="ac7690a6c58c38d57958855a453b1878300d6931beba9d4348595600e3de7de9"
Jul 28 18:58:00.171724 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:58:00.171673    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac7690a6c58c38d57958855a453b1878300d6931beba9d4348595600e3de7de9"} err="failed to get container status \"ac7690a6c58c38d57958855a453b1878300d6931beba9d4348595600e3de7de9\": rpc error: code = NotFound desc = could not find container \"ac7690a6c58c38d57958855a453b1878300d6931beba9d4348595600e3de7de9\": container with ID starting with ac7690a6c58c38d57958855a453b1878300d6931beba9d4348595600e3de7de9 not found: ID does not exist"
Jul 28 18:58:00.186803 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:58:00.186782    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf"]
Jul 28 18:58:00.193753 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:58:00.193698    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-zqcmf"]
Jul 28 18:58:01.154680 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:58:01.154642    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1156c8f2-7183-4b71-9cc5-1ec2ebc04f06" path="/var/lib/kubelet/pods/1156c8f2-7183-4b71-9cc5-1ec2ebc04f06/volumes"
Jul 28 18:58:02.539678 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:58:02.539640    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8" podUID="1c409c83-3d4a-40a6-9d31-4982e37030c4"
Jul 28 18:58:03.175575 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:58:03.175544    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 18:58:58.455405 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:58:58.455372    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-545d98d4c7-5zmwp"]
Jul 28 18:58:58.456052 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:58:58.455602    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/llmisvc-controller-manager-545d98d4c7-5zmwp" podUID="0ef61165-c90b-4f43-b6e8-43b9bb0909bd" containerName="manager" containerID="cri-o://ee3641e8554bfb89f82e0467d513661ca3e6a80b258488785dd4027b271965bc" gracePeriod=30
Jul 28 18:59:04.009414 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:59:04.009386    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-545d98d4c7-5zmwp"
Jul 28 18:59:04.097685 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:59:04.097597    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ef61165-c90b-4f43-b6e8-43b9bb0909bd-cert\") pod \"0ef61165-c90b-4f43-b6e8-43b9bb0909bd\" (UID: \"0ef61165-c90b-4f43-b6e8-43b9bb0909bd\") "
Jul 28 18:59:04.097685 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:59:04.097635    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cdz8\" (UniqueName: \"kubernetes.io/projected/0ef61165-c90b-4f43-b6e8-43b9bb0909bd-kube-api-access-9cdz8\") pod \"0ef61165-c90b-4f43-b6e8-43b9bb0909bd\" (UID: \"0ef61165-c90b-4f43-b6e8-43b9bb0909bd\") "
Jul 28 18:59:04.099789 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:59:04.099766    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ef61165-c90b-4f43-b6e8-43b9bb0909bd-cert" (OuterVolumeSpecName: "cert") pod "0ef61165-c90b-4f43-b6e8-43b9bb0909bd" (UID: "0ef61165-c90b-4f43-b6e8-43b9bb0909bd"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:59:04.099789 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:59:04.099780    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ef61165-c90b-4f43-b6e8-43b9bb0909bd-kube-api-access-9cdz8" (OuterVolumeSpecName: "kube-api-access-9cdz8") pod "0ef61165-c90b-4f43-b6e8-43b9bb0909bd" (UID: "0ef61165-c90b-4f43-b6e8-43b9bb0909bd"). InnerVolumeSpecName "kube-api-access-9cdz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:59:04.199055 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:59:04.199016    2543 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ef61165-c90b-4f43-b6e8-43b9bb0909bd-cert\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:59:04.199055 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:59:04.199059    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cdz8\" (UniqueName: \"kubernetes.io/projected/0ef61165-c90b-4f43-b6e8-43b9bb0909bd-kube-api-access-9cdz8\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 18:59:04.488326 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:59:04.488279    2543 generic.go:356] "Generic (PLEG): container finished" podID="0ef61165-c90b-4f43-b6e8-43b9bb0909bd" containerID="ee3641e8554bfb89f82e0467d513661ca3e6a80b258488785dd4027b271965bc" exitCode=0
Jul 28 18:59:04.488326 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:59:04.488319    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-545d98d4c7-5zmwp" event={"ID":"0ef61165-c90b-4f43-b6e8-43b9bb0909bd","Type":"ContainerDied","Data":"ee3641e8554bfb89f82e0467d513661ca3e6a80b258488785dd4027b271965bc"}
Jul 28 18:59:04.488598 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:59:04.488340    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-545d98d4c7-5zmwp"
Jul 28 18:59:04.488598 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:59:04.488374    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-545d98d4c7-5zmwp" event={"ID":"0ef61165-c90b-4f43-b6e8-43b9bb0909bd","Type":"ContainerDied","Data":"dcc56d006bb2091f6b7ffb0a6e972993fd6c922b5a259153681b802a31f5c846"}
Jul 28 18:59:04.488598 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:59:04.488399    2543 scope.go:117] "RemoveContainer" containerID="ee3641e8554bfb89f82e0467d513661ca3e6a80b258488785dd4027b271965bc"
Jul 28 18:59:04.501216 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:59:04.501196    2543 scope.go:117] "RemoveContainer" containerID="ee3641e8554bfb89f82e0467d513661ca3e6a80b258488785dd4027b271965bc"
Jul 28 18:59:04.501487 ip-10-0-135-177 kubenswrapper[2543]: E0728 18:59:04.501465    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee3641e8554bfb89f82e0467d513661ca3e6a80b258488785dd4027b271965bc\": container with ID starting with ee3641e8554bfb89f82e0467d513661ca3e6a80b258488785dd4027b271965bc not found: ID does not exist" containerID="ee3641e8554bfb89f82e0467d513661ca3e6a80b258488785dd4027b271965bc"
Jul 28 18:59:04.501560 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:59:04.501495    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee3641e8554bfb89f82e0467d513661ca3e6a80b258488785dd4027b271965bc"} err="failed to get container status \"ee3641e8554bfb89f82e0467d513661ca3e6a80b258488785dd4027b271965bc\": rpc error: code = NotFound desc = could not find container \"ee3641e8554bfb89f82e0467d513661ca3e6a80b258488785dd4027b271965bc\": container with ID starting with ee3641e8554bfb89f82e0467d513661ca3e6a80b258488785dd4027b271965bc not found: ID does not exist"
Jul 28 18:59:04.510012 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:59:04.509989    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-545d98d4c7-5zmwp"]
Jul 28 18:59:04.513777 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:59:04.513753    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/llmisvc-controller-manager-545d98d4c7-5zmwp"]
Jul 28 18:59:05.152385 ip-10-0-135-177 kubenswrapper[2543]: I0728 18:59:05.152356    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ef61165-c90b-4f43-b6e8-43b9bb0909bd" path="/var/lib/kubelet/pods/0ef61165-c90b-4f43-b6e8-43b9bb0909bd/volumes"
Jul 28 19:00:06.176297 ip-10-0-135-177 kubenswrapper[2543]: E0728 19:00:06.176259    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8" podUID="1c409c83-3d4a-40a6-9d31-4982e37030c4"
Jul 28 19:00:06.813903 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:00:06.813867    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 19:02:09.814963 ip-10-0-135-177 kubenswrapper[2543]: E0728 19:02:09.814855    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8" podUID="1c409c83-3d4a-40a6-9d31-4982e37030c4"
Jul 28 19:02:10.427643 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:02:10.427603    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 19:04:13.429199 ip-10-0-135-177 kubenswrapper[2543]: E0728 19:04:13.429086    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8" podUID="1c409c83-3d4a-40a6-9d31-4982e37030c4"
Jul 28 19:04:14.046623 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:04:14.046595    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 19:05:15.785914 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:15.785820    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx"]
Jul 28 19:05:15.800364 ip-10-0-135-177 kubenswrapper[2543]: E0728 19:05:15.800321    2543 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 19:05:15.800364 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 19:05:15.800364 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 19:05:15.800364 ip-10-0-135-177 kubenswrapper[2543]:  > containerID="8311bd7e9dec2959eb3a7650a558c899606c3d192104be257dddb8d321451396" cmd=["/bin/sleep","15"]
Jul 28 19:05:15.800661 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:15.800381    2543 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx" message="" err=<
Jul 28 19:05:15.800661 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 19:05:15.800661 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 19:05:15.800661 ip-10-0-135-177 kubenswrapper[2543]:  >
Jul 28 19:05:15.800661 ip-10-0-135-177 kubenswrapper[2543]: E0728 19:05:15.800394    2543 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 19:05:15.800661 ip-10-0-135-177 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 19:05:15.800661 ip-10-0-135-177 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 28 19:05:15.800661 ip-10-0-135-177 kubenswrapper[2543]:  > pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx" podUID="49991427-8664-4d2a-ac37-31467363ecde" containerName="main" containerID="cri-o://8311bd7e9dec2959eb3a7650a558c899606c3d192104be257dddb8d321451396"
Jul 28 19:05:15.800661 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:15.800441    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx" podUID="49991427-8664-4d2a-ac37-31467363ecde" containerName="main" containerID="cri-o://8311bd7e9dec2959eb3a7650a558c899606c3d192104be257dddb8d321451396" gracePeriod=60
Jul 28 19:05:16.007072 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:16.007030    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx" podUID="49991427-8664-4d2a-ac37-31467363ecde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": dial tcp 10.134.0.78:8000: connect: connection refused"
Jul 28 19:05:16.152826 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:16.152797    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx"
Jul 28 19:05:16.257501 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:16.257470    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49991427-8664-4d2a-ac37-31467363ecde-dshm\") pod \"49991427-8664-4d2a-ac37-31467363ecde\" (UID: \"49991427-8664-4d2a-ac37-31467363ecde\") "
Jul 28 19:05:16.257704 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:16.257520    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49991427-8664-4d2a-ac37-31467363ecde-tls-certs\") pod \"49991427-8664-4d2a-ac37-31467363ecde\" (UID: \"49991427-8664-4d2a-ac37-31467363ecde\") "
Jul 28 19:05:16.257704 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:16.257554    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrqnb\" (UniqueName: \"kubernetes.io/projected/49991427-8664-4d2a-ac37-31467363ecde-kube-api-access-qrqnb\") pod \"49991427-8664-4d2a-ac37-31467363ecde\" (UID: \"49991427-8664-4d2a-ac37-31467363ecde\") "
Jul 28 19:05:16.257704 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:16.257598    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49991427-8664-4d2a-ac37-31467363ecde-tmp-dir\") pod \"49991427-8664-4d2a-ac37-31467363ecde\" (UID: \"49991427-8664-4d2a-ac37-31467363ecde\") "
Jul 28 19:05:16.257704 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:16.257629    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49991427-8664-4d2a-ac37-31467363ecde-model-cache\") pod \"49991427-8664-4d2a-ac37-31467363ecde\" (UID: \"49991427-8664-4d2a-ac37-31467363ecde\") "
Jul 28 19:05:16.257704 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:16.257659    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49991427-8664-4d2a-ac37-31467363ecde-home\") pod \"49991427-8664-4d2a-ac37-31467363ecde\" (UID: \"49991427-8664-4d2a-ac37-31467363ecde\") "
Jul 28 19:05:16.257963 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:16.257895    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49991427-8664-4d2a-ac37-31467363ecde-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "49991427-8664-4d2a-ac37-31467363ecde" (UID: "49991427-8664-4d2a-ac37-31467363ecde"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 19:05:16.257963 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:16.257931    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49991427-8664-4d2a-ac37-31467363ecde-model-cache" (OuterVolumeSpecName: "model-cache") pod "49991427-8664-4d2a-ac37-31467363ecde" (UID: "49991427-8664-4d2a-ac37-31467363ecde"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 19:05:16.258057 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:16.258021    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49991427-8664-4d2a-ac37-31467363ecde-tmp-dir\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 19:05:16.258095 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:16.258073    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49991427-8664-4d2a-ac37-31467363ecde-model-cache\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 19:05:16.258165 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:16.258112    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49991427-8664-4d2a-ac37-31467363ecde-home" (OuterVolumeSpecName: "home") pod "49991427-8664-4d2a-ac37-31467363ecde" (UID: "49991427-8664-4d2a-ac37-31467363ecde"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 19:05:16.259863 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:16.259840    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49991427-8664-4d2a-ac37-31467363ecde-dshm" (OuterVolumeSpecName: "dshm") pod "49991427-8664-4d2a-ac37-31467363ecde" (UID: "49991427-8664-4d2a-ac37-31467363ecde"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 19:05:16.259983 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:16.259886    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49991427-8664-4d2a-ac37-31467363ecde-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "49991427-8664-4d2a-ac37-31467363ecde" (UID: "49991427-8664-4d2a-ac37-31467363ecde"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 19:05:16.260031 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:16.259983    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49991427-8664-4d2a-ac37-31467363ecde-kube-api-access-qrqnb" (OuterVolumeSpecName: "kube-api-access-qrqnb") pod "49991427-8664-4d2a-ac37-31467363ecde" (UID: "49991427-8664-4d2a-ac37-31467363ecde"). InnerVolumeSpecName "kube-api-access-qrqnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 19:05:16.358608 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:16.358510    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49991427-8664-4d2a-ac37-31467363ecde-dshm\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 19:05:16.358608 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:16.358539    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49991427-8664-4d2a-ac37-31467363ecde-tls-certs\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 19:05:16.358608 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:16.358554    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrqnb\" (UniqueName: \"kubernetes.io/projected/49991427-8664-4d2a-ac37-31467363ecde-kube-api-access-qrqnb\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 19:05:16.358608 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:16.358564    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49991427-8664-4d2a-ac37-31467363ecde-home\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 19:05:16.359212 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:16.359184    2543 generic.go:356] "Generic (PLEG): container finished" podID="49991427-8664-4d2a-ac37-31467363ecde" containerID="8311bd7e9dec2959eb3a7650a558c899606c3d192104be257dddb8d321451396" exitCode=0
Jul 28 19:05:16.359330 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:16.359238    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx"
Jul 28 19:05:16.359330 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:16.359272    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx" event={"ID":"49991427-8664-4d2a-ac37-31467363ecde","Type":"ContainerDied","Data":"8311bd7e9dec2959eb3a7650a558c899606c3d192104be257dddb8d321451396"}
Jul 28 19:05:16.359420 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:16.359330    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx" event={"ID":"49991427-8664-4d2a-ac37-31467363ecde","Type":"ContainerDied","Data":"085f4c0b72efe3fbbfdbff4b046790d33b2ddfe9ae7c84374df6c65b7d0b8e96"}
Jul 28 19:05:16.359420 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:16.359360    2543 scope.go:117] "RemoveContainer" containerID="8311bd7e9dec2959eb3a7650a558c899606c3d192104be257dddb8d321451396"
Jul 28 19:05:16.368579 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:16.368562    2543 scope.go:117] "RemoveContainer" containerID="8311bd7e9dec2959eb3a7650a558c899606c3d192104be257dddb8d321451396"
Jul 28 19:05:16.368852 ip-10-0-135-177 kubenswrapper[2543]: E0728 19:05:16.368834    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8311bd7e9dec2959eb3a7650a558c899606c3d192104be257dddb8d321451396\": container with ID starting with 8311bd7e9dec2959eb3a7650a558c899606c3d192104be257dddb8d321451396 not found: ID does not exist" containerID="8311bd7e9dec2959eb3a7650a558c899606c3d192104be257dddb8d321451396"
Jul 28 19:05:16.368905 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:16.368862    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8311bd7e9dec2959eb3a7650a558c899606c3d192104be257dddb8d321451396"} err="failed to get container status \"8311bd7e9dec2959eb3a7650a558c899606c3d192104be257dddb8d321451396\": rpc error: code = NotFound desc = could not find container \"8311bd7e9dec2959eb3a7650a558c899606c3d192104be257dddb8d321451396\": container with ID starting with 8311bd7e9dec2959eb3a7650a558c899606c3d192104be257dddb8d321451396 not found: ID does not exist"
Jul 28 19:05:16.382162 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:16.382126    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx"]
Jul 28 19:05:16.388305 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:16.388285    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-cleanup-hpa-a41f0e40/autoscale-cleanup-hpa-kserve-6cfdf55fc9-9mnrx"]
Jul 28 19:05:17.153231 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:17.153197    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49991427-8664-4d2a-ac37-31467363ecde" path="/var/lib/kubelet/pods/49991427-8664-4d2a-ac37-31467363ecde/volumes"
Jul 28 19:05:34.765716 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:34.765680    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jhlmk/must-gather-2lkr5"]
Jul 28 19:05:34.766201 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:34.766006    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1156c8f2-7183-4b71-9cc5-1ec2ebc04f06" containerName="main"
Jul 28 19:05:34.766201 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:34.766016    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="1156c8f2-7183-4b71-9cc5-1ec2ebc04f06" containerName="main"
Jul 28 19:05:34.766201 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:34.766025    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49991427-8664-4d2a-ac37-31467363ecde" containerName="main"
Jul 28 19:05:34.766201 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:34.766030    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="49991427-8664-4d2a-ac37-31467363ecde" containerName="main"
Jul 28 19:05:34.766201 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:34.766039    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ef61165-c90b-4f43-b6e8-43b9bb0909bd" containerName="manager"
Jul 28 19:05:34.766201 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:34.766045    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ef61165-c90b-4f43-b6e8-43b9bb0909bd" containerName="manager"
Jul 28 19:05:34.766201 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:34.766095    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="49991427-8664-4d2a-ac37-31467363ecde" containerName="main"
Jul 28 19:05:34.766201 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:34.766104    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="1156c8f2-7183-4b71-9cc5-1ec2ebc04f06" containerName="main"
Jul 28 19:05:34.766201 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:34.766111    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ef61165-c90b-4f43-b6e8-43b9bb0909bd" containerName="manager"
Jul 28 19:05:34.769197 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:34.769181    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jhlmk/must-gather-2lkr5"
Jul 28 19:05:34.771887 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:34.771865    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jhlmk\"/\"openshift-service-ca.crt\""
Jul 28 19:05:34.772536 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:34.772514    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jhlmk\"/\"kube-root-ca.crt\""
Jul 28 19:05:34.772651 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:34.772522    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jhlmk\"/\"default-dockercfg-xm8zx\""
Jul 28 19:05:34.777973 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:34.777946    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jhlmk/must-gather-2lkr5"]
Jul 28 19:05:34.928378 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:34.928341    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh9bw\" (UniqueName: \"kubernetes.io/projected/b396318f-dfeb-4f93-b7b2-bf83bb6fe670-kube-api-access-qh9bw\") pod \"must-gather-2lkr5\" (UID: \"b396318f-dfeb-4f93-b7b2-bf83bb6fe670\") " pod="openshift-must-gather-jhlmk/must-gather-2lkr5"
Jul 28 19:05:34.928549 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:34.928455    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b396318f-dfeb-4f93-b7b2-bf83bb6fe670-must-gather-output\") pod \"must-gather-2lkr5\" (UID: \"b396318f-dfeb-4f93-b7b2-bf83bb6fe670\") " pod="openshift-must-gather-jhlmk/must-gather-2lkr5"
Jul 28 19:05:35.029346 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:35.029251    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b396318f-dfeb-4f93-b7b2-bf83bb6fe670-must-gather-output\") pod \"must-gather-2lkr5\" (UID: \"b396318f-dfeb-4f93-b7b2-bf83bb6fe670\") " pod="openshift-must-gather-jhlmk/must-gather-2lkr5"
Jul 28 19:05:35.029346 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:35.029311    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh9bw\" (UniqueName: \"kubernetes.io/projected/b396318f-dfeb-4f93-b7b2-bf83bb6fe670-kube-api-access-qh9bw\") pod \"must-gather-2lkr5\" (UID: \"b396318f-dfeb-4f93-b7b2-bf83bb6fe670\") " pod="openshift-must-gather-jhlmk/must-gather-2lkr5"
Jul 28 19:05:35.029688 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:35.029664    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b396318f-dfeb-4f93-b7b2-bf83bb6fe670-must-gather-output\") pod \"must-gather-2lkr5\" (UID: \"b396318f-dfeb-4f93-b7b2-bf83bb6fe670\") " pod="openshift-must-gather-jhlmk/must-gather-2lkr5"
Jul 28 19:05:35.037290 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:35.037257    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh9bw\" (UniqueName: \"kubernetes.io/projected/b396318f-dfeb-4f93-b7b2-bf83bb6fe670-kube-api-access-qh9bw\") pod \"must-gather-2lkr5\" (UID: \"b396318f-dfeb-4f93-b7b2-bf83bb6fe670\") " pod="openshift-must-gather-jhlmk/must-gather-2lkr5"
Jul 28 19:05:35.078940 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:35.078916    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jhlmk/must-gather-2lkr5"
Jul 28 19:05:35.211887 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:35.211858    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jhlmk/must-gather-2lkr5"]
Jul 28 19:05:35.213742 ip-10-0-135-177 kubenswrapper[2543]: W0728 19:05:35.213721    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb396318f_dfeb_4f93_b7b2_bf83bb6fe670.slice/crio-6db77df2001777feffc3fb214655c15ac3a3f48a622df69b9f8c52c080af6445 WatchSource:0}: Error finding container 6db77df2001777feffc3fb214655c15ac3a3f48a622df69b9f8c52c080af6445: Status 404 returned error can't find the container with id 6db77df2001777feffc3fb214655c15ac3a3f48a622df69b9f8c52c080af6445
Jul 28 19:05:35.215667 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:35.215648    2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 19:05:35.452077 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:35.451995    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jhlmk/must-gather-2lkr5" event={"ID":"b396318f-dfeb-4f93-b7b2-bf83bb6fe670","Type":"ContainerStarted","Data":"6db77df2001777feffc3fb214655c15ac3a3f48a622df69b9f8c52c080af6445"}
Jul 28 19:05:41.499377 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:41.499342    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jhlmk/must-gather-2lkr5" event={"ID":"b396318f-dfeb-4f93-b7b2-bf83bb6fe670","Type":"ContainerStarted","Data":"3e6f16a24f49217e31c29fc48da2f62c321c62414e3e4a35ab49aed5a9492f71"}
Jul 28 19:05:42.507177 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:42.507123    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jhlmk/must-gather-2lkr5" event={"ID":"b396318f-dfeb-4f93-b7b2-bf83bb6fe670","Type":"ContainerStarted","Data":"e3031f0d136830cd47837542d5d42dd540aa01b23d4cc0586aeb4eaafeaf41f1"}
Jul 28 19:05:42.524384 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:42.524330    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jhlmk/must-gather-2lkr5" podStartSLOduration=2.40445884 podStartE2EDuration="8.52431439s" totalImagesPullingTime="6.11985555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:05:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:05:42.522530325 +0000 UTC m=+6145.928804066" watchObservedRunningTime="2026-07-28 19:05:42.52431439 +0000 UTC m=+6145.930588131"
Jul 28 19:05:58.201963 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:58.201931    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5657cc6765-srj79_908c16e7-90b9-4f02-a950-e62eafb70d81/router/0.log"
Jul 28 19:05:59.334658 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:05:59.334623    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5657cc6765-srj79_908c16e7-90b9-4f02-a950-e62eafb70d81/router/0.log"
Jul 28 19:06:00.443823 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:00.443779    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5657cc6765-srj79_908c16e7-90b9-4f02-a950-e62eafb70d81/router/0.log"
Jul 28 19:06:01.595827 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:01.595774    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5657cc6765-srj79_908c16e7-90b9-4f02-a950-e62eafb70d81/router/0.log"
Jul 28 19:06:02.549004 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:02.548969    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5shw9_ae97ec5d-a62d-4674-bf00-578d8afe1cad/authorino/0.log"
Jul 28 19:06:03.825220 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:03.825183    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5shw9_ae97ec5d-a62d-4674-bf00-578d8afe1cad/authorino/0.log"
Jul 28 19:06:05.252434 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:05.252399    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5shw9_ae97ec5d-a62d-4674-bf00-578d8afe1cad/authorino/0.log"
Jul 28 19:06:06.439742 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:06.439703    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5shw9_ae97ec5d-a62d-4674-bf00-578d8afe1cad/authorino/0.log"
Jul 28 19:06:07.630636 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:07.630598    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5shw9_ae97ec5d-a62d-4674-bf00-578d8afe1cad/authorino/0.log"
Jul 28 19:06:08.814544 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:08.814501    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5shw9_ae97ec5d-a62d-4674-bf00-578d8afe1cad/authorino/0.log"
Jul 28 19:06:10.026314 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:10.026276    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5shw9_ae97ec5d-a62d-4674-bf00-578d8afe1cad/authorino/0.log"
Jul 28 19:06:11.197751 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:11.197716    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5shw9_ae97ec5d-a62d-4674-bf00-578d8afe1cad/authorino/0.log"
Jul 28 19:06:12.400048 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:12.400009    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5shw9_ae97ec5d-a62d-4674-bf00-578d8afe1cad/authorino/0.log"
Jul 28 19:06:13.616021 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:13.615986    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5shw9_ae97ec5d-a62d-4674-bf00-578d8afe1cad/authorino/0.log"
Jul 28 19:06:14.795727 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:14.795698    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5shw9_ae97ec5d-a62d-4674-bf00-578d8afe1cad/authorino/0.log"
Jul 28 19:06:16.062868 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:16.062810    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5shw9_ae97ec5d-a62d-4674-bf00-578d8afe1cad/authorino/0.log"
Jul 28 19:06:17.047223 ip-10-0-135-177 kubenswrapper[2543]: E0728 19:06:17.047170    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8" podUID="1c409c83-3d4a-40a6-9d31-4982e37030c4"
Jul 28 19:06:17.151852 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:17.151811    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5shw9_ae97ec5d-a62d-4674-bf00-578d8afe1cad/authorino/0.log"
Jul 28 19:06:17.708076 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:17.708035    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56crj4q8"
Jul 28 19:06:18.367053 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:18.367016    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5shw9_ae97ec5d-a62d-4674-bf00-578d8afe1cad/authorino/0.log"
Jul 28 19:06:19.581291 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:19.581247    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5shw9_ae97ec5d-a62d-4674-bf00-578d8afe1cad/authorino/0.log"
Jul 28 19:06:20.763399 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:20.763368    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5shw9_ae97ec5d-a62d-4674-bf00-578d8afe1cad/authorino/0.log"
Jul 28 19:06:22.086444 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:22.086407    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5shw9_ae97ec5d-a62d-4674-bf00-578d8afe1cad/authorino/0.log"
Jul 28 19:06:23.277248 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:23.277212    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5shw9_ae97ec5d-a62d-4674-bf00-578d8afe1cad/authorino/0.log"
Jul 28 19:06:24.467080 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:24.467044    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5shw9_ae97ec5d-a62d-4674-bf00-578d8afe1cad/authorino/0.log"
Jul 28 19:06:25.618638 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:25.618605    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5shw9_ae97ec5d-a62d-4674-bf00-578d8afe1cad/authorino/0.log"
Jul 28 19:06:26.853007 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:26.852973    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5shw9_ae97ec5d-a62d-4674-bf00-578d8afe1cad/authorino/0.log"
Jul 28 19:06:28.043515 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:28.043477    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5shw9_ae97ec5d-a62d-4674-bf00-578d8afe1cad/authorino/0.log"
Jul 28 19:06:29.288574 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:29.288541    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5shw9_ae97ec5d-a62d-4674-bf00-578d8afe1cad/authorino/0.log"
Jul 28 19:06:30.498380 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:30.498344    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5shw9_ae97ec5d-a62d-4674-bf00-578d8afe1cad/authorino/0.log"
Jul 28 19:06:31.809491 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:31.809462    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5shw9_ae97ec5d-a62d-4674-bf00-578d8afe1cad/authorino/0.log"
Jul 28 19:06:32.808500 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:32.808462    2543 generic.go:356] "Generic (PLEG): container finished" podID="b396318f-dfeb-4f93-b7b2-bf83bb6fe670" containerID="3e6f16a24f49217e31c29fc48da2f62c321c62414e3e4a35ab49aed5a9492f71" exitCode=0
Jul 28 19:06:32.808717 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:32.808539    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jhlmk/must-gather-2lkr5" event={"ID":"b396318f-dfeb-4f93-b7b2-bf83bb6fe670","Type":"ContainerDied","Data":"3e6f16a24f49217e31c29fc48da2f62c321c62414e3e4a35ab49aed5a9492f71"}
Jul 28 19:06:32.808859 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:32.808846    2543 scope.go:117] "RemoveContainer" containerID="3e6f16a24f49217e31c29fc48da2f62c321c62414e3e4a35ab49aed5a9492f71"
Jul 28 19:06:33.603818 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:33.603787    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-jhlmk_must-gather-2lkr5_b396318f-dfeb-4f93-b7b2-bf83bb6fe670/gather/0.log"
Jul 28 19:06:37.143194 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:37.143160    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jtb46_145627c2-9b95-4251-9f16-2920555de2fa/global-pull-secret-syncer/0.log"
Jul 28 19:06:37.416034 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:37.415945    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7qw2x_89471f5d-8eda-4a72-ab86-5f2290e75842/konnectivity-agent/0.log"
Jul 28 19:06:37.599189 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:37.599156    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-177.ec2.internal_a0b77ac91bc5dbb86ffeab81485172d7/haproxy/0.log"
Jul 28 19:06:39.048028 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:39.047947    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-jhlmk/must-gather-2lkr5"]
Jul 28 19:06:39.048479 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:39.048157    2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-jhlmk/must-gather-2lkr5" podUID="b396318f-dfeb-4f93-b7b2-bf83bb6fe670" containerName="copy" containerID="cri-o://e3031f0d136830cd47837542d5d42dd540aa01b23d4cc0586aeb4eaafeaf41f1" gracePeriod=2
Jul 28 19:06:39.050863 ip-10-0-135-177 kubenswrapper[2543]: E0728 19:06:39.050815    2543 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-2lkr5\" is forbidden: User \"system:node:ip-10-0-135-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-jhlmk\": no relationship found between node 'ip-10-0-135-177.ec2.internal' and this object" podUID="b396318f-dfeb-4f93-b7b2-bf83bb6fe670" pod="openshift-must-gather-jhlmk/must-gather-2lkr5"
Jul 28 19:06:39.051695 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:39.051666    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-jhlmk/must-gather-2lkr5"]
Jul 28 19:06:39.281651 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:39.281626    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-jhlmk_must-gather-2lkr5_b396318f-dfeb-4f93-b7b2-bf83bb6fe670/copy/0.log"
Jul 28 19:06:39.284178 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:39.284157    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jhlmk/must-gather-2lkr5"
Jul 28 19:06:39.316970 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:39.316907    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qh9bw\" (UniqueName: \"kubernetes.io/projected/b396318f-dfeb-4f93-b7b2-bf83bb6fe670-kube-api-access-qh9bw\") pod \"b396318f-dfeb-4f93-b7b2-bf83bb6fe670\" (UID: \"b396318f-dfeb-4f93-b7b2-bf83bb6fe670\") "
Jul 28 19:06:39.317086 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:39.316989    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b396318f-dfeb-4f93-b7b2-bf83bb6fe670-must-gather-output\") pod \"b396318f-dfeb-4f93-b7b2-bf83bb6fe670\" (UID: \"b396318f-dfeb-4f93-b7b2-bf83bb6fe670\") "
Jul 28 19:06:39.319340 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:39.319313    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b396318f-dfeb-4f93-b7b2-bf83bb6fe670-kube-api-access-qh9bw" (OuterVolumeSpecName: "kube-api-access-qh9bw") pod "b396318f-dfeb-4f93-b7b2-bf83bb6fe670" (UID: "b396318f-dfeb-4f93-b7b2-bf83bb6fe670"). InnerVolumeSpecName "kube-api-access-qh9bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 19:06:39.323505 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:39.323476    2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b396318f-dfeb-4f93-b7b2-bf83bb6fe670-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "b396318f-dfeb-4f93-b7b2-bf83bb6fe670" (UID: "b396318f-dfeb-4f93-b7b2-bf83bb6fe670"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 19:06:39.418336 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:39.418312    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh9bw\" (UniqueName: \"kubernetes.io/projected/b396318f-dfeb-4f93-b7b2-bf83bb6fe670-kube-api-access-qh9bw\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 19:06:39.418336 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:39.418334    2543 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b396318f-dfeb-4f93-b7b2-bf83bb6fe670-must-gather-output\") on node \"ip-10-0-135-177.ec2.internal\" DevicePath \"\""
Jul 28 19:06:39.847052 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:39.847027    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-jhlmk_must-gather-2lkr5_b396318f-dfeb-4f93-b7b2-bf83bb6fe670/copy/0.log"
Jul 28 19:06:39.847421 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:39.847397    2543 generic.go:356] "Generic (PLEG): container finished" podID="b396318f-dfeb-4f93-b7b2-bf83bb6fe670" containerID="e3031f0d136830cd47837542d5d42dd540aa01b23d4cc0586aeb4eaafeaf41f1" exitCode=143
Jul 28 19:06:39.847476 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:39.847449    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jhlmk/must-gather-2lkr5"
Jul 28 19:06:39.847514 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:39.847496    2543 scope.go:117] "RemoveContainer" containerID="e3031f0d136830cd47837542d5d42dd540aa01b23d4cc0586aeb4eaafeaf41f1"
Jul 28 19:06:39.856019 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:39.856004    2543 scope.go:117] "RemoveContainer" containerID="3e6f16a24f49217e31c29fc48da2f62c321c62414e3e4a35ab49aed5a9492f71"
Jul 28 19:06:39.868841 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:39.868825    2543 scope.go:117] "RemoveContainer" containerID="e3031f0d136830cd47837542d5d42dd540aa01b23d4cc0586aeb4eaafeaf41f1"
Jul 28 19:06:39.869090 ip-10-0-135-177 kubenswrapper[2543]: E0728 19:06:39.869072    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3031f0d136830cd47837542d5d42dd540aa01b23d4cc0586aeb4eaafeaf41f1\": container with ID starting with e3031f0d136830cd47837542d5d42dd540aa01b23d4cc0586aeb4eaafeaf41f1 not found: ID does not exist" containerID="e3031f0d136830cd47837542d5d42dd540aa01b23d4cc0586aeb4eaafeaf41f1"
Jul 28 19:06:39.869188 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:39.869097    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3031f0d136830cd47837542d5d42dd540aa01b23d4cc0586aeb4eaafeaf41f1"} err="failed to get container status \"e3031f0d136830cd47837542d5d42dd540aa01b23d4cc0586aeb4eaafeaf41f1\": rpc error: code = NotFound desc = could not find container \"e3031f0d136830cd47837542d5d42dd540aa01b23d4cc0586aeb4eaafeaf41f1\": container with ID starting with e3031f0d136830cd47837542d5d42dd540aa01b23d4cc0586aeb4eaafeaf41f1 not found: ID does not exist"
Jul 28 19:06:39.869188 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:39.869114    2543 scope.go:117] "RemoveContainer" containerID="3e6f16a24f49217e31c29fc48da2f62c321c62414e3e4a35ab49aed5a9492f71"
Jul 28 19:06:39.869423 ip-10-0-135-177 kubenswrapper[2543]: E0728 19:06:39.869401    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e6f16a24f49217e31c29fc48da2f62c321c62414e3e4a35ab49aed5a9492f71\": container with ID starting with 3e6f16a24f49217e31c29fc48da2f62c321c62414e3e4a35ab49aed5a9492f71 not found: ID does not exist" containerID="3e6f16a24f49217e31c29fc48da2f62c321c62414e3e4a35ab49aed5a9492f71"
Jul 28 19:06:39.869464 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:39.869429    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e6f16a24f49217e31c29fc48da2f62c321c62414e3e4a35ab49aed5a9492f71"} err="failed to get container status \"3e6f16a24f49217e31c29fc48da2f62c321c62414e3e4a35ab49aed5a9492f71\": rpc error: code = NotFound desc = could not find container \"3e6f16a24f49217e31c29fc48da2f62c321c62414e3e4a35ab49aed5a9492f71\": container with ID starting with 3e6f16a24f49217e31c29fc48da2f62c321c62414e3e4a35ab49aed5a9492f71 not found: ID does not exist"
Jul 28 19:06:41.153187 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:41.153127    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b396318f-dfeb-4f93-b7b2-bf83bb6fe670" path="/var/lib/kubelet/pods/b396318f-dfeb-4f93-b7b2-bf83bb6fe670/volumes"
Jul 28 19:06:41.876757 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:41.876719    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5shw9_ae97ec5d-a62d-4674-bf00-578d8afe1cad/authorino/0.log"
Jul 28 19:06:43.855184 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:43.855156    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_573ee9e7-316a-4218-bafe-70345e39c3bd/alertmanager/0.log"
Jul 28 19:06:43.897379 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:43.897355    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_573ee9e7-316a-4218-bafe-70345e39c3bd/config-reloader/0.log"
Jul 28 19:06:43.938179 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:43.938153    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_573ee9e7-316a-4218-bafe-70345e39c3bd/kube-rbac-proxy-web/0.log"
Jul 28 19:06:43.964415 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:43.964394    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_573ee9e7-316a-4218-bafe-70345e39c3bd/kube-rbac-proxy/0.log"
Jul 28 19:06:43.986184 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:43.986153    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_573ee9e7-316a-4218-bafe-70345e39c3bd/kube-rbac-proxy-metric/0.log"
Jul 28 19:06:44.010968 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:44.010945    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_573ee9e7-316a-4218-bafe-70345e39c3bd/prom-label-proxy/0.log"
Jul 28 19:06:44.035392 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:44.035371    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_573ee9e7-316a-4218-bafe-70345e39c3bd/init-config-reloader/0.log"
Jul 28 19:06:44.227106 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:44.227031    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-789948b4bd-tbvt4_9853b4ab-2da8-411e-8f52-089d275dedb0/metrics-server/0.log"
Jul 28 19:06:44.483478 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:44.483397    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g2cmr_341c4c28-6384-46c0-964b-cc2a892e8b5f/node-exporter/0.log"
Jul 28 19:06:44.505150 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:44.505107    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g2cmr_341c4c28-6384-46c0-964b-cc2a892e8b5f/kube-rbac-proxy/0.log"
Jul 28 19:06:44.529357 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:44.529331    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g2cmr_341c4c28-6384-46c0-964b-cc2a892e8b5f/init-textfile/0.log"
Jul 28 19:06:44.954842 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:44.954812    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-zkptp_0231f8b7-62f6-4677-9ff7-a35efb83743d/prometheus-operator-admission-webhook/0.log"
Jul 28 19:06:44.995800 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:44.995765    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5478597cd8-pw7rm_af758b14-72fa-48eb-bb20-4dc7c85693b7/telemeter-client/0.log"
Jul 28 19:06:45.026380 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:45.026355    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5478597cd8-pw7rm_af758b14-72fa-48eb-bb20-4dc7c85693b7/reload/0.log"
Jul 28 19:06:45.056454 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:45.056431    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5478597cd8-pw7rm_af758b14-72fa-48eb-bb20-4dc7c85693b7/kube-rbac-proxy/0.log"
Jul 28 19:06:48.224632 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.224598    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rd9v9/perf-node-gather-daemonset-xc9qm"]
Jul 28 19:06:48.225050 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.225009    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b396318f-dfeb-4f93-b7b2-bf83bb6fe670" containerName="copy"
Jul 28 19:06:48.225050 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.225024    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="b396318f-dfeb-4f93-b7b2-bf83bb6fe670" containerName="copy"
Jul 28 19:06:48.225050 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.225041    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b396318f-dfeb-4f93-b7b2-bf83bb6fe670" containerName="gather"
Jul 28 19:06:48.225050 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.225049    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="b396318f-dfeb-4f93-b7b2-bf83bb6fe670" containerName="gather"
Jul 28 19:06:48.225392 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.225159    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="b396318f-dfeb-4f93-b7b2-bf83bb6fe670" containerName="copy"
Jul 28 19:06:48.225392 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.225175    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="b396318f-dfeb-4f93-b7b2-bf83bb6fe670" containerName="gather"
Jul 28 19:06:48.231860 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.231843    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-xc9qm"
Jul 28 19:06:48.234222 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.234195    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rd9v9\"/\"default-dockercfg-k9wgm\""
Jul 28 19:06:48.234355 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.234239    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rd9v9\"/\"openshift-service-ca.crt\""
Jul 28 19:06:48.234355 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.234315    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rd9v9\"/\"kube-root-ca.crt\""
Jul 28 19:06:48.235799 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.235768    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rd9v9/perf-node-gather-daemonset-xc9qm"]
Jul 28 19:06:48.290839 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.290810    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4074598b-ad05-46fe-aa04-c4efc6734ded-lib-modules\") pod \"perf-node-gather-daemonset-xc9qm\" (UID: \"4074598b-ad05-46fe-aa04-c4efc6734ded\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-xc9qm"
Jul 28 19:06:48.290967 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.290848    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4074598b-ad05-46fe-aa04-c4efc6734ded-proc\") pod \"perf-node-gather-daemonset-xc9qm\" (UID: \"4074598b-ad05-46fe-aa04-c4efc6734ded\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-xc9qm"
Jul 28 19:06:48.290967 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.290912    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4074598b-ad05-46fe-aa04-c4efc6734ded-sys\") pod \"perf-node-gather-daemonset-xc9qm\" (UID: \"4074598b-ad05-46fe-aa04-c4efc6734ded\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-xc9qm"
Jul 28 19:06:48.291039 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.290987    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ck2k\" (UniqueName: \"kubernetes.io/projected/4074598b-ad05-46fe-aa04-c4efc6734ded-kube-api-access-6ck2k\") pod \"perf-node-gather-daemonset-xc9qm\" (UID: \"4074598b-ad05-46fe-aa04-c4efc6734ded\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-xc9qm"
Jul 28 19:06:48.291079 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.291056    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4074598b-ad05-46fe-aa04-c4efc6734ded-podres\") pod \"perf-node-gather-daemonset-xc9qm\" (UID: \"4074598b-ad05-46fe-aa04-c4efc6734ded\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-xc9qm"
Jul 28 19:06:48.392278 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.392250    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4074598b-ad05-46fe-aa04-c4efc6734ded-sys\") pod \"perf-node-gather-daemonset-xc9qm\" (UID: \"4074598b-ad05-46fe-aa04-c4efc6734ded\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-xc9qm"
Jul 28 19:06:48.392434 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.392293    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ck2k\" (UniqueName: \"kubernetes.io/projected/4074598b-ad05-46fe-aa04-c4efc6734ded-kube-api-access-6ck2k\") pod \"perf-node-gather-daemonset-xc9qm\" (UID: \"4074598b-ad05-46fe-aa04-c4efc6734ded\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-xc9qm"
Jul 28 19:06:48.392434 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.392364    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4074598b-ad05-46fe-aa04-c4efc6734ded-podres\") pod \"perf-node-gather-daemonset-xc9qm\" (UID: \"4074598b-ad05-46fe-aa04-c4efc6734ded\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-xc9qm"
Jul 28 19:06:48.392434 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.392370    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4074598b-ad05-46fe-aa04-c4efc6734ded-sys\") pod \"perf-node-gather-daemonset-xc9qm\" (UID: \"4074598b-ad05-46fe-aa04-c4efc6734ded\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-xc9qm"
Jul 28 19:06:48.392434 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.392416    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4074598b-ad05-46fe-aa04-c4efc6734ded-lib-modules\") pod \"perf-node-gather-daemonset-xc9qm\" (UID: \"4074598b-ad05-46fe-aa04-c4efc6734ded\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-xc9qm"
Jul 28 19:06:48.392434 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.392435    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4074598b-ad05-46fe-aa04-c4efc6734ded-proc\") pod \"perf-node-gather-daemonset-xc9qm\" (UID: \"4074598b-ad05-46fe-aa04-c4efc6734ded\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-xc9qm"
Jul 28 19:06:48.392701 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.392490    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4074598b-ad05-46fe-aa04-c4efc6734ded-podres\") pod \"perf-node-gather-daemonset-xc9qm\" (UID: \"4074598b-ad05-46fe-aa04-c4efc6734ded\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-xc9qm"
Jul 28 19:06:48.392701 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.392515    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4074598b-ad05-46fe-aa04-c4efc6734ded-proc\") pod \"perf-node-gather-daemonset-xc9qm\" (UID: \"4074598b-ad05-46fe-aa04-c4efc6734ded\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-xc9qm"
Jul 28 19:06:48.392701 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.392554    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4074598b-ad05-46fe-aa04-c4efc6734ded-lib-modules\") pod \"perf-node-gather-daemonset-xc9qm\" (UID: \"4074598b-ad05-46fe-aa04-c4efc6734ded\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-xc9qm"
Jul 28 19:06:48.400263 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.400240    2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ck2k\" (UniqueName: \"kubernetes.io/projected/4074598b-ad05-46fe-aa04-c4efc6734ded-kube-api-access-6ck2k\") pod \"perf-node-gather-daemonset-xc9qm\" (UID: \"4074598b-ad05-46fe-aa04-c4efc6734ded\") " pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-xc9qm"
Jul 28 19:06:48.542522 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.542490    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-xc9qm"
Jul 28 19:06:48.674694 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.674658    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rd9v9/perf-node-gather-daemonset-xc9qm"]
Jul 28 19:06:48.676291 ip-10-0-135-177 kubenswrapper[2543]: W0728 19:06:48.676262    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4074598b_ad05_46fe_aa04_c4efc6734ded.slice/crio-7c2f208a9c8a9b91c3f4f3943fe355aeaaf5e40aeaf7ae4b9cc3badbb48a5b20 WatchSource:0}: Error finding container 7c2f208a9c8a9b91c3f4f3943fe355aeaaf5e40aeaf7ae4b9cc3badbb48a5b20: Status 404 returned error can't find the container with id 7c2f208a9c8a9b91c3f4f3943fe355aeaaf5e40aeaf7ae4b9cc3badbb48a5b20
Jul 28 19:06:48.898919 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.898836    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-xc9qm" event={"ID":"4074598b-ad05-46fe-aa04-c4efc6734ded","Type":"ContainerStarted","Data":"0b855f4edf7b3001405b367f4f3652f24e6fb09e8ed9d92588c17081c86e08f0"}
Jul 28 19:06:48.898919 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.898885    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-xc9qm" event={"ID":"4074598b-ad05-46fe-aa04-c4efc6734ded","Type":"ContainerStarted","Data":"7c2f208a9c8a9b91c3f4f3943fe355aeaaf5e40aeaf7ae4b9cc3badbb48a5b20"}
Jul 28 19:06:48.899121 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.898940    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-xc9qm"
Jul 28 19:06:48.915884 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:48.915839    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-xc9qm" podStartSLOduration=0.915823284 podStartE2EDuration="915.823284ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:06:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:06:48.914906613 +0000 UTC m=+6212.321180362" watchObservedRunningTime="2026-07-28 19:06:48.915823284 +0000 UTC m=+6212.322097025"
Jul 28 19:06:49.690885 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:49.690850    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hg8ms_3af720f1-3267-48f6-94e2-6f6527c0fcf6/dns/0.log"
Jul 28 19:06:49.716637 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:49.716613    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hg8ms_3af720f1-3267-48f6-94e2-6f6527c0fcf6/kube-rbac-proxy/0.log"
Jul 28 19:06:49.769658 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:49.769635    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kvtkx_3acecb5c-18d4-4260-aa3e-8758184e6733/dns-node-resolver/0.log"
Jul 28 19:06:50.369476 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:50.369443    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5b7ff5c8fb-j995z_949ffdd5-088b-4f73-b4d7-41921e724eab/registry/0.log"
Jul 28 19:06:50.398806 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:50.398778    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4kw4j_ea4785ea-08ca-45f8-a8bc-42524b8d81b4/node-ca/0.log"
Jul 28 19:06:51.955237 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:51.955205    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5657cc6765-srj79_908c16e7-90b9-4f02-a950-e62eafb70d81/router/0.log"
Jul 28 19:06:52.508039 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:52.508006    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9r84r_6a43fd4b-3d38-44b5-9b92-de7f6228df26/serve-healthcheck-canary/0.log"
Jul 28 19:06:53.042192 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:53.042154    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-czx2d_66911cb6-549d-4f4f-8ac6-286401ad1025/kube-rbac-proxy/0.log"
Jul 28 19:06:53.062242 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:53.062216    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-czx2d_66911cb6-549d-4f4f-8ac6-286401ad1025/exporter/0.log"
Jul 28 19:06:53.084589 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:53.084543    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-czx2d_66911cb6-549d-4f4f-8ac6-286401ad1025/extractor/0.log"
Jul 28 19:06:54.913530 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:54.913503    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rd9v9/perf-node-gather-daemonset-xc9qm"
Jul 28 19:06:56.318888 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:56.318850    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-fd6565d47-w5jsc_44f15402-5b8d-4073-a77e-c32c4ac33d23/manager/0.log"
Jul 28 19:06:56.339587 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:56.339562    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-z4vbw_05119f32-e9b9-4557-806d-74dd2c9e2ad0/openshift-lws-operator/0.log"
Jul 28 19:06:57.066071 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:57.066042    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-cd4d44c67-tftvf_2eeab2fe-1bad-4c77-a992-3d4db33d5387/manager/0.log"
Jul 28 19:06:58.469754 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:58.469721    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-jmftq_5258d5f9-6236-4bd2-80d8-8397d993c189/s3-init/0.log"
Jul 28 19:06:58.493243 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:58.493218    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-jmftq_5258d5f9-6236-4bd2-80d8-8397d993c189/download-hf-model/0.log"
Jul 28 19:06:58.536658 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:06:58.536629    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5657cc6765-srj79_908c16e7-90b9-4f02-a950-e62eafb70d81/router/0.log"
Jul 28 19:07:00.116303 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:07:00.116214    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5657cc6765-srj79_908c16e7-90b9-4f02-a950-e62eafb70d81/router/0.log"
Jul 28 19:07:03.664913 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:07:03.664877    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-zqrks_ef7e5979-918f-49c1-9724-7c736713e0ae/migrator/0.log"
Jul 28 19:07:03.747721 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:07:03.747677    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-zqrks_ef7e5979-918f-49c1-9724-7c736713e0ae/graceful-termination/0.log"
Jul 28 19:07:04.105824 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:07:04.105744    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s_6a0600d3-cc7b-4caf-a420-e669a2c09715/kube-storage-version-migrator-operator/1.log"
Jul 28 19:07:04.107334 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:07:04.107303    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-tmb4s_6a0600d3-cc7b-4caf-a420-e669a2c09715/kube-storage-version-migrator-operator/0.log"
Jul 28 19:07:05.453164 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:07:05.453107    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ljfhd_849d3b0f-9f02-4f4d-8811-13794309755a/kube-multus-additional-cni-plugins/0.log"
Jul 28 19:07:05.484118 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:07:05.484087    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ljfhd_849d3b0f-9f02-4f4d-8811-13794309755a/egress-router-binary-copy/0.log"
Jul 28 19:07:05.506854 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:07:05.506818    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ljfhd_849d3b0f-9f02-4f4d-8811-13794309755a/cni-plugins/0.log"
Jul 28 19:07:05.528007 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:07:05.527964    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ljfhd_849d3b0f-9f02-4f4d-8811-13794309755a/bond-cni-plugin/0.log"
Jul 28 19:07:05.550126 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:07:05.550096    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ljfhd_849d3b0f-9f02-4f4d-8811-13794309755a/routeoverride-cni/0.log"
Jul 28 19:07:05.578943 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:07:05.578909    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ljfhd_849d3b0f-9f02-4f4d-8811-13794309755a/whereabouts-cni-bincopy/0.log"
Jul 28 19:07:05.602659 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:07:05.602631    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ljfhd_849d3b0f-9f02-4f4d-8811-13794309755a/whereabouts-cni/0.log"
Jul 28 19:07:05.842096 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:07:05.842067    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-st8s2_be7a3829-e798-4b35-8489-c872935568a8/kube-multus/0.log"
Jul 28 19:07:06.003824 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:07:06.003793    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-sjtrc_a2c1c169-0fd2-4e73-a497-0b69fe85d11f/network-metrics-daemon/0.log"
Jul 28 19:07:06.025036 ip-10-0-135-177 kubenswrapper[2543]: I0728 19:07:06.024997    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-sjtrc_a2c1c169-0fd2-4e73-a497-0b69fe85d11f/kube-rbac-proxy/0.log"