Jul 27 17:28:27.384094 ip-10-0-131-114 systemd[1]: Starting Kubernetes Kubelet...
Jul 27 17:28:27.838694 ip-10-0-131-114 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 17:28:27.838694 ip-10-0-131-114 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 27 17:28:27.838694 ip-10-0-131-114 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 17:28:27.838694 ip-10-0-131-114 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 27 17:28:27.838694 ip-10-0-131-114 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 17:28:27.839791 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.839689    2557 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 27 17:28:27.844927 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.844907    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:28:27.844927 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.844928    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:28:27.844997 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.844933    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:28:27.844997 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.844936    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:28:27.844997 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.844940    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:28:27.844997 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.844943    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:28:27.844997 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.844946    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:28:27.844997 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.844949    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:28:27.844997 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.844952    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:28:27.844997 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.844955    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:28:27.844997 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.844958    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:28:27.844997 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.844960    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:28:27.844997 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.844963    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:28:27.844997 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.844965    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:28:27.844997 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.844968    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:28:27.844997 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.844971    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:28:27.844997 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.844973    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:28:27.844997 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.844976    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:28:27.844997 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.844978    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:28:27.844997 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.844981    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:28:27.844997 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.844983    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:28:27.844997 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.844987    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:28:27.845466 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.844989    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:28:27.845466 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.844992    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:28:27.845466 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.844996    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:28:27.845466 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.844998    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:28:27.845466 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845001    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:28:27.845466 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845004    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:28:27.845466 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845007    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:28:27.845466 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845010    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:28:27.845466 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845013    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:28:27.845466 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845015    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:28:27.845466 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845017    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:28:27.845466 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845020    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:28:27.845466 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845023    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:28:27.845466 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845025    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:28:27.845466 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845028    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:28:27.845466 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845030    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:28:27.845466 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845033    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:28:27.845466 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845035    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:28:27.845466 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845037    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:28:27.845466 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845040    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:28:27.845970 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845042    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:28:27.845970 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845045    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:28:27.845970 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845047    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:28:27.845970 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845050    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:28:27.845970 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845052    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:28:27.845970 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845055    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:28:27.845970 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845057    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:28:27.845970 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845060    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:28:27.845970 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845062    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:28:27.845970 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845065    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:28:27.845970 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845068    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:28:27.845970 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845071    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:28:27.845970 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845074    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:28:27.845970 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845077    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:28:27.845970 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845080    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:28:27.845970 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845082    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:28:27.845970 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845085    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:28:27.845970 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845087    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:28:27.845970 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845091    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:28:27.846475 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845096    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:28:27.846475 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845100    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:28:27.846475 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845103    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:28:27.846475 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845106    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:28:27.846475 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845109    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:28:27.846475 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845112    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:28:27.846475 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845115    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:28:27.846475 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845118    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:28:27.846475 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845121    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:28:27.846475 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845124    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:28:27.846475 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845126    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:28:27.846475 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845129    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:28:27.846475 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845132    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:28:27.846475 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845134    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:28:27.846475 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845137    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:28:27.846475 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845140    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:28:27.846475 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845142    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:28:27.846475 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845147    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:28:27.846475 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845149    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:28:27.846475 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845154    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:28:27.846976 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845157    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:28:27.846976 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845159    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:28:27.846976 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845162    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:28:27.846976 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845164    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:28:27.846976 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845168    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:28:27.846976 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845170    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:28:27.846976 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845173    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:28:27.846976 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845176    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:28:27.846976 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845179    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:28:27.846976 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845181    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:28:27.846976 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845186    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:28:27.846976 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845189    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:28:27.846976 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845192    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:28:27.846976 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845196    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:28:27.846976 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845199    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:28:27.846976 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845202    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:28:27.846976 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845205    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:28:27.846976 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845207    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:28:27.846976 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845962    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:28:27.847427 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845969    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:28:27.847427 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845972    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:28:27.847427 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845975    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:28:27.847427 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845978    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:28:27.847427 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845981    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:28:27.847427 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845984    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:28:27.847427 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845987    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:28:27.847427 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845990    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:28:27.847427 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845992    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:28:27.847427 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845995    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:28:27.847427 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.845998    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:28:27.847427 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846001    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:28:27.847427 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846003    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:28:27.847427 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846006    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:28:27.847427 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846009    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:28:27.847427 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846011    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:28:27.847427 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846015    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:28:27.847427 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846019    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:28:27.847427 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846031    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:28:27.847427 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846035    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:28:27.848080 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846038    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:28:27.848080 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846041    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:28:27.848080 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846043    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:28:27.848080 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846046    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:28:27.848080 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846049    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:28:27.848080 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846051    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:28:27.848080 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846054    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:28:27.848080 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846057    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:28:27.848080 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846059    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:28:27.848080 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846062    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:28:27.848080 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846065    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:28:27.848080 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846067    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:28:27.848080 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846070    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:28:27.848080 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846072    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:28:27.848080 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846075    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:28:27.848080 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846077    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:28:27.848080 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846079    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:28:27.848080 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846082    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:28:27.848080 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846085    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:28:27.848080 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846087    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:28:27.848080 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846090    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:28:27.849049 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846092    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:28:27.849049 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846095    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:28:27.849049 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846098    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:28:27.849049 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846101    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:28:27.849049 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846103    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:28:27.849049 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846107    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:28:27.849049 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846110    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:28:27.849049 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846112    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:28:27.849049 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846115    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:28:27.849049 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846118    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:28:27.849049 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846121    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:28:27.849049 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846124    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:28:27.849049 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846126    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:28:27.849049 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846129    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:28:27.849049 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846133    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:28:27.849049 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846136    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:28:27.849049 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846140    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:28:27.849049 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846143    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:28:27.849049 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846147    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:28:27.849562 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846151    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:28:27.849562 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846154    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:28:27.849562 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846157    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:28:27.849562 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846160    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:28:27.849562 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846163    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:28:27.849562 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846166    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:28:27.849562 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846169    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:28:27.849562 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846172    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:28:27.849562 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846174    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:28:27.849562 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846177    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:28:27.849562 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846179    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:28:27.849562 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846181    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:28:27.849562 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846184    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:28:27.849562 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846187    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:28:27.849562 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846190    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:28:27.849562 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846193    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:28:27.849562 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846195    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:28:27.849562 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846198    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:28:27.849562 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846201    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:28:27.850050 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846204    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:28:27.850050 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846206    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:28:27.850050 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846209    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:28:27.850050 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846212    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:28:27.850050 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846215    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:28:27.850050 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846218    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:28:27.850050 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846220    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:28:27.850050 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846222    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:28:27.850050 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846225    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:28:27.850050 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846228    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:28:27.850050 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846230    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:28:27.850050 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846233    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:28:27.850050 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846235    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:28:27.850050 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846238    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:28:27.850050 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846240    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:28:27.850050 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846242    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:28:27.850050 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846245    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:28:27.850050 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846248    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:28:27.850050 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846250    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:28:27.850520 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846669    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:28:27.850520 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846675    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:28:27.850520 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846678    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:28:27.850520 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846681    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:28:27.850520 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846684    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:28:27.850520 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846687    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:28:27.850520 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846689    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:28:27.850520 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846693    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:28:27.850520 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846695    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:28:27.850520 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846698    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:28:27.850520 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846700    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:28:27.850520 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846703    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:28:27.850520 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846706    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:28:27.850520 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846708    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:28:27.850520 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846711    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:28:27.850520 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846714    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:28:27.850520 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846716    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:28:27.850520 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846719    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:28:27.850520 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846722    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:28:27.850520 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846725    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:28:27.851037 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846728    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:28:27.851037 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846730    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:28:27.851037 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846734    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:28:27.851037 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846738    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:28:27.851037 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846741    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:28:27.851037 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846744    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:28:27.851037 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846747    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:28:27.851037 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846749    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:28:27.851037 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846766    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:28:27.851037 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846769    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:28:27.851037 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846771    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:28:27.851037 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846774    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:28:27.851037 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846788    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:28:27.851037 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846791    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:28:27.851037 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846794    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:28:27.851037 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846797    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:28:27.851037 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846799    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:28:27.851037 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846802    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:28:27.851037 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846804    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:28:27.851550 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846807    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:28:27.851550 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846810    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:28:27.851550 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846813    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:28:27.851550 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846815    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:28:27.851550 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846818    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:28:27.851550 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846821    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:28:27.851550 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846823    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:28:27.851550 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846826    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:28:27.851550 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846830    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:28:27.851550 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846832    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:28:27.851550 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846835    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:28:27.851550 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846838    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:28:27.851550 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846841    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:28:27.851550 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846843    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:28:27.851550 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846846    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:28:27.851550 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846848    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:28:27.851550 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846851    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:28:27.851550 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846854    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:28:27.851550 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846857    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:28:27.851550 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846859    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:28:27.852058 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846862    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:28:27.852058 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846864    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:28:27.852058 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846867    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:28:27.852058 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846869    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:28:27.852058 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846872    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:28:27.852058 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846874    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:28:27.852058 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846877    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:28:27.852058 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846879    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:28:27.852058 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846881    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:28:27.852058 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846884    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:28:27.852058 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846887    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:28:27.852058 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846889    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:28:27.852058 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846892    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:28:27.852058 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846895    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:28:27.852058 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846897    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:28:27.852058 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846900    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:28:27.852058 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846903    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:28:27.852058 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846907    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:28:27.852058 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846910    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:28:27.852547 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846914    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:28:27.852547 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846919    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:28:27.852547 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846922    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:28:27.852547 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846926    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:28:27.852547 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846931    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:28:27.852547 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846934    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:28:27.852547 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846937    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:28:27.852547 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846940    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:28:27.852547 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846943    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:28:27.852547 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846945    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:28:27.852547 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846948    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:28:27.852547 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846951    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:28:27.852547 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846953    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:28:27.852547 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846956    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:28:27.852547 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846958    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:28:27.852547 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846961    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:28:27.852547 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846963    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:28:27.852547 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846966    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:28:27.852547 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846968    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:28:27.853040 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846971    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:28:27.853040 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.846973    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:28:27.853040 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847042    2557 flags.go:64] FLAG: --address="0.0.0.0"
Jul 27 17:28:27.853040 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847049    2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 27 17:28:27.853040 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847055    2557 flags.go:64] FLAG: --anonymous-auth="true"
Jul 27 17:28:27.853040 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847059    2557 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 27 17:28:27.853040 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847064    2557 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 27 17:28:27.853040 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847068    2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 27 17:28:27.853040 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847072    2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 27 17:28:27.853040 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847077    2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 27 17:28:27.853040 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847080    2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 27 17:28:27.853040 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847083    2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 27 17:28:27.853040 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847087    2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 27 17:28:27.853040 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847090    2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 27 17:28:27.853040 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847095    2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 27 17:28:27.853040 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847098    2557 flags.go:64] FLAG: --cgroup-root=""
Jul 27 17:28:27.853040 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847101    2557 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 27 17:28:27.853040 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847104    2557 flags.go:64] FLAG: --client-ca-file=""
Jul 27 17:28:27.853040 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847107    2557 flags.go:64] FLAG: --cloud-provider="external"
Jul 27 17:28:27.853040 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847110    2557 flags.go:64] FLAG: --cluster-dns="[]"
Jul 27 17:28:27.853040 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847115    2557 flags.go:64] FLAG: --cluster-domain=""
Jul 27 17:28:27.853040 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847118    2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 27 17:28:27.853040 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847121    2557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 27 17:28:27.853612 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847124    2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 27 17:28:27.853612 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847128    2557 flags.go:64] FLAG: --container-log-max-files="5"
Jul 27 17:28:27.853612 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847132    2557 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 27 17:28:27.853612 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847135    2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 27 17:28:27.853612 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847138    2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 27 17:28:27.853612 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847141    2557 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 27 17:28:27.853612 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847144    2557 flags.go:64] FLAG: --contention-profiling="false"
Jul 27 17:28:27.853612 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847148    2557 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 27 17:28:27.853612 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847151    2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 27 17:28:27.853612 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847154    2557 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 27 17:28:27.853612 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847156    2557 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 27 17:28:27.853612 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847160    2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 27 17:28:27.853612 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847164    2557 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 27 17:28:27.853612 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847167    2557 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 27 17:28:27.853612 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847170    2557 flags.go:64] FLAG: --enable-load-reader="false"
Jul 27 17:28:27.853612 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847175    2557 flags.go:64] FLAG: --enable-server="true"
Jul 27 17:28:27.853612 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847178    2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 27 17:28:27.853612 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847182    2557 flags.go:64] FLAG: --event-burst="100"
Jul 27 17:28:27.853612 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847186    2557 flags.go:64] FLAG: --event-qps="50"
Jul 27 17:28:27.853612 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847189    2557 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 27 17:28:27.853612 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847192    2557 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 27 17:28:27.853612 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847195    2557 flags.go:64] FLAG: --eviction-hard=""
Jul 27 17:28:27.853612 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847199    2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 27 17:28:27.853612 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847203    2557 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 27 17:28:27.854205 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847206    2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 27 17:28:27.854205 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847209    2557 flags.go:64] FLAG: --eviction-soft=""
Jul 27 17:28:27.854205 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847212    2557 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 27 17:28:27.854205 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847215    2557 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 27 17:28:27.854205 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847218    2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 27 17:28:27.854205 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847221    2557 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 27 17:28:27.854205 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847224    2557 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 27 17:28:27.854205 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847227    2557 flags.go:64] FLAG: --fail-swap-on="true"
Jul 27 17:28:27.854205 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847230    2557 flags.go:64] FLAG: --feature-gates=""
Jul 27 17:28:27.854205 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847234    2557 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 27 17:28:27.854205 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847237    2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 27 17:28:27.854205 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847241    2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 27 17:28:27.854205 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847244    2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 27 17:28:27.854205 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847247    2557 flags.go:64] FLAG: --healthz-port="10248"
Jul 27 17:28:27.854205 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847250    2557 flags.go:64] FLAG: --help="false"
Jul 27 17:28:27.854205 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847253    2557 flags.go:64] FLAG: --hostname-override="ip-10-0-131-114.ec2.internal"
Jul 27 17:28:27.854205 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847256    2557 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 27 17:28:27.854205 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847259    2557 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 27 17:28:27.854205 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847261    2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 27 17:28:27.854205 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847265    2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 27 17:28:27.854205 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847268    2557 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 27 17:28:27.854205 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847271    2557 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 27 17:28:27.854205 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847274    2557 flags.go:64] FLAG: --image-service-endpoint=""
Jul 27 17:28:27.854820 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847278    2557 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 27 17:28:27.854820 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847281    2557 flags.go:64] FLAG: --kube-api-burst="100"
Jul 27 17:28:27.854820 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847283    2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 27 17:28:27.854820 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847287    2557 flags.go:64] FLAG: --kube-api-qps="50"
Jul 27 17:28:27.854820 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847290    2557 flags.go:64] FLAG: --kube-reserved=""
Jul 27 17:28:27.854820 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847293    2557 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 27 17:28:27.854820 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847295    2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 27 17:28:27.854820 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847298    2557 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 27 17:28:27.854820 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847302    2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 27 17:28:27.854820 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847305    2557 flags.go:64] FLAG: --lock-file=""
Jul 27 17:28:27.854820 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847308    2557 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 27 17:28:27.854820 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847311    2557 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 27 17:28:27.854820 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847314    2557 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 27 17:28:27.854820 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847319    2557 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 27 17:28:27.854820 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847322    2557 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 27 17:28:27.854820 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847325    2557 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 27 17:28:27.854820 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847328    2557 flags.go:64] FLAG: --logging-format="text"
Jul 27 17:28:27.854820 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847331    2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 27 17:28:27.854820 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847335    2557 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 27 17:28:27.854820 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847338    2557 flags.go:64] FLAG: --manifest-url=""
Jul 27 17:28:27.854820 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847341    2557 flags.go:64] FLAG: --manifest-url-header=""
Jul 27 17:28:27.854820 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847345    2557 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 27 17:28:27.854820 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847348    2557 flags.go:64] FLAG: --max-open-files="1000000"
Jul 27 17:28:27.854820 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847352    2557 flags.go:64] FLAG: --max-pods="110"
Jul 27 17:28:27.854820 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847356    2557 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 27 17:28:27.855462 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847358    2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 27 17:28:27.855462 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847361    2557 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 27 17:28:27.855462 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847365    2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 27 17:28:27.855462 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847368    2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 27 17:28:27.855462 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847370    2557 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 27 17:28:27.855462 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847373    2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 27 17:28:27.855462 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847381    2557 flags.go:64] FLAG: --node-status-max-images="50"
Jul 27 17:28:27.855462 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847385    2557 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 27 17:28:27.855462 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847388    2557 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 27 17:28:27.855462 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847391    2557 flags.go:64] FLAG: --pod-cidr=""
Jul 27 17:28:27.855462 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847394    2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 27 17:28:27.855462 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847400    2557 flags.go:64] FLAG: --pod-manifest-path=""
Jul 27 17:28:27.855462 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847402    2557 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 27 17:28:27.855462 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847405    2557 flags.go:64] FLAG: --pods-per-core="0"
Jul 27 17:28:27.855462 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847408    2557 flags.go:64] FLAG: --port="10250"
Jul 27 17:28:27.855462 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847411    2557 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 27 17:28:27.855462 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847416    2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f4e9c5d1b826592d"
Jul 27 17:28:27.855462 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847419    2557 flags.go:64] FLAG: --qos-reserved=""
Jul 27 17:28:27.855462 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847422    2557 flags.go:64] FLAG: --read-only-port="10255"
Jul 27 17:28:27.855462 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847425    2557 flags.go:64] FLAG: --register-node="true"
Jul 27 17:28:27.855462 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847428    2557 flags.go:64] FLAG: --register-with-taints=""
Jul 27 17:28:27.855462 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847431    2557 flags.go:64] FLAG: --registry-burst="10"
Jul 27 17:28:27.855462 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847434    2557 flags.go:64] FLAG: --registry-qps="5"
Jul 27 17:28:27.855462 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847437    2557 flags.go:64] FLAG: --reserved-cpus=""
Jul 27 17:28:27.856094 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847441    2557 flags.go:64] FLAG: --reserved-memory=""
Jul 27 17:28:27.856094 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847445    2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 27 17:28:27.856094 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847448    2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 27 17:28:27.856094 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847451    2557 flags.go:64] FLAG: --rotate-certificates="false"
Jul 27 17:28:27.856094 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847454    2557 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 27 17:28:27.856094 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847457    2557 flags.go:64] FLAG: --runonce="false"
Jul 27 17:28:27.856094 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847460    2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 27 17:28:27.856094 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847463    2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 27 17:28:27.856094 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847465    2557 flags.go:64] FLAG: --seccomp-default="false"
Jul 27 17:28:27.856094 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847468    2557 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 27 17:28:27.856094 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847471    2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 27 17:28:27.856094 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847474    2557 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 27 17:28:27.856094 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847477    2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 27 17:28:27.856094 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847480    2557 flags.go:64] FLAG: --storage-driver-password="root"
Jul 27 17:28:27.856094 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847483    2557 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 27 17:28:27.856094 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847487    2557 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 27 17:28:27.856094 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847490    2557 flags.go:64] FLAG: --storage-driver-user="root"
Jul 27 17:28:27.856094 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847493    2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 27 17:28:27.856094 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847496    2557 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 27 17:28:27.856094 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847499    2557 flags.go:64] FLAG: --system-cgroups=""
Jul 27 17:28:27.856094 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847502    2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 27 17:28:27.856094 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847507    2557 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 27 17:28:27.856094 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847509    2557 flags.go:64] FLAG: --tls-cert-file=""
Jul 27 17:28:27.856094 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847512    2557 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 27 17:28:27.856094 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847517    2557 flags.go:64] FLAG: --tls-min-version=""
Jul 27 17:28:27.856702 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847520    2557 flags.go:64] FLAG: --tls-private-key-file=""
Jul 27 17:28:27.856702 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847523    2557 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 27 17:28:27.856702 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847526    2557 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 27 17:28:27.856702 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847529    2557 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 27 17:28:27.856702 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847532    2557 flags.go:64] FLAG: --v="2"
Jul 27 17:28:27.856702 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847536    2557 flags.go:64] FLAG: --version="false"
Jul 27 17:28:27.856702 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847540    2557 flags.go:64] FLAG: --vmodule=""
Jul 27 17:28:27.856702 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847544    2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 27 17:28:27.856702 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.847598    2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 27 17:28:27.856702 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848558    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:28:27.856702 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848570    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:28:27.856702 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848575    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:28:27.856702 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848587    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:28:27.856702 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848595    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:28:27.856702 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848599    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:28:27.856702 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848604    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:28:27.856702 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848608    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:28:27.856702 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848613    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:28:27.856702 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848617    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:28:27.856702 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848621    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:28:27.856702 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848625    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:28:27.857248 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848630    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:28:27.857248 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848637    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:28:27.857248 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848641    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:28:27.857248 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848650    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:28:27.857248 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848654    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:28:27.857248 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848658    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:28:27.857248 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848663    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:28:27.857248 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848667    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:28:27.857248 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848671    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:28:27.857248 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848675    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:28:27.857248 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848691    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:28:27.857248 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848721    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:28:27.857248 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848729    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:28:27.857248 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848735    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:28:27.857248 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848740    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:28:27.857248 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848747    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:28:27.857248 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848768    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:28:27.857248 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848773    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:28:27.857248 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848778    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:28:27.857248 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848783    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:28:27.857737 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848787    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:28:27.857737 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848792    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:28:27.857737 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848795    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:28:27.857737 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848801    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:28:27.857737 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848804    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:28:27.857737 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848807    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:28:27.857737 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848810    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:28:27.857737 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848813    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:28:27.857737 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848816    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:28:27.857737 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848957    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:28:27.857737 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848962    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:28:27.857737 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848965    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:28:27.857737 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848968    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:28:27.857737 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848970    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:28:27.857737 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848975    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:28:27.857737 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848978    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:28:27.857737 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848980    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:28:27.857737 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848983    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:28:27.857737 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848986    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:28:27.858242 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848990    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:28:27.858242 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848995    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:28:27.858242 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.848997    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:28:27.858242 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849000    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:28:27.858242 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849004    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:28:27.858242 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849007    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:28:27.858242 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849009    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:28:27.858242 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849012    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:28:27.858242 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849014    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:28:27.858242 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849017    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:28:27.858242 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849020    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:28:27.858242 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849022    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:28:27.858242 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849025    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:28:27.858242 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849027    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:28:27.858242 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849030    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:28:27.858242 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849032    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:28:27.858242 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849035    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:28:27.858242 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849038    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:28:27.858242 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849040    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:28:27.858701 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849042    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:28:27.858701 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849045    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:28:27.858701 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849048    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:28:27.858701 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849052    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:28:27.858701 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849055    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:28:27.858701 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849057    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:28:27.858701 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849060    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:28:27.858701 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849063    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:28:27.858701 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849067    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:28:27.858701 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849070    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:28:27.858701 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849072    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:28:27.858701 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849075    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:28:27.858701 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849077    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:28:27.858701 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849080    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:28:27.858701 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849082    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:28:27.858701 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849085    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:28:27.858701 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849087    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:28:27.858701 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849092    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:28:27.858701 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849094    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:28:27.858701 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849097    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:28:27.859210 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849100    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:28:27.859210 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849102    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:28:27.859210 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849105    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:28:27.859210 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849107    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:28:27.859210 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849110    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:28:27.859210 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849112    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:28:27.859210 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849115    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:28:27.859210 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849118    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:28:27.859210 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.849121    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:28:27.859210 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.849828    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 17:28:27.859210 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.857278    2557 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 27 17:28:27.859210 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.857297    2557 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 27 17:28:27.859210 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.858182    2557 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 27 17:28:27.859210 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.858191    2557 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 27 17:28:27.859210 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858264    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:28:27.859603 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858270    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:28:27.859603 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858274    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:28:27.859603 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858277    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:28:27.859603 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858280    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:28:27.859603 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858283    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:28:27.859603 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858286    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:28:27.859603 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858289    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:28:27.859603 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858291    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:28:27.859603 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858294    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:28:27.859603 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858296    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:28:27.859603 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858299    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:28:27.859603 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858303    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:28:27.859603 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858305    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:28:27.859603 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858308    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:28:27.859603 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858310    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:28:27.859603 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858313    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:28:27.859603 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858316    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:28:27.859603 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858319    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:28:27.859603 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858322    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:28:27.859603 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858324    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:28:27.860109 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858327    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:28:27.860109 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858330    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:28:27.860109 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858332    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:28:27.860109 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858337    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:28:27.860109 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858341    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:28:27.860109 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858344    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:28:27.860109 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858347    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:28:27.860109 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858350    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:28:27.860109 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858353    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:28:27.860109 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858356    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:28:27.860109 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858358    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:28:27.860109 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858361    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:28:27.860109 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858364    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:28:27.860109 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858366    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:28:27.860109 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858369    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:28:27.860109 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858372    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:28:27.860109 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858375    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:28:27.860109 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858378    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:28:27.860109 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858380    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:28:27.860584 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858383    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:28:27.860584 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858386    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:28:27.860584 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858388    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:28:27.860584 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858391    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:28:27.860584 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858394    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:28:27.860584 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858396    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:28:27.860584 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858399    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:28:27.860584 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858401    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:28:27.860584 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858404    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:28:27.860584 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858407    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:28:27.860584 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858409    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:28:27.860584 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858412    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:28:27.860584 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858414    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:28:27.860584 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858417    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:28:27.860584 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858419    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:28:27.860584 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858422    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:28:27.860584 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858424    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:28:27.860584 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858427    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:28:27.860584 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858430    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:28:27.860584 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858432    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:28:27.861240 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858435    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:28:27.861240 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858456    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:28:27.861240 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858460    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:28:27.861240 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858463    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:28:27.861240 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858466    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:28:27.861240 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858468    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:28:27.861240 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858471    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:28:27.861240 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858474    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:28:27.861240 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858477    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:28:27.861240 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858479    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:28:27.861240 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858482    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:28:27.861240 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858485    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:28:27.861240 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858488    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:28:27.861240 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858490    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:28:27.861240 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858493    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:28:27.861240 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858495    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:28:27.861240 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858499    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:28:27.861240 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858502    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:28:27.861240 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858504    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:28:27.861240 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858507    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:28:27.861729 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858510    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:28:27.861729 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858514    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:28:27.861729 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858518    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:28:27.861729 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858521    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:28:27.861729 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858523    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:28:27.861729 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858526    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:28:27.861729 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858530    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:28:27.861729 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858533    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:28:27.861729 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858536    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:28:27.861729 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858539    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:28:27.861729 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858543    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:28:27.861729 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858546    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:28:27.861729 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858548    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:28:27.861729 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858552    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:28:27.861729 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858555    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:28:27.861729 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858558    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:28:27.861729 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858561    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:28:27.861729 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858563    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:28:27.861729 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858566    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:28:27.862207 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.858572    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 17:28:27.862207 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858694    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:28:27.862207 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858698    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:28:27.862207 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858701    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:28:27.862207 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858704    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:28:27.862207 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858707    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:28:27.862207 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858710    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:28:27.862207 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858713    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:28:27.862207 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858716    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:28:27.862207 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858718    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:28:27.862207 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858722    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:28:27.862207 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858725    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:28:27.862207 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858728    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:28:27.862207 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858731    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:28:27.862567 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858733    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:28:27.862567 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858736    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:28:27.862567 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858739    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:28:27.862567 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858741    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:28:27.862567 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858744    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:28:27.862567 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858746    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:28:27.862567 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858749    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:28:27.862567 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858767    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:28:27.862567 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858770    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:28:27.862567 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858773    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:28:27.862567 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858776    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:28:27.862567 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858779    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:28:27.862567 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858782    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:28:27.862567 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858785    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:28:27.862567 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858788    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:28:27.862567 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858791    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:28:27.862567 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858793    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:28:27.862567 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858796    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:28:27.862567 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858800    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:28:27.863035 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858804    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:28:27.863035 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858807    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:28:27.863035 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858810    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:28:27.863035 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858812    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:28:27.863035 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858815    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:28:27.863035 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858817    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:28:27.863035 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858820    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:28:27.863035 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858823    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:28:27.863035 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858825    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:28:27.863035 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858828    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:28:27.863035 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858832    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:28:27.863035 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858834    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:28:27.863035 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858837    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:28:27.863035 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858840    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:28:27.863035 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858842    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:28:27.863035 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858845    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:28:27.863035 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858847    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:28:27.863035 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858850    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:28:27.863035 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858853    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:28:27.863035 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858856    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:28:27.863538 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858858    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:28:27.863538 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858860    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:28:27.863538 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858863    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:28:27.863538 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858866    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:28:27.863538 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858869    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:28:27.863538 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858872    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:28:27.863538 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858875    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:28:27.863538 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858877    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:28:27.863538 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858880    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:28:27.863538 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858882    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:28:27.863538 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858885    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:28:27.863538 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858889    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:28:27.863538 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858892    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:28:27.863538 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858895    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:28:27.863538 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858898    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:28:27.863538 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858900    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:28:27.863538 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858903    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:28:27.863538 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858907    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:28:27.863538 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858911    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:28:27.864032 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858914    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:28:27.864032 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858916    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:28:27.864032 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858919    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:28:27.864032 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858921    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:28:27.864032 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858925    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:28:27.864032 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858927    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:28:27.864032 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858929    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:28:27.864032 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858932    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:28:27.864032 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858934    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:28:27.864032 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858937    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:28:27.864032 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858940    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:28:27.864032 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858942    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:28:27.864032 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858945    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:28:27.864032 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858947    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:28:27.864032 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858950    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:28:27.864032 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858952    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:28:27.864032 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858955    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:28:27.864032 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858958    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:28:27.864032 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858960    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:28:27.864032 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858963    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:28:27.864560 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858966    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:28:27.864560 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858968    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:28:27.864560 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858971    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:28:27.864560 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858973    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:28:27.864560 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858976    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:28:27.864560 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858978    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:28:27.864560 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858981    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:28:27.864560 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:27.858984    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:28:27.864560 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.858990    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 17:28:27.864560 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.859133    2557 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 27 17:28:27.864560 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.863490    2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 27 17:28:27.865531 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.865516    2557 server.go:1019] "Starting client certificate rotation"
Jul 27 17:28:27.865638 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.865624    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 17:28:27.865672 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.865664    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 17:28:27.868825 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.868800    2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 17:28:27.871347 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.871329    2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 17:28:27.881376 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.881351    2557 log.go:25] "Validated CRI v1 runtime API"
Jul 27 17:28:27.884324 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.884308    2557 log.go:25] "Validated CRI v1 image API"
Jul 27 17:28:27.885156 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.885140    2557 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 27 17:28:27.890179 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.890158    2557 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7dcead2b-5f80-41ed-8170-956c20f27c43:/dev/nvme0n1p3 eeaf4a46-cd8f-4128-86e6-e137ad509212:/dev/nvme0n1p4]
Jul 27 17:28:27.890258 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.890180    2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 27 17:28:27.897480 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.897350    2557 manager.go:216] Machine: {Timestamp:2026-07-27 17:28:27.895598952 +0000 UTC m=+0.382410232 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099865 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2db60fa21ee5262d17da25cf74c7c9 SystemUUID:ec2db60f-a21e-e526-2d17-da25cf74c7c9 BootID:1c77f363-78ee-4ad6-a408-5b11bbf129fa Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:25:8d:63:3c:57 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:25:8d:63:3c:57 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:39:d1:b8:3a:4d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 27 17:28:27.897480 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.897463    2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 27 17:28:27.897632 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.897555    2557 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 27 17:28:27.901264 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.901233    2557 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 27 17:28:27.901403 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.901265    2557 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-114.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 27 17:28:27.902097 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.902086    2557 topology_manager.go:138] "Creating topology manager with none policy"
Jul 27 17:28:27.902140 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.902100    2557 container_manager_linux.go:309] "Creating device plugin manager"
Jul 27 17:28:27.902140 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.902108    2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 17:28:27.902140 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.902123    2557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 17:28:27.903821 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.903811    2557 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 27 17:28:27.904330 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.904306    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 17:28:27.908472 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.908459    2557 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 17:28:27.908587 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.908578    2557 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 27 17:28:27.908633 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.908609    2557 kubelet.go:486] "Attempting to sync node with API server"
Jul 27 17:28:27.908633 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.908619    2557 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 27 17:28:27.908633 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.908630    2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 27 17:28:27.908746 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.908639    2557 kubelet.go:398] "Adding apiserver pod source"
Jul 27 17:28:27.908746 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.908649    2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 27 17:28:27.909106 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.909093    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 17:28:27.909146 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.909114    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 17:28:27.911710 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.911691    2557 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 27 17:28:27.913934 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.913918    2557 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 27 17:28:27.914015 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.913959    2557 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 27 17:28:27.914845 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.914831    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 27 17:28:27.914920 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.914853    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 27 17:28:27.914920 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.914864    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 27 17:28:27.914920 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.914872    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 27 17:28:27.914920 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.914885    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 27 17:28:27.914920 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.914893    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 27 17:28:27.914920 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.914904    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 27 17:28:27.914920 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.914912    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 27 17:28:27.914920 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.914922    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 27 17:28:27.915126 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.914930    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 27 17:28:27.915126 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.914943    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 27 17:28:27.915126 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.914952    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 27 17:28:27.915925 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.915912    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 27 17:28:27.915925 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.915926    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 27 17:28:27.918354 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:27.918325    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-114.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 27 17:28:27.918433 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.918386    2557 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-114.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 27 17:28:27.918433 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:27.918400    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 27 17:28:27.920069 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.920051    2557 server.go:1268] "Started kubelet"
Jul 27 17:28:27.920175 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.920153    2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 27 17:28:27.920548 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.920509    2557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 27 17:28:27.920640 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.920572    2557 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 27 17:28:27.921026 ip-10-0-131-114 systemd[1]: Started Kubernetes Kubelet.
Jul 27 17:28:27.921560 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.921539    2557 server.go:310] "Adding debug handlers to kubelet server"
Jul 27 17:28:27.922786 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.922770    2557 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 27 17:28:27.926436 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:27.924895    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-114.ec2.internal.18c635aa51116270  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-114.ec2.internal,UID:ip-10-0-131-114.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-114.ec2.internal,},FirstTimestamp:2026-07-27 17:28:27.919786608 +0000 UTC m=+0.406597888,LastTimestamp:2026-07-27 17:28:27.919786608 +0000 UTC m=+0.406597888,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-114.ec2.internal,}"
Jul 27 17:28:27.929075 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:27.929055    2557 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 27 17:28:27.933291 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.933277    2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 27 17:28:27.933362 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.933293    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 27 17:28:27.933473 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:27.933455    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-114.ec2.internal\" not found"
Jul 27 17:28:27.934206 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.934097    2557 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 27 17:28:27.934206 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.934114    2557 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 27 17:28:27.934206 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.934136    2557 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 27 17:28:27.934441 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.934213    2557 reconstruct.go:97] "Volume reconstruction finished"
Jul 27 17:28:27.934441 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.934220    2557 reconciler.go:29] "Reconciler: start to sync state"
Jul 27 17:28:27.935413 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.935396    2557 factory.go:55] Registering systemd factory
Jul 27 17:28:27.935500 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.935415    2557 factory.go:223] Registration of the systemd container factory successfully
Jul 27 17:28:27.935700 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.935682    2557 factory.go:153] Registering CRI-O factory
Jul 27 17:28:27.935700 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.935701    2557 factory.go:223] Registration of the crio container factory successfully
Jul 27 17:28:27.935842 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.935749    2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 27 17:28:27.935842 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.935788    2557 factory.go:103] Registering Raw factory
Jul 27 17:28:27.935842 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.935806    2557 manager.go:1196] Started watching for new ooms in manager
Jul 27 17:28:27.936271 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.936258    2557 manager.go:349] Starting recovery of all containers
Jul 27 17:28:27.947521 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:27.947494    2557 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-114.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 27 17:28:27.947623 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:27.947559    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 27 17:28:27.950029 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.950012    2557 manager.go:354] Recovery completed
Jul 27 17:28:27.950447 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.950425    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sx88g"
Jul 27 17:28:27.954137 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.954120    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:28:27.956745 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.956731    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-114.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:28:27.956867 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.956796    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-114.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:28:27.956867 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.956811    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-114.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:28:27.957468 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.957451    2557 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 27 17:28:27.957548 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.957468    2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 27 17:28:27.957548 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.957458    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:28:27.957548 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.957491    2557 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 17:28:27.958228 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.958201    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-114.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:28:27.958280 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.958232    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-114.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:28:27.958280 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.958244    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-114.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:28:27.958939 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.958920    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sx88g"
Jul 27 17:28:27.961294 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.961280    2557 policy_none.go:49] "None policy: Start"
Jul 27 17:28:27.961294 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.961296    2557 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 27 17:28:27.961463 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.961309    2557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 27 17:28:27.963259 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:27.963247    2557 policy_none.go:47] "Start"
Jul 27 17:28:27.965037 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:27.964935    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-114.ec2.internal.18c635aa53455725  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-114.ec2.internal,UID:ip-10-0-131-114.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-114.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-114.ec2.internal,},FirstTimestamp:2026-07-27 17:28:27.956746021 +0000 UTC m=+0.443557302,LastTimestamp:2026-07-27 17:28:27.956746021 +0000 UTC m=+0.443557302,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-114.ec2.internal,}"
Jul 27 17:28:28.005589 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.005388    2557 manager.go:337] "Starting Device Plugin manager"
Jul 27 17:28:28.005722 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:28.005611    2557 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 27 17:28:28.005722 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.005623    2557 server.go:87] "Starting device plugin registration server"
Jul 27 17:28:28.005895 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.005881    2557 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 27 17:28:28.005960 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.005897    2557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 27 17:28:28.006014 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.005996    2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 27 17:28:28.006111 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.006093    2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 27 17:28:28.006111 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.006108    2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 27 17:28:28.007806 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:28.007789    2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 27 17:28:28.007902 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:28.007847    2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-114.ec2.internal\" not found"
Jul 27 17:28:28.106328 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.106236    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:28:28.108508 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.107499    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-114.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:28:28.108508 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.107527    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-114.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:28:28.108508 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.107537    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-114.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:28:28.108508 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.107563    2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-114.ec2.internal"
Jul 27 17:28:28.110881 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.110847    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 27 17:28:28.112133 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.112118    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 27 17:28:28.112248 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.112161    2557 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 27 17:28:28.112248 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.112187    2557 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 27 17:28:28.112248 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:28.112225    2557 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 27 17:28:28.114622 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.114606    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:28:28.119347 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.119332    2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-114.ec2.internal"
Jul 27 17:28:28.119407 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:28.119358    2557 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-114.ec2.internal\": node \"ip-10-0-131-114.ec2.internal\" not found"
Jul 27 17:28:28.148824 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:28.148795    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-114.ec2.internal\" not found"
Jul 27 17:28:28.212960 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.212888    2557 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-114.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-114.ec2.internal"]
Jul 27 17:28:28.213055 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.213012    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:28:28.214440 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.214418    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-114.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:28:28.214558 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.214459    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-114.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:28:28.214558 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.214472    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-114.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:28:28.216807 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.216791    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:28:28.216909 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.216891    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-114.ec2.internal"
Jul 27 17:28:28.216963 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.216923    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:28:28.218316 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.218298    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-114.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:28:28.218316 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.218308    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-114.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:28:28.218430 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.218326    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-114.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:28:28.218430 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.218341    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-114.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:28:28.218430 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.218326    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-114.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:28:28.218430 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.218424    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-114.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:28:28.220783 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.220767    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-114.ec2.internal"
Jul 27 17:28:28.220870 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.220799    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:28:28.221571 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.221553    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-114.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:28:28.221668 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.221581    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-114.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:28:28.221668 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.221597    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-114.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:28:28.236120 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.236093    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5572cbb25a4673cdd3eb29fa42ae078c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-114.ec2.internal\" (UID: \"5572cbb25a4673cdd3eb29fa42ae078c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-114.ec2.internal"
Jul 27 17:28:28.236206 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.236125    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5572cbb25a4673cdd3eb29fa42ae078c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-114.ec2.internal\" (UID: \"5572cbb25a4673cdd3eb29fa42ae078c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-114.ec2.internal"
Jul 27 17:28:28.236206 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.236142    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14088cbd064fda87605e72bf989bc1ec-config\") pod \"kube-apiserver-proxy-ip-10-0-131-114.ec2.internal\" (UID: \"14088cbd064fda87605e72bf989bc1ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-114.ec2.internal"
Jul 27 17:28:28.237363 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:28.237346    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-114.ec2.internal\" not found" node="ip-10-0-131-114.ec2.internal"
Jul 27 17:28:28.241542 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:28.241528    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-114.ec2.internal\" not found" node="ip-10-0-131-114.ec2.internal"
Jul 27 17:28:28.249640 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:28.249625    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-114.ec2.internal\" not found"
Jul 27 17:28:28.336786 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.336740    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5572cbb25a4673cdd3eb29fa42ae078c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-114.ec2.internal\" (UID: \"5572cbb25a4673cdd3eb29fa42ae078c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-114.ec2.internal"
Jul 27 17:28:28.336889 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.336792    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5572cbb25a4673cdd3eb29fa42ae078c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-114.ec2.internal\" (UID: \"5572cbb25a4673cdd3eb29fa42ae078c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-114.ec2.internal"
Jul 27 17:28:28.336889 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.336814    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14088cbd064fda87605e72bf989bc1ec-config\") pod \"kube-apiserver-proxy-ip-10-0-131-114.ec2.internal\" (UID: \"14088cbd064fda87605e72bf989bc1ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-114.ec2.internal"
Jul 27 17:28:28.336889 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.336843    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5572cbb25a4673cdd3eb29fa42ae078c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-114.ec2.internal\" (UID: \"5572cbb25a4673cdd3eb29fa42ae078c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-114.ec2.internal"
Jul 27 17:28:28.336889 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.336863    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5572cbb25a4673cdd3eb29fa42ae078c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-114.ec2.internal\" (UID: \"5572cbb25a4673cdd3eb29fa42ae078c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-114.ec2.internal"
Jul 27 17:28:28.337017 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.336916    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14088cbd064fda87605e72bf989bc1ec-config\") pod \"kube-apiserver-proxy-ip-10-0-131-114.ec2.internal\" (UID: \"14088cbd064fda87605e72bf989bc1ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-114.ec2.internal"
Jul 27 17:28:28.349910 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:28.349879    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-114.ec2.internal\" not found"
Jul 27 17:28:28.450672 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:28.450637    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-114.ec2.internal\" not found"
Jul 27 17:28:28.539894 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.539863    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-114.ec2.internal"
Jul 27 17:28:28.545944 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.545717    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-114.ec2.internal"
Jul 27 17:28:28.551315 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:28.551291    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-114.ec2.internal\" not found"
Jul 27 17:28:28.652024 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:28.651975    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-114.ec2.internal\" not found"
Jul 27 17:28:28.752395 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:28.752301    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-114.ec2.internal\" not found"
Jul 27 17:28:28.853485 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:28.853442    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-114.ec2.internal\" not found"
Jul 27 17:28:28.864379 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.864354    2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 27 17:28:28.864547 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.864529    2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 17:28:28.954258 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:28.954228    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-114.ec2.internal\" not found"
Jul 27 17:28:28.966273 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.966236    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 17:23:28 +0000 UTC" deadline="2028-05-05 21:06:38.581089226 +0000 UTC"
Jul 27 17:28:28.966332 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:28.966272    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15555h38m9.614821523s"
Jul 27 17:28:29.054559 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:29.054472    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-114.ec2.internal\" not found"
Jul 27 17:28:29.108807 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.108774    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:28:29.109865 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.109847    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-114.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:28:29.109976 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.109879    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-114.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:28:29.109976 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.109890    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-114.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:28:29.155599 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:29.155564    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-114.ec2.internal\" not found"
Jul 27 17:28:29.256708 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:29.256672    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-114.ec2.internal\" not found"
Jul 27 17:28:29.357810 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:29.357730    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-114.ec2.internal\" not found"
Jul 27 17:28:29.407890 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.407860    2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:28:29.408055 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:29.408028    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14088cbd064fda87605e72bf989bc1ec.slice/crio-07b3e01cac0d7d0af03b9fc1fc5c947e39a2833249665b8fcdeb5ced553a0337 WatchSource:0}: Error finding container 07b3e01cac0d7d0af03b9fc1fc5c947e39a2833249665b8fcdeb5ced553a0337: Status 404 returned error can't find the container with id 07b3e01cac0d7d0af03b9fc1fc5c947e39a2833249665b8fcdeb5ced553a0337
Jul 27 17:28:29.412025 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.412007    2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:28:29.412519 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.412503    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:28:29.438993 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.438965    2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:28:29.534138 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.534077    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-114.ec2.internal"
Jul 27 17:28:29.546815 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.546784    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 17:28:29.547685 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.547657    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-114.ec2.internal"
Jul 27 17:28:29.555885 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.555863    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 17:28:29.630094 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:29.630061    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5572cbb25a4673cdd3eb29fa42ae078c.slice/crio-29c40acfa9446e4b65020782769a5d27bdc40ab750a096a202d6ee5cd84a5087 WatchSource:0}: Error finding container 29c40acfa9446e4b65020782769a5d27bdc40ab750a096a202d6ee5cd84a5087: Status 404 returned error can't find the container with id 29c40acfa9446e4b65020782769a5d27bdc40ab750a096a202d6ee5cd84a5087
Jul 27 17:28:29.910161 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.910065    2557 apiserver.go:52] "Watching apiserver"
Jul 27 17:28:29.915265 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.915236    2557 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 27 17:28:29.917189 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.917160    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-k56kz","openshift-image-registry/node-ca-q8nkb","openshift-multus/multus-tr5lq","openshift-network-diagnostics/network-check-target-4vhkj","openshift-network-operator/iptables-alerter-pzwrj","openshift-ovn-kubernetes/ovnkube-node-gwwvs","kube-system/global-pull-secret-syncer-zzcb2","kube-system/konnectivity-agent-xqqgl","kube-system/kube-apiserver-proxy-ip-10-0-131-114.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtrlp","openshift-cluster-node-tuning-operator/tuned-zgsmw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-114.ec2.internal","openshift-multus/multus-additional-cni-plugins-w7blt","openshift-multus/network-metrics-daemon-mzhnc"]
Jul 27 17:28:29.920099 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.920075    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:28:29.920228 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:29.920149    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzcb2" podUID="8eac848d-fc37-4bc7-b58d-54eaf719c28f"
Jul 27 17:28:29.922379 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.922357    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w7blt"
Jul 27 17:28:29.925625 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.925598    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 27 17:28:29.926030 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.926009    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 27 17:28:29.926157 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.926134    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 27 17:28:29.926236 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.926214    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 27 17:28:29.926364 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.926093    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 27 17:28:29.926466 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.926032    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rd5k7\""
Jul 27 17:28:29.927047 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.926675    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:29.928279 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.928260    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4vhkj"
Jul 27 17:28:29.928388 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:29.928334    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4vhkj" podUID="b96d56dc-19ef-41e7-a185-73926a19a744"
Jul 27 17:28:29.928581 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.928564    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 27 17:28:29.928967 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.928950    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hpmw9\""
Jul 27 17:28:29.930622 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.930600    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pzwrj"
Jul 27 17:28:29.932715 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.932687    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c7m4d\""
Jul 27 17:28:29.932861 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.932726    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 27 17:28:29.932861 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.932764    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 27 17:28:29.932861 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.932855    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 27 17:28:29.933088 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.933074    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:29.934566 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.934545    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 27 17:28:29.935240 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.935218    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 27 17:28:29.935379 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.935353    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2g5mv\""
Jul 27 17:28:29.935450 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.935414    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 27 17:28:29.935514 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.935428    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 27 17:28:29.935563 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.935544    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 27 17:28:29.935679 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.935655    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:28:29.935792 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:29.935753    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzhnc" podUID="57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec"
Jul 27 17:28:29.935892 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.935811    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 27 17:28:29.936028 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.936009    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 27 17:28:29.938138 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.938121    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k56kz"
Jul 27 17:28:29.940041 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.940026    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 27 17:28:29.940311 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.940290    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 27 17:28:29.940539 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.940525    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fljwx\""
Jul 27 17:28:29.940848 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.940825    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q8nkb"
Jul 27 17:28:29.942336 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.942318    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 27 17:28:29.942865 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.942834    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 27 17:28:29.942941 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.942880    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-25q8b\""
Jul 27 17:28:29.943323 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.943298    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 27 17:28:29.943574 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.943556    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtrlp"
Jul 27 17:28:29.945277 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.945260    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 17:28:29.945562 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.945543    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 27 17:28:29.945750 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.945726    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 27 17:28:29.945885 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.945867    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 27 17:28:29.946063 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.946044    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tcmmz\""
Jul 27 17:28:29.946136 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.946117    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:29.947880 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.947862    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-m94gx\""
Jul 27 17:28:29.948133 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.948120    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 27 17:28:29.948451 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.948433    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 27 17:28:29.948544 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.948460    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-systemd-units\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:29.948544 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.948494    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-etc-openvswitch\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:29.948544 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.948521    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8eac848d-fc37-4bc7-b58d-54eaf719c28f-kubelet-config\") pod \"global-pull-secret-syncer-zzcb2\" (UID: \"8eac848d-fc37-4bc7-b58d-54eaf719c28f\") " pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:28:29.948707 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.948548    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-var-lib-openvswitch\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:29.948707 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.948561    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xqqgl"
Jul 27 17:28:29.948707 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.948576    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-os-release\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:29.948707 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.948598    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-host-run-k8s-cni-cncf-io\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:29.948707 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.948613    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-multus-conf-dir\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:29.948707 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.948629    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqkmc\" (UniqueName: \"kubernetes.io/projected/6ea8ac48-a2a4-4f67-a740-89a5e4032966-kube-api-access-wqkmc\") pod \"iptables-alerter-pzwrj\" (UID: \"6ea8ac48-a2a4-4f67-a740-89a5e4032966\") " pod="openshift-network-operator/iptables-alerter-pzwrj"
Jul 27 17:28:29.948707 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.948649    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-host-cni-bin\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:29.948707 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.948674    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-cnibin\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:29.948707 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.948698    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-etc-kubernetes\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:29.949161 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.948722    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwzdd\" (UniqueName: \"kubernetes.io/projected/3ffc168b-79b8-46ef-8f17-636a697cb4e7-kube-api-access-kwzdd\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:29.949161 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.948749    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs\") pod \"network-metrics-daemon-mzhnc\" (UID: \"57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec\") " pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:28:29.949161 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.948769    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee935ed2-91cd-41fb-a3c3-fc5c52f42882-system-cni-dir\") pod \"multus-additional-cni-plugins-w7blt\" (UID: \"ee935ed2-91cd-41fb-a3c3-fc5c52f42882\") " pod="openshift-multus/multus-additional-cni-plugins-w7blt"
Jul 27 17:28:29.949161 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.948787    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-host-cni-netd\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:29.949161 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.948839    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3ffc168b-79b8-46ef-8f17-636a697cb4e7-multus-daemon-config\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:29.949161 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.948857    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8eac848d-fc37-4bc7-b58d-54eaf719c28f-original-pull-secret\") pod \"global-pull-secret-syncer-zzcb2\" (UID: \"8eac848d-fc37-4bc7-b58d-54eaf719c28f\") " pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:28:29.949161 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.948876    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-system-cni-dir\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:29.949161 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.948911    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-host-var-lib-cni-bin\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:29.949161 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.948939    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-host-kubelet\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:29.949161 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.948958    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-node-log\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:29.949161 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.948974    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-multus-socket-dir-parent\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:29.949161 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.948992    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-host-run-netns\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:29.949161 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.949006    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-host-var-lib-kubelet\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:29.949161 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.949046    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:29.949161 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.949065    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-ovnkube-config\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:29.949161 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.949082    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-env-overrides\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:29.949663 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.949108    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-host-run-ovn-kubernetes\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:29.949663 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.949126    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-multus-cni-dir\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:29.949663 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.949164    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qn4f\" (UniqueName: \"kubernetes.io/projected/ee935ed2-91cd-41fb-a3c3-fc5c52f42882-kube-api-access-8qn4f\") pod \"multus-additional-cni-plugins-w7blt\" (UID: \"ee935ed2-91cd-41fb-a3c3-fc5c52f42882\") " pod="openshift-multus/multus-additional-cni-plugins-w7blt"
Jul 27 17:28:29.949663 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.949185    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6ea8ac48-a2a4-4f67-a740-89a5e4032966-iptables-alerter-script\") pod \"iptables-alerter-pzwrj\" (UID: \"6ea8ac48-a2a4-4f67-a740-89a5e4032966\") " pod="openshift-network-operator/iptables-alerter-pzwrj"
Jul 27 17:28:29.949663 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.949202    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-run-systemd\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:29.949663 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.949222    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ffwd\" (UniqueName: \"kubernetes.io/projected/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-kube-api-access-4ffwd\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:29.949663 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.949246    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-host-var-lib-cni-multus\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:29.949663 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.949262    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee935ed2-91cd-41fb-a3c3-fc5c52f42882-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w7blt\" (UID: \"ee935ed2-91cd-41fb-a3c3-fc5c52f42882\") " pod="openshift-multus/multus-additional-cni-plugins-w7blt"
Jul 27 17:28:29.949663 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.949279    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ea8ac48-a2a4-4f67-a740-89a5e4032966-host-slash\") pod \"iptables-alerter-pzwrj\" (UID: \"6ea8ac48-a2a4-4f67-a740-89a5e4032966\") " pod="openshift-network-operator/iptables-alerter-pzwrj"
Jul 27 17:28:29.949663 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.949294    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-run-openvswitch\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:29.949663 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.949335    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-run-ovn\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:29.949663 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.949351    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-hostroot\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:29.949663 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.949366    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccfnx\" (UniqueName: \"kubernetes.io/projected/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-kube-api-access-ccfnx\") pod \"network-metrics-daemon-mzhnc\" (UID: \"57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec\") " pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:28:29.949663 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.949390    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ee935ed2-91cd-41fb-a3c3-fc5c52f42882-os-release\") pod \"multus-additional-cni-plugins-w7blt\" (UID: \"ee935ed2-91cd-41fb-a3c3-fc5c52f42882\") " pod="openshift-multus/multus-additional-cni-plugins-w7blt"
Jul 27 17:28:29.949663 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.949413    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee935ed2-91cd-41fb-a3c3-fc5c52f42882-cni-binary-copy\") pod \"multus-additional-cni-plugins-w7blt\" (UID: \"ee935ed2-91cd-41fb-a3c3-fc5c52f42882\") " pod="openshift-multus/multus-additional-cni-plugins-w7blt"
Jul 27 17:28:29.949663 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.949436    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee935ed2-91cd-41fb-a3c3-fc5c52f42882-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w7blt\" (UID: \"ee935ed2-91cd-41fb-a3c3-fc5c52f42882\") " pod="openshift-multus/multus-additional-cni-plugins-w7blt"
Jul 27 17:28:29.950299 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.949451    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-host-run-netns\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:29.950299 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.949473    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-ovnkube-script-lib\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:29.950299 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.949490    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8eac848d-fc37-4bc7-b58d-54eaf719c28f-dbus\") pod \"global-pull-secret-syncer-zzcb2\" (UID: \"8eac848d-fc37-4bc7-b58d-54eaf719c28f\") " pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:28:29.950299 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.949506    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-log-socket\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:29.950299 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.949522    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ffc168b-79b8-46ef-8f17-636a697cb4e7-cni-binary-copy\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:29.950299 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.949565    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee935ed2-91cd-41fb-a3c3-fc5c52f42882-cnibin\") pod \"multus-additional-cni-plugins-w7blt\" (UID: \"ee935ed2-91cd-41fb-a3c3-fc5c52f42882\") " pod="openshift-multus/multus-additional-cni-plugins-w7blt"
Jul 27 17:28:29.950299 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.949596    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee935ed2-91cd-41fb-a3c3-fc5c52f42882-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w7blt\" (UID: \"ee935ed2-91cd-41fb-a3c3-fc5c52f42882\") " pod="openshift-multus/multus-additional-cni-plugins-w7blt"
Jul 27 17:28:29.950299 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.949635    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhn7z\" (UniqueName: \"kubernetes.io/projected/b96d56dc-19ef-41e7-a185-73926a19a744-kube-api-access-zhn7z\") pod \"network-check-target-4vhkj\" (UID: \"b96d56dc-19ef-41e7-a185-73926a19a744\") " pod="openshift-network-diagnostics/network-check-target-4vhkj"
Jul 27 17:28:29.950299 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.949671    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-host-slash\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:29.950299 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.949699    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-ovn-node-metrics-cert\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:29.950299 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.949723    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-host-run-multus-certs\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:29.950705 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.950471    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-fnh6h\""
Jul 27 17:28:29.950793 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.950740    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 27 17:28:29.950850 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.950811    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 27 17:28:29.965882 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.965854    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wnxq8"
Jul 27 17:28:29.974539 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:29.974510    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wnxq8"
Jul 27 17:28:30.035113 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.035081    2557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 27 17:28:30.050849 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.050812    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ee935ed2-91cd-41fb-a3c3-fc5c52f42882-os-release\") pod \"multus-additional-cni-plugins-w7blt\" (UID: \"ee935ed2-91cd-41fb-a3c3-fc5c52f42882\") " pod="openshift-multus/multus-additional-cni-plugins-w7blt"
Jul 27 17:28:30.050849 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.050847    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee935ed2-91cd-41fb-a3c3-fc5c52f42882-cni-binary-copy\") pod \"multus-additional-cni-plugins-w7blt\" (UID: \"ee935ed2-91cd-41fb-a3c3-fc5c52f42882\") " pod="openshift-multus/multus-additional-cni-plugins-w7blt"
Jul 27 17:28:30.051097 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.050875    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee935ed2-91cd-41fb-a3c3-fc5c52f42882-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w7blt\" (UID: \"ee935ed2-91cd-41fb-a3c3-fc5c52f42882\") " pod="openshift-multus/multus-additional-cni-plugins-w7blt"
Jul 27 17:28:30.051188 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.051161    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ee935ed2-91cd-41fb-a3c3-fc5c52f42882-os-release\") pod \"multus-additional-cni-plugins-w7blt\" (UID: \"ee935ed2-91cd-41fb-a3c3-fc5c52f42882\") " pod="openshift-multus/multus-additional-cni-plugins-w7blt"
Jul 27 17:28:30.051243 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.051214    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17308a6d-e821-4aae-afe4-592c1f5f0175-tmp-dir\") pod \"node-resolver-k56kz\" (UID: \"17308a6d-e821-4aae-afe4-592c1f5f0175\") " pod="openshift-dns/node-resolver-k56kz"
Jul 27 17:28:30.051307 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.051262    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-host-run-netns\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.051307 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.051290    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-ovnkube-script-lib\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.051410 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.051319    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8eac848d-fc37-4bc7-b58d-54eaf719c28f-dbus\") pod \"global-pull-secret-syncer-zzcb2\" (UID: \"8eac848d-fc37-4bc7-b58d-54eaf719c28f\") " pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:28:30.051410 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.051335    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-host-run-netns\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.051410 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.051345    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-etc-kubernetes\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.051410 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.051369    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-run\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.051410 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.051395    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-lib-modules\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.051669 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.051422    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-log-socket\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.051669 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.051468    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ffc168b-79b8-46ef-8f17-636a697cb4e7-cni-binary-copy\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.051669 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.051498    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee935ed2-91cd-41fb-a3c3-fc5c52f42882-cnibin\") pod \"multus-additional-cni-plugins-w7blt\" (UID: \"ee935ed2-91cd-41fb-a3c3-fc5c52f42882\") " pod="openshift-multus/multus-additional-cni-plugins-w7blt"
Jul 27 17:28:30.051669 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.051522    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee935ed2-91cd-41fb-a3c3-fc5c52f42882-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w7blt\" (UID: \"ee935ed2-91cd-41fb-a3c3-fc5c52f42882\") " pod="openshift-multus/multus-additional-cni-plugins-w7blt"
Jul 27 17:28:30.051669 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.051543    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-log-socket\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.051669 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.051551    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6652a856-18e7-4544-8fd4-9c616f32e854-sys-fs\") pod \"aws-ebs-csi-driver-node-rtrlp\" (UID: \"6652a856-18e7-4544-8fd4-9c616f32e854\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtrlp"
Jul 27 17:28:30.051669 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.051591    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8eac848d-fc37-4bc7-b58d-54eaf719c28f-dbus\") pod \"global-pull-secret-syncer-zzcb2\" (UID: \"8eac848d-fc37-4bc7-b58d-54eaf719c28f\") " pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:28:30.051669 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.051596    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee935ed2-91cd-41fb-a3c3-fc5c52f42882-cni-binary-copy\") pod \"multus-additional-cni-plugins-w7blt\" (UID: \"ee935ed2-91cd-41fb-a3c3-fc5c52f42882\") " pod="openshift-multus/multus-additional-cni-plugins-w7blt"
Jul 27 17:28:30.052084 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.051625    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-host\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.052084 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.051696    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee935ed2-91cd-41fb-a3c3-fc5c52f42882-cnibin\") pod \"multus-additional-cni-plugins-w7blt\" (UID: \"ee935ed2-91cd-41fb-a3c3-fc5c52f42882\") " pod="openshift-multus/multus-additional-cni-plugins-w7blt"
Jul 27 17:28:30.052084 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.051720    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-tmp\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.052084 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.051757    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhn7z\" (UniqueName: \"kubernetes.io/projected/b96d56dc-19ef-41e7-a185-73926a19a744-kube-api-access-zhn7z\") pod \"network-check-target-4vhkj\" (UID: \"b96d56dc-19ef-41e7-a185-73926a19a744\") " pod="openshift-network-diagnostics/network-check-target-4vhkj"
Jul 27 17:28:30.052084 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.051803    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6652a856-18e7-4544-8fd4-9c616f32e854-socket-dir\") pod \"aws-ebs-csi-driver-node-rtrlp\" (UID: \"6652a856-18e7-4544-8fd4-9c616f32e854\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtrlp"
Jul 27 17:28:30.052084 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.051832    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-host-slash\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.052084 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.051858    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-ovn-node-metrics-cert\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.052084 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.051884    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-host-run-multus-certs\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.052084 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.051911    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-host-slash\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.052084 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.051926    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnvjv\" (UniqueName: \"kubernetes.io/projected/17308a6d-e821-4aae-afe4-592c1f5f0175-kube-api-access-nnvjv\") pod \"node-resolver-k56kz\" (UID: \"17308a6d-e821-4aae-afe4-592c1f5f0175\") " pod="openshift-dns/node-resolver-k56kz"
Jul 27 17:28:30.052084 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.051959    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-etc-modprobe-d\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.052084 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.051987    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-systemd-units\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.052084 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052015    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-etc-openvswitch\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.052084 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052042    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8eac848d-fc37-4bc7-b58d-54eaf719c28f-kubelet-config\") pod \"global-pull-secret-syncer-zzcb2\" (UID: \"8eac848d-fc37-4bc7-b58d-54eaf719c28f\") " pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:28:30.052084 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052061    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-ovnkube-script-lib\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.052084 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052090    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ffc168b-79b8-46ef-8f17-636a697cb4e7-cni-binary-copy\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.052855 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052068    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/231ae1f3-5af5-445f-92d1-2a8901275b9f-host\") pod \"node-ca-q8nkb\" (UID: \"231ae1f3-5af5-445f-92d1-2a8901275b9f\") " pod="openshift-image-registry/node-ca-q8nkb"
Jul 27 17:28:30.052855 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052132    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-var-lib-openvswitch\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.052855 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052150    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-systemd-units\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.052855 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052116    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-host-run-multus-certs\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.052855 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052162    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-os-release\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.052855 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052193    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-host-run-k8s-cni-cncf-io\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.052855 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052224    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-os-release\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.052855 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052236    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-etc-openvswitch\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.052855 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052269    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-host-run-k8s-cni-cncf-io\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.052855 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052225    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/17308a6d-e821-4aae-afe4-592c1f5f0175-hosts-file\") pod \"node-resolver-k56kz\" (UID: \"17308a6d-e821-4aae-afe4-592c1f5f0175\") " pod="openshift-dns/node-resolver-k56kz"
Jul 27 17:28:30.052855 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052308    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-var-lib-openvswitch\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.052855 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052313    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6652a856-18e7-4544-8fd4-9c616f32e854-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rtrlp\" (UID: \"6652a856-18e7-4544-8fd4-9c616f32e854\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtrlp"
Jul 27 17:28:30.052855 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052293    2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 27 17:28:30.052855 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052358    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6652a856-18e7-4544-8fd4-9c616f32e854-device-dir\") pod \"aws-ebs-csi-driver-node-rtrlp\" (UID: \"6652a856-18e7-4544-8fd4-9c616f32e854\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtrlp"
Jul 27 17:28:30.052855 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052363    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8eac848d-fc37-4bc7-b58d-54eaf719c28f-kubelet-config\") pod \"global-pull-secret-syncer-zzcb2\" (UID: \"8eac848d-fc37-4bc7-b58d-54eaf719c28f\") " pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:28:30.052855 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052396    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-multus-conf-dir\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.052855 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052420    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqkmc\" (UniqueName: \"kubernetes.io/projected/6ea8ac48-a2a4-4f67-a740-89a5e4032966-kube-api-access-wqkmc\") pod \"iptables-alerter-pzwrj\" (UID: \"6ea8ac48-a2a4-4f67-a740-89a5e4032966\") " pod="openshift-network-operator/iptables-alerter-pzwrj"
Jul 27 17:28:30.052855 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052437    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-host-cni-bin\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.053672 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052458    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-cnibin\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.053672 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052467    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-multus-conf-dir\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.053672 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052484    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-etc-kubernetes\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.053672 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052532    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-host-cni-bin\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.053672 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052553    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwzdd\" (UniqueName: \"kubernetes.io/projected/3ffc168b-79b8-46ef-8f17-636a697cb4e7-kube-api-access-kwzdd\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.053672 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052581    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs\") pod \"network-metrics-daemon-mzhnc\" (UID: \"57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec\") " pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:28:30.053672 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052601    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-cnibin\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.053672 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052626    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee935ed2-91cd-41fb-a3c3-fc5c52f42882-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w7blt\" (UID: \"ee935ed2-91cd-41fb-a3c3-fc5c52f42882\") " pod="openshift-multus/multus-additional-cni-plugins-w7blt"
Jul 27 17:28:30.053672 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052606    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee935ed2-91cd-41fb-a3c3-fc5c52f42882-system-cni-dir\") pod \"multus-additional-cni-plugins-w7blt\" (UID: \"ee935ed2-91cd-41fb-a3c3-fc5c52f42882\") " pod="openshift-multus/multus-additional-cni-plugins-w7blt"
Jul 27 17:28:30.053672 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052690    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-etc-kubernetes\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.053672 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052718    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/231ae1f3-5af5-445f-92d1-2a8901275b9f-serviceca\") pod \"node-ca-q8nkb\" (UID: \"231ae1f3-5af5-445f-92d1-2a8901275b9f\") " pod="openshift-image-registry/node-ca-q8nkb"
Jul 27 17:28:30.053672 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:30.052804    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:28:30.053672 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052815    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee935ed2-91cd-41fb-a3c3-fc5c52f42882-system-cni-dir\") pod \"multus-additional-cni-plugins-w7blt\" (UID: \"ee935ed2-91cd-41fb-a3c3-fc5c52f42882\") " pod="openshift-multus/multus-additional-cni-plugins-w7blt"
Jul 27 17:28:30.053672 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052856    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6652a856-18e7-4544-8fd4-9c616f32e854-etc-selinux\") pod \"aws-ebs-csi-driver-node-rtrlp\" (UID: \"6652a856-18e7-4544-8fd4-9c616f32e854\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtrlp"
Jul 27 17:28:30.053672 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:30.052904    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs podName:57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec nodeName:}" failed. No retries permitted until 2026-07-27 17:28:30.552865519 +0000 UTC m=+3.039676805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs") pod "network-metrics-daemon-mzhnc" (UID: "57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:28:30.053672 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.052983    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-host-cni-netd\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.053672 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053014    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3ffc168b-79b8-46ef-8f17-636a697cb4e7-multus-daemon-config\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.054561 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053039    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-host-cni-netd\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.054561 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053063    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltjmd\" (UniqueName: \"kubernetes.io/projected/231ae1f3-5af5-445f-92d1-2a8901275b9f-kube-api-access-ltjmd\") pod \"node-ca-q8nkb\" (UID: \"231ae1f3-5af5-445f-92d1-2a8901275b9f\") " pod="openshift-image-registry/node-ca-q8nkb"
Jul 27 17:28:30.054561 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053090    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-etc-sysctl-d\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.054561 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053140    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-etc-sysctl-conf\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.054561 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053223    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g426z\" (UniqueName: \"kubernetes.io/projected/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-kube-api-access-g426z\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.054561 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053279    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee935ed2-91cd-41fb-a3c3-fc5c52f42882-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w7blt\" (UID: \"ee935ed2-91cd-41fb-a3c3-fc5c52f42882\") " pod="openshift-multus/multus-additional-cni-plugins-w7blt"
Jul 27 17:28:30.054561 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053291    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9086a46f-a7bf-4e78-b049-1a1d363043a8-agent-certs\") pod \"konnectivity-agent-xqqgl\" (UID: \"9086a46f-a7bf-4e78-b049-1a1d363043a8\") " pod="kube-system/konnectivity-agent-xqqgl"
Jul 27 17:28:30.054561 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053339    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8eac848d-fc37-4bc7-b58d-54eaf719c28f-original-pull-secret\") pod \"global-pull-secret-syncer-zzcb2\" (UID: \"8eac848d-fc37-4bc7-b58d-54eaf719c28f\") " pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:28:30.054561 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053362    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-system-cni-dir\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.054561 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053378    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-host-var-lib-cni-bin\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.054561 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053437    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-host-kubelet\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.054561 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053463    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-node-log\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.054561 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:30.053472    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:28:30.054561 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053488    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-multus-socket-dir-parent\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.054561 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053515    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-host-run-netns\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.054561 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053516    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-host-var-lib-cni-bin\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.054561 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:30.053526    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eac848d-fc37-4bc7-b58d-54eaf719c28f-original-pull-secret podName:8eac848d-fc37-4bc7-b58d-54eaf719c28f nodeName:}" failed. No retries permitted until 2026-07-27 17:28:30.553511124 +0000 UTC m=+3.040322410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8eac848d-fc37-4bc7-b58d-54eaf719c28f-original-pull-secret") pod "global-pull-secret-syncer-zzcb2" (UID: "8eac848d-fc37-4bc7-b58d-54eaf719c28f") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:28:30.055481 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053562    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-node-log\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.055481 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053567    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3ffc168b-79b8-46ef-8f17-636a697cb4e7-multus-daemon-config\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.055481 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053571    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-system-cni-dir\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.055481 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053594    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-multus-socket-dir-parent\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.055481 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053603    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-host-var-lib-kubelet\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.055481 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053563    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-host-var-lib-kubelet\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.055481 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053613    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-host-run-netns\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.055481 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053523    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-host-kubelet\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.055481 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053641    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-968k8\" (UniqueName: \"kubernetes.io/projected/6652a856-18e7-4544-8fd4-9c616f32e854-kube-api-access-968k8\") pod \"aws-ebs-csi-driver-node-rtrlp\" (UID: \"6652a856-18e7-4544-8fd4-9c616f32e854\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtrlp"
Jul 27 17:28:30.055481 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053675    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.055481 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053702    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-ovnkube-config\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.055481 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053727    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-env-overrides\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.055481 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053752    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6652a856-18e7-4544-8fd4-9c616f32e854-registration-dir\") pod \"aws-ebs-csi-driver-node-rtrlp\" (UID: \"6652a856-18e7-4544-8fd4-9c616f32e854\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtrlp"
Jul 27 17:28:30.055481 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053783    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.055481 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053794    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-etc-systemd\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.055481 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053818    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-sys\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.055481 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053843    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9086a46f-a7bf-4e78-b049-1a1d363043a8-konnectivity-ca\") pod \"konnectivity-agent-xqqgl\" (UID: \"9086a46f-a7bf-4e78-b049-1a1d363043a8\") " pod="kube-system/konnectivity-agent-xqqgl"
Jul 27 17:28:30.056354 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053871    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-host-run-ovn-kubernetes\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.056354 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053896    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-multus-cni-dir\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.056354 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053922    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qn4f\" (UniqueName: \"kubernetes.io/projected/ee935ed2-91cd-41fb-a3c3-fc5c52f42882-kube-api-access-8qn4f\") pod \"multus-additional-cni-plugins-w7blt\" (UID: \"ee935ed2-91cd-41fb-a3c3-fc5c52f42882\") " pod="openshift-multus/multus-additional-cni-plugins-w7blt"
Jul 27 17:28:30.056354 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053945    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6ea8ac48-a2a4-4f67-a740-89a5e4032966-iptables-alerter-script\") pod \"iptables-alerter-pzwrj\" (UID: \"6ea8ac48-a2a4-4f67-a740-89a5e4032966\") " pod="openshift-network-operator/iptables-alerter-pzwrj"
Jul 27 17:28:30.056354 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053963    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-var-lib-kubelet\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.056354 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053975    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-multus-cni-dir\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.056354 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.053988    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-run-systemd\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.056354 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.054025    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ffwd\" (UniqueName: \"kubernetes.io/projected/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-kube-api-access-4ffwd\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.056354 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.054033    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-run-systemd\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.056354 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.054040    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-host-run-ovn-kubernetes\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.056354 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.054072    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-host-var-lib-cni-multus\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.056354 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.054094    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee935ed2-91cd-41fb-a3c3-fc5c52f42882-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w7blt\" (UID: \"ee935ed2-91cd-41fb-a3c3-fc5c52f42882\") " pod="openshift-multus/multus-additional-cni-plugins-w7blt"
Jul 27 17:28:30.056354 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.054112    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ea8ac48-a2a4-4f67-a740-89a5e4032966-host-slash\") pod \"iptables-alerter-pzwrj\" (UID: \"6ea8ac48-a2a4-4f67-a740-89a5e4032966\") " pod="openshift-network-operator/iptables-alerter-pzwrj"
Jul 27 17:28:30.056354 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.054141    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-etc-sysconfig\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.056354 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.054149    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-host-var-lib-cni-multus\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.056354 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.054166    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-etc-tuned\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.056354 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.054189    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ea8ac48-a2a4-4f67-a740-89a5e4032966-host-slash\") pod \"iptables-alerter-pzwrj\" (UID: \"6ea8ac48-a2a4-4f67-a740-89a5e4032966\") " pod="openshift-network-operator/iptables-alerter-pzwrj"
Jul 27 17:28:30.057146 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.054194    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-run-openvswitch\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.057146 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.054219    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-run-ovn\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.057146 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.054250    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-ovnkube-config\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.057146 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.054250    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-env-overrides\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.057146 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.054260    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-hostroot\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.057146 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.054303    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3ffc168b-79b8-46ef-8f17-636a697cb4e7-hostroot\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.057146 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.054338    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee935ed2-91cd-41fb-a3c3-fc5c52f42882-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w7blt\" (UID: \"ee935ed2-91cd-41fb-a3c3-fc5c52f42882\") " pod="openshift-multus/multus-additional-cni-plugins-w7blt"
Jul 27 17:28:30.057146 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.054346    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-run-ovn\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.057146 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.054378    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccfnx\" (UniqueName: \"kubernetes.io/projected/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-kube-api-access-ccfnx\") pod \"network-metrics-daemon-mzhnc\" (UID: \"57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec\") " pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:28:30.057146 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.054387    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-run-openvswitch\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.057146 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.054604    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6ea8ac48-a2a4-4f67-a740-89a5e4032966-iptables-alerter-script\") pod \"iptables-alerter-pzwrj\" (UID: \"6ea8ac48-a2a4-4f67-a740-89a5e4032966\") " pod="openshift-network-operator/iptables-alerter-pzwrj"
Jul 27 17:28:30.057146 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.057051    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-ovn-node-metrics-cert\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.059471 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:30.059425    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:28:30.059599 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:30.059477    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:28:30.059599 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:30.059493    2557 projected.go:196] Error preparing data for projected volume kube-api-access-zhn7z for pod openshift-network-diagnostics/network-check-target-4vhkj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:28:30.059599 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:30.059557    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b96d56dc-19ef-41e7-a185-73926a19a744-kube-api-access-zhn7z podName:b96d56dc-19ef-41e7-a185-73926a19a744 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:30.559540382 +0000 UTC m=+3.046351652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zhn7z" (UniqueName: "kubernetes.io/projected/b96d56dc-19ef-41e7-a185-73926a19a744-kube-api-access-zhn7z") pod "network-check-target-4vhkj" (UID: "b96d56dc-19ef-41e7-a185-73926a19a744") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:28:30.062428 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.062404    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqkmc\" (UniqueName: \"kubernetes.io/projected/6ea8ac48-a2a4-4f67-a740-89a5e4032966-kube-api-access-wqkmc\") pod \"iptables-alerter-pzwrj\" (UID: \"6ea8ac48-a2a4-4f67-a740-89a5e4032966\") " pod="openshift-network-operator/iptables-alerter-pzwrj"
Jul 27 17:28:30.063228 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.063203    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwzdd\" (UniqueName: \"kubernetes.io/projected/3ffc168b-79b8-46ef-8f17-636a697cb4e7-kube-api-access-kwzdd\") pod \"multus-tr5lq\" (UID: \"3ffc168b-79b8-46ef-8f17-636a697cb4e7\") " pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.063228 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.063223    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccfnx\" (UniqueName: \"kubernetes.io/projected/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-kube-api-access-ccfnx\") pod \"network-metrics-daemon-mzhnc\" (UID: \"57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec\") " pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:28:30.063369 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.063227    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qn4f\" (UniqueName: \"kubernetes.io/projected/ee935ed2-91cd-41fb-a3c3-fc5c52f42882-kube-api-access-8qn4f\") pod \"multus-additional-cni-plugins-w7blt\" (UID: \"ee935ed2-91cd-41fb-a3c3-fc5c52f42882\") " pod="openshift-multus/multus-additional-cni-plugins-w7blt"
Jul 27 17:28:30.063369 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.063288    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ffwd\" (UniqueName: \"kubernetes.io/projected/05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28-kube-api-access-4ffwd\") pod \"ovnkube-node-gwwvs\" (UID: \"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28\") " pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.116996 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.116941    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-114.ec2.internal" event={"ID":"5572cbb25a4673cdd3eb29fa42ae078c","Type":"ContainerStarted","Data":"29c40acfa9446e4b65020782769a5d27bdc40ab750a096a202d6ee5cd84a5087"}
Jul 27 17:28:30.118270 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.118227    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-114.ec2.internal" event={"ID":"14088cbd064fda87605e72bf989bc1ec","Type":"ContainerStarted","Data":"07b3e01cac0d7d0af03b9fc1fc5c947e39a2833249665b8fcdeb5ced553a0337"}
Jul 27 17:28:30.155611 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.155569    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-tmp\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.155833 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.155640    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6652a856-18e7-4544-8fd4-9c616f32e854-socket-dir\") pod \"aws-ebs-csi-driver-node-rtrlp\" (UID: \"6652a856-18e7-4544-8fd4-9c616f32e854\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtrlp"
Jul 27 17:28:30.155833 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.155672    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnvjv\" (UniqueName: \"kubernetes.io/projected/17308a6d-e821-4aae-afe4-592c1f5f0175-kube-api-access-nnvjv\") pod \"node-resolver-k56kz\" (UID: \"17308a6d-e821-4aae-afe4-592c1f5f0175\") " pod="openshift-dns/node-resolver-k56kz"
Jul 27 17:28:30.155833 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.155699    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-etc-modprobe-d\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.155833 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.155726    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/231ae1f3-5af5-445f-92d1-2a8901275b9f-host\") pod \"node-ca-q8nkb\" (UID: \"231ae1f3-5af5-445f-92d1-2a8901275b9f\") " pod="openshift-image-registry/node-ca-q8nkb"
Jul 27 17:28:30.155833 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.155753    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/17308a6d-e821-4aae-afe4-592c1f5f0175-hosts-file\") pod \"node-resolver-k56kz\" (UID: \"17308a6d-e821-4aae-afe4-592c1f5f0175\") " pod="openshift-dns/node-resolver-k56kz"
Jul 27 17:28:30.155833 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.155778    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6652a856-18e7-4544-8fd4-9c616f32e854-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rtrlp\" (UID: \"6652a856-18e7-4544-8fd4-9c616f32e854\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtrlp"
Jul 27 17:28:30.155833 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.155818    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6652a856-18e7-4544-8fd4-9c616f32e854-device-dir\") pod \"aws-ebs-csi-driver-node-rtrlp\" (UID: \"6652a856-18e7-4544-8fd4-9c616f32e854\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtrlp"
Jul 27 17:28:30.156186 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.155837    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/231ae1f3-5af5-445f-92d1-2a8901275b9f-host\") pod \"node-ca-q8nkb\" (UID: \"231ae1f3-5af5-445f-92d1-2a8901275b9f\") " pod="openshift-image-registry/node-ca-q8nkb"
Jul 27 17:28:30.156186 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.155863    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/231ae1f3-5af5-445f-92d1-2a8901275b9f-serviceca\") pod \"node-ca-q8nkb\" (UID: \"231ae1f3-5af5-445f-92d1-2a8901275b9f\") " pod="openshift-image-registry/node-ca-q8nkb"
Jul 27 17:28:30.156186 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.155862    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6652a856-18e7-4544-8fd4-9c616f32e854-socket-dir\") pod \"aws-ebs-csi-driver-node-rtrlp\" (UID: \"6652a856-18e7-4544-8fd4-9c616f32e854\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtrlp"
Jul 27 17:28:30.156186 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.155890    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6652a856-18e7-4544-8fd4-9c616f32e854-etc-selinux\") pod \"aws-ebs-csi-driver-node-rtrlp\" (UID: \"6652a856-18e7-4544-8fd4-9c616f32e854\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtrlp"
Jul 27 17:28:30.156186 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.155896    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-etc-modprobe-d\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.156186 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.155918    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltjmd\" (UniqueName: \"kubernetes.io/projected/231ae1f3-5af5-445f-92d1-2a8901275b9f-kube-api-access-ltjmd\") pod \"node-ca-q8nkb\" (UID: \"231ae1f3-5af5-445f-92d1-2a8901275b9f\") " pod="openshift-image-registry/node-ca-q8nkb"
Jul 27 17:28:30.156186 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.155930    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6652a856-18e7-4544-8fd4-9c616f32e854-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rtrlp\" (UID: \"6652a856-18e7-4544-8fd4-9c616f32e854\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtrlp"
Jul 27 17:28:30.156186 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.155943    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-etc-sysctl-d\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.156186 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.155959    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6652a856-18e7-4544-8fd4-9c616f32e854-device-dir\") pod \"aws-ebs-csi-driver-node-rtrlp\" (UID: \"6652a856-18e7-4544-8fd4-9c616f32e854\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtrlp"
Jul 27 17:28:30.156186 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.155969    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-etc-sysctl-conf\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.156186 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.155919    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/17308a6d-e821-4aae-afe4-592c1f5f0175-hosts-file\") pod \"node-resolver-k56kz\" (UID: \"17308a6d-e821-4aae-afe4-592c1f5f0175\") " pod="openshift-dns/node-resolver-k56kz"
Jul 27 17:28:30.156186 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.155997    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g426z\" (UniqueName: \"kubernetes.io/projected/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-kube-api-access-g426z\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.156186 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.156043    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9086a46f-a7bf-4e78-b049-1a1d363043a8-agent-certs\") pod \"konnectivity-agent-xqqgl\" (UID: \"9086a46f-a7bf-4e78-b049-1a1d363043a8\") " pod="kube-system/konnectivity-agent-xqqgl"
Jul 27 17:28:30.156186 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.156068    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6652a856-18e7-4544-8fd4-9c616f32e854-etc-selinux\") pod \"aws-ebs-csi-driver-node-rtrlp\" (UID: \"6652a856-18e7-4544-8fd4-9c616f32e854\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtrlp"
Jul 27 17:28:30.156186 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.156086    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-968k8\" (UniqueName: \"kubernetes.io/projected/6652a856-18e7-4544-8fd4-9c616f32e854-kube-api-access-968k8\") pod \"aws-ebs-csi-driver-node-rtrlp\" (UID: \"6652a856-18e7-4544-8fd4-9c616f32e854\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtrlp"
Jul 27 17:28:30.156186 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.156132    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6652a856-18e7-4544-8fd4-9c616f32e854-registration-dir\") pod \"aws-ebs-csi-driver-node-rtrlp\" (UID: \"6652a856-18e7-4544-8fd4-9c616f32e854\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtrlp"
Jul 27 17:28:30.156186 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.156154    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-etc-systemd\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.157136 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.156201    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-etc-sysctl-conf\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.157136 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.156211    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-etc-sysctl-d\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.157136 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.156222    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-etc-systemd\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.157136 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.156278    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6652a856-18e7-4544-8fd4-9c616f32e854-registration-dir\") pod \"aws-ebs-csi-driver-node-rtrlp\" (UID: \"6652a856-18e7-4544-8fd4-9c616f32e854\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtrlp"
Jul 27 17:28:30.157136 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.156293    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-sys\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.157136 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.156327    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9086a46f-a7bf-4e78-b049-1a1d363043a8-konnectivity-ca\") pod \"konnectivity-agent-xqqgl\" (UID: \"9086a46f-a7bf-4e78-b049-1a1d363043a8\") " pod="kube-system/konnectivity-agent-xqqgl"
Jul 27 17:28:30.157136 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.156359    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-var-lib-kubelet\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.157136 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.156363    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-sys\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.157136 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.156389    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-etc-sysconfig\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.157136 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.156390    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/231ae1f3-5af5-445f-92d1-2a8901275b9f-serviceca\") pod \"node-ca-q8nkb\" (UID: \"231ae1f3-5af5-445f-92d1-2a8901275b9f\") " pod="openshift-image-registry/node-ca-q8nkb"
Jul 27 17:28:30.157136 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.156407    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-etc-tuned\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.157136 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.156424    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-var-lib-kubelet\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.157136 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.156442    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17308a6d-e821-4aae-afe4-592c1f5f0175-tmp-dir\") pod \"node-resolver-k56kz\" (UID: \"17308a6d-e821-4aae-afe4-592c1f5f0175\") " pod="openshift-dns/node-resolver-k56kz"
Jul 27 17:28:30.157136 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.156469    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-etc-sysconfig\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.157136 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.156471    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-etc-kubernetes\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.157136 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.156519    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-etc-kubernetes\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.157136 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.156526    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-run\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.157136 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.156557    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-lib-modules\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.158077 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.156581    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-run\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.158077 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.156592    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6652a856-18e7-4544-8fd4-9c616f32e854-sys-fs\") pod \"aws-ebs-csi-driver-node-rtrlp\" (UID: \"6652a856-18e7-4544-8fd4-9c616f32e854\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtrlp"
Jul 27 17:28:30.158077 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.156617    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-host\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.158077 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.156687    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-lib-modules\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.158077 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.156699    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-host\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.158077 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.156754    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6652a856-18e7-4544-8fd4-9c616f32e854-sys-fs\") pod \"aws-ebs-csi-driver-node-rtrlp\" (UID: \"6652a856-18e7-4544-8fd4-9c616f32e854\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtrlp"
Jul 27 17:28:30.158077 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.156771    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17308a6d-e821-4aae-afe4-592c1f5f0175-tmp-dir\") pod \"node-resolver-k56kz\" (UID: \"17308a6d-e821-4aae-afe4-592c1f5f0175\") " pod="openshift-dns/node-resolver-k56kz"
Jul 27 17:28:30.158077 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.156980    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9086a46f-a7bf-4e78-b049-1a1d363043a8-konnectivity-ca\") pod \"konnectivity-agent-xqqgl\" (UID: \"9086a46f-a7bf-4e78-b049-1a1d363043a8\") " pod="kube-system/konnectivity-agent-xqqgl"
Jul 27 17:28:30.158996 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.158970    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-etc-tuned\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.159208 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.159188    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9086a46f-a7bf-4e78-b049-1a1d363043a8-agent-certs\") pod \"konnectivity-agent-xqqgl\" (UID: \"9086a46f-a7bf-4e78-b049-1a1d363043a8\") " pod="kube-system/konnectivity-agent-xqqgl"
Jul 27 17:28:30.159648 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.159623    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-tmp\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.163937 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.163817    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnvjv\" (UniqueName: \"kubernetes.io/projected/17308a6d-e821-4aae-afe4-592c1f5f0175-kube-api-access-nnvjv\") pod \"node-resolver-k56kz\" (UID: \"17308a6d-e821-4aae-afe4-592c1f5f0175\") " pod="openshift-dns/node-resolver-k56kz"
Jul 27 17:28:30.164095 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.164073    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g426z\" (UniqueName: \"kubernetes.io/projected/5cac4e21-c3de-42a6-9265-bd6fb7de3b42-kube-api-access-g426z\") pod \"tuned-zgsmw\" (UID: \"5cac4e21-c3de-42a6-9265-bd6fb7de3b42\") " pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.164462 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.164438    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltjmd\" (UniqueName: \"kubernetes.io/projected/231ae1f3-5af5-445f-92d1-2a8901275b9f-kube-api-access-ltjmd\") pod \"node-ca-q8nkb\" (UID: \"231ae1f3-5af5-445f-92d1-2a8901275b9f\") " pod="openshift-image-registry/node-ca-q8nkb"
Jul 27 17:28:30.165235 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.165214    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-968k8\" (UniqueName: \"kubernetes.io/projected/6652a856-18e7-4544-8fd4-9c616f32e854-kube-api-access-968k8\") pod \"aws-ebs-csi-driver-node-rtrlp\" (UID: \"6652a856-18e7-4544-8fd4-9c616f32e854\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtrlp"
Jul 27 17:28:30.235580 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.235541    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w7blt"
Jul 27 17:28:30.240810 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.240604    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tr5lq"
Jul 27 17:28:30.245148 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.245121    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:28:30.248478 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:30.248439    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee935ed2_91cd_41fb_a3c3_fc5c52f42882.slice/crio-be6010922f8c8c9a1dba34bc5d5f546ef90e8602c3d4ba5dc19dff77e010202e WatchSource:0}: Error finding container be6010922f8c8c9a1dba34bc5d5f546ef90e8602c3d4ba5dc19dff77e010202e: Status 404 returned error can't find the container with id be6010922f8c8c9a1dba34bc5d5f546ef90e8602c3d4ba5dc19dff77e010202e
Jul 27 17:28:30.250103 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.250080    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pzwrj"
Jul 27 17:28:30.251099 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:30.251071    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ffc168b_79b8_46ef_8f17_636a697cb4e7.slice/crio-48b700965375a56e7a149f5d5b11df9243df7c1195e21082eca3ddfa49a86f16 WatchSource:0}: Error finding container 48b700965375a56e7a149f5d5b11df9243df7c1195e21082eca3ddfa49a86f16: Status 404 returned error can't find the container with id 48b700965375a56e7a149f5d5b11df9243df7c1195e21082eca3ddfa49a86f16
Jul 27 17:28:30.256625 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.256603    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:30.262039 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.262015    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k56kz"
Jul 27 17:28:30.262412 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:30.262385    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea8ac48_a2a4_4f67_a740_89a5e4032966.slice/crio-ccf504871fd1b8a436b8da53acf11c12f97599d54b5a721ca66a575df7cbdc42 WatchSource:0}: Error finding container ccf504871fd1b8a436b8da53acf11c12f97599d54b5a721ca66a575df7cbdc42: Status 404 returned error can't find the container with id ccf504871fd1b8a436b8da53acf11c12f97599d54b5a721ca66a575df7cbdc42
Jul 27 17:28:30.268214 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.268018    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q8nkb"
Jul 27 17:28:30.268779 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:30.268747    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ef0d6a_aef6_4dd6_9b09_a7d86a3d8a28.slice/crio-3f62e5057425d459e728c7a0d2800a2dd2385b302ffb66e50426894b6dc453d2 WatchSource:0}: Error finding container 3f62e5057425d459e728c7a0d2800a2dd2385b302ffb66e50426894b6dc453d2: Status 404 returned error can't find the container with id 3f62e5057425d459e728c7a0d2800a2dd2385b302ffb66e50426894b6dc453d2
Jul 27 17:28:30.276259 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.276236    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtrlp"
Jul 27 17:28:30.276546 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:30.276515    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17308a6d_e821_4aae_afe4_592c1f5f0175.slice/crio-11e909769eb0089cb8890f1a03ce030539539fd08accf4c295ec63975a72320f WatchSource:0}: Error finding container 11e909769eb0089cb8890f1a03ce030539539fd08accf4c295ec63975a72320f: Status 404 returned error can't find the container with id 11e909769eb0089cb8890f1a03ce030539539fd08accf4c295ec63975a72320f
Jul 27 17:28:30.281093 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:30.281064    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod231ae1f3_5af5_445f_92d1_2a8901275b9f.slice/crio-101004bb41e1446f4e69f05cf0c85c6f816994c2dddb7d37ea8b678d068c4425 WatchSource:0}: Error finding container 101004bb41e1446f4e69f05cf0c85c6f816994c2dddb7d37ea8b678d068c4425: Status 404 returned error can't find the container with id 101004bb41e1446f4e69f05cf0c85c6f816994c2dddb7d37ea8b678d068c4425
Jul 27 17:28:30.282100 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.282012    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zgsmw"
Jul 27 17:28:30.287475 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.287415    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xqqgl"
Jul 27 17:28:30.287928 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:30.287886    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6652a856_18e7_4544_8fd4_9c616f32e854.slice/crio-e04d9ad29559a13232b05f293028e4fc2fd28d21fe40becbb98335e4b76d1536 WatchSource:0}: Error finding container e04d9ad29559a13232b05f293028e4fc2fd28d21fe40becbb98335e4b76d1536: Status 404 returned error can't find the container with id e04d9ad29559a13232b05f293028e4fc2fd28d21fe40becbb98335e4b76d1536
Jul 27 17:28:30.300975 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:30.300945    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cac4e21_c3de_42a6_9265_bd6fb7de3b42.slice/crio-988e6e14d2c8a30cc35e720dc0df82f5e1db1a876b52c90f3204d84f0604aae5 WatchSource:0}: Error finding container 988e6e14d2c8a30cc35e720dc0df82f5e1db1a876b52c90f3204d84f0604aae5: Status 404 returned error can't find the container with id 988e6e14d2c8a30cc35e720dc0df82f5e1db1a876b52c90f3204d84f0604aae5
Jul 27 17:28:30.305093 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:28:30.305063    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9086a46f_a7bf_4e78_b049_1a1d363043a8.slice/crio-eb294dcf25132b1abe3d12554a49f910bddd60101c3170006935e86f01e5e217 WatchSource:0}: Error finding container eb294dcf25132b1abe3d12554a49f910bddd60101c3170006935e86f01e5e217: Status 404 returned error can't find the container with id eb294dcf25132b1abe3d12554a49f910bddd60101c3170006935e86f01e5e217
Jul 27 17:28:30.560824 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.559793    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhn7z\" (UniqueName: \"kubernetes.io/projected/b96d56dc-19ef-41e7-a185-73926a19a744-kube-api-access-zhn7z\") pod \"network-check-target-4vhkj\" (UID: \"b96d56dc-19ef-41e7-a185-73926a19a744\") " pod="openshift-network-diagnostics/network-check-target-4vhkj"
Jul 27 17:28:30.560824 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.559871    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs\") pod \"network-metrics-daemon-mzhnc\" (UID: \"57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec\") " pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:28:30.560824 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.559908    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8eac848d-fc37-4bc7-b58d-54eaf719c28f-original-pull-secret\") pod \"global-pull-secret-syncer-zzcb2\" (UID: \"8eac848d-fc37-4bc7-b58d-54eaf719c28f\") " pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:28:30.560824 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:30.560059    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:28:30.560824 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:30.560120    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eac848d-fc37-4bc7-b58d-54eaf719c28f-original-pull-secret podName:8eac848d-fc37-4bc7-b58d-54eaf719c28f nodeName:}" failed. No retries permitted until 2026-07-27 17:28:31.560105871 +0000 UTC m=+4.046917138 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8eac848d-fc37-4bc7-b58d-54eaf719c28f-original-pull-secret") pod "global-pull-secret-syncer-zzcb2" (UID: "8eac848d-fc37-4bc7-b58d-54eaf719c28f") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:28:30.560824 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:30.560540    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:28:30.560824 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:30.560561    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:28:30.560824 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:30.560576    2557 projected.go:196] Error preparing data for projected volume kube-api-access-zhn7z for pod openshift-network-diagnostics/network-check-target-4vhkj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:28:30.560824 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:30.560630    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b96d56dc-19ef-41e7-a185-73926a19a744-kube-api-access-zhn7z podName:b96d56dc-19ef-41e7-a185-73926a19a744 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:31.560612935 +0000 UTC m=+4.047424204 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zhn7z" (UniqueName: "kubernetes.io/projected/b96d56dc-19ef-41e7-a185-73926a19a744-kube-api-access-zhn7z") pod "network-check-target-4vhkj" (UID: "b96d56dc-19ef-41e7-a185-73926a19a744") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:28:30.560824 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:30.560696    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:28:30.560824 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:30.560730    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs podName:57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec nodeName:}" failed. No retries permitted until 2026-07-27 17:28:31.560720956 +0000 UTC m=+4.047532223 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs") pod "network-metrics-daemon-mzhnc" (UID: "57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:28:30.978394 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.978330    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 17:23:30 +0000 UTC" deadline="2028-05-01 08:19:29.47050089 +0000 UTC"
Jul 27 17:28:30.978394 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:30.978398    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15446h50m58.492115733s"
Jul 27 17:28:31.113640 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:31.113044    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:28:31.113640 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:31.113197    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzcb2" podUID="8eac848d-fc37-4bc7-b58d-54eaf719c28f"
Jul 27 17:28:31.123963 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:31.123906    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pzwrj" event={"ID":"6ea8ac48-a2a4-4f67-a740-89a5e4032966","Type":"ContainerStarted","Data":"ccf504871fd1b8a436b8da53acf11c12f97599d54b5a721ca66a575df7cbdc42"}
Jul 27 17:28:31.127836 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:31.127799    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w7blt" event={"ID":"ee935ed2-91cd-41fb-a3c3-fc5c52f42882","Type":"ContainerStarted","Data":"be6010922f8c8c9a1dba34bc5d5f546ef90e8602c3d4ba5dc19dff77e010202e"}
Jul 27 17:28:31.130853 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:31.130814    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtrlp" event={"ID":"6652a856-18e7-4544-8fd4-9c616f32e854","Type":"ContainerStarted","Data":"e04d9ad29559a13232b05f293028e4fc2fd28d21fe40becbb98335e4b76d1536"}
Jul 27 17:28:31.135169 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:31.135134    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k56kz" event={"ID":"17308a6d-e821-4aae-afe4-592c1f5f0175","Type":"ContainerStarted","Data":"11e909769eb0089cb8890f1a03ce030539539fd08accf4c295ec63975a72320f"}
Jul 27 17:28:31.140219 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:31.140185    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tr5lq" event={"ID":"3ffc168b-79b8-46ef-8f17-636a697cb4e7","Type":"ContainerStarted","Data":"48b700965375a56e7a149f5d5b11df9243df7c1195e21082eca3ddfa49a86f16"}
Jul 27 17:28:31.142114 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:31.142085    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xqqgl" event={"ID":"9086a46f-a7bf-4e78-b049-1a1d363043a8","Type":"ContainerStarted","Data":"eb294dcf25132b1abe3d12554a49f910bddd60101c3170006935e86f01e5e217"}
Jul 27 17:28:31.151055 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:31.151019    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zgsmw" event={"ID":"5cac4e21-c3de-42a6-9265-bd6fb7de3b42","Type":"ContainerStarted","Data":"988e6e14d2c8a30cc35e720dc0df82f5e1db1a876b52c90f3204d84f0604aae5"}
Jul 27 17:28:31.153625 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:31.153593    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q8nkb" event={"ID":"231ae1f3-5af5-445f-92d1-2a8901275b9f","Type":"ContainerStarted","Data":"101004bb41e1446f4e69f05cf0c85c6f816994c2dddb7d37ea8b678d068c4425"}
Jul 27 17:28:31.157290 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:31.157261    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs" event={"ID":"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28","Type":"ContainerStarted","Data":"3f62e5057425d459e728c7a0d2800a2dd2385b302ffb66e50426894b6dc453d2"}
Jul 27 17:28:31.569501 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:31.569455    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhn7z\" (UniqueName: \"kubernetes.io/projected/b96d56dc-19ef-41e7-a185-73926a19a744-kube-api-access-zhn7z\") pod \"network-check-target-4vhkj\" (UID: \"b96d56dc-19ef-41e7-a185-73926a19a744\") " pod="openshift-network-diagnostics/network-check-target-4vhkj"
Jul 27 17:28:31.569719 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:31.569534    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs\") pod \"network-metrics-daemon-mzhnc\" (UID: \"57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec\") " pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:28:31.569719 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:31.569571    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8eac848d-fc37-4bc7-b58d-54eaf719c28f-original-pull-secret\") pod \"global-pull-secret-syncer-zzcb2\" (UID: \"8eac848d-fc37-4bc7-b58d-54eaf719c28f\") " pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:28:31.569875 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:31.569718    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:28:31.569875 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:31.569787    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eac848d-fc37-4bc7-b58d-54eaf719c28f-original-pull-secret podName:8eac848d-fc37-4bc7-b58d-54eaf719c28f nodeName:}" failed. No retries permitted until 2026-07-27 17:28:33.569767511 +0000 UTC m=+6.056578782 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8eac848d-fc37-4bc7-b58d-54eaf719c28f-original-pull-secret") pod "global-pull-secret-syncer-zzcb2" (UID: "8eac848d-fc37-4bc7-b58d-54eaf719c28f") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:28:31.570287 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:31.570261    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:28:31.570287 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:31.570291    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:28:31.570443 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:31.570304    2557 projected.go:196] Error preparing data for projected volume kube-api-access-zhn7z for pod openshift-network-diagnostics/network-check-target-4vhkj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:28:31.570443 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:31.570368    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b96d56dc-19ef-41e7-a185-73926a19a744-kube-api-access-zhn7z podName:b96d56dc-19ef-41e7-a185-73926a19a744 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:33.570337773 +0000 UTC m=+6.057149040 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zhn7z" (UniqueName: "kubernetes.io/projected/b96d56dc-19ef-41e7-a185-73926a19a744-kube-api-access-zhn7z") pod "network-check-target-4vhkj" (UID: "b96d56dc-19ef-41e7-a185-73926a19a744") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:28:31.570568 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:31.570451    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:28:31.570568 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:31.570484    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs podName:57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec nodeName:}" failed. No retries permitted until 2026-07-27 17:28:33.570473023 +0000 UTC m=+6.057284295 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs") pod "network-metrics-daemon-mzhnc" (UID: "57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:28:31.978669 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:31.978593    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 17:23:30 +0000 UTC" deadline="2028-02-02 06:49:11.073144476 +0000 UTC"
Jul 27 17:28:31.978669 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:31.978667    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13309h20m39.094482481s"
Jul 27 17:28:32.114969 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:32.114937    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4vhkj"
Jul 27 17:28:32.115180 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:32.115112    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4vhkj" podUID="b96d56dc-19ef-41e7-a185-73926a19a744"
Jul 27 17:28:32.115696 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:32.115666    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:28:32.115853 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:32.115814    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzhnc" podUID="57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec"
Jul 27 17:28:33.112822 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:33.112783    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:28:33.113266 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:33.112920    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzcb2" podUID="8eac848d-fc37-4bc7-b58d-54eaf719c28f"
Jul 27 17:28:33.591094 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:33.591054    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhn7z\" (UniqueName: \"kubernetes.io/projected/b96d56dc-19ef-41e7-a185-73926a19a744-kube-api-access-zhn7z\") pod \"network-check-target-4vhkj\" (UID: \"b96d56dc-19ef-41e7-a185-73926a19a744\") " pod="openshift-network-diagnostics/network-check-target-4vhkj"
Jul 27 17:28:33.591280 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:33.591125    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs\") pod \"network-metrics-daemon-mzhnc\" (UID: \"57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec\") " pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:28:33.591280 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:33.591158    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8eac848d-fc37-4bc7-b58d-54eaf719c28f-original-pull-secret\") pod \"global-pull-secret-syncer-zzcb2\" (UID: \"8eac848d-fc37-4bc7-b58d-54eaf719c28f\") " pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:28:33.591403 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:33.591298    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:28:33.591403 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:33.591361    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eac848d-fc37-4bc7-b58d-54eaf719c28f-original-pull-secret podName:8eac848d-fc37-4bc7-b58d-54eaf719c28f nodeName:}" failed. No retries permitted until 2026-07-27 17:28:37.591342775 +0000 UTC m=+10.078154044 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8eac848d-fc37-4bc7-b58d-54eaf719c28f-original-pull-secret") pod "global-pull-secret-syncer-zzcb2" (UID: "8eac848d-fc37-4bc7-b58d-54eaf719c28f") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:28:33.592147 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:33.591872    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:28:33.592147 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:33.591931    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs podName:57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec nodeName:}" failed. No retries permitted until 2026-07-27 17:28:37.591915276 +0000 UTC m=+10.078726544 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs") pod "network-metrics-daemon-mzhnc" (UID: "57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:28:33.592147 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:33.592014    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:28:33.592147 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:33.592034    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:28:33.592147 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:33.592048    2557 projected.go:196] Error preparing data for projected volume kube-api-access-zhn7z for pod openshift-network-diagnostics/network-check-target-4vhkj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:28:33.592147 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:33.592086    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b96d56dc-19ef-41e7-a185-73926a19a744-kube-api-access-zhn7z podName:b96d56dc-19ef-41e7-a185-73926a19a744 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:37.592072903 +0000 UTC m=+10.078884173 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zhn7z" (UniqueName: "kubernetes.io/projected/b96d56dc-19ef-41e7-a185-73926a19a744-kube-api-access-zhn7z") pod "network-check-target-4vhkj" (UID: "b96d56dc-19ef-41e7-a185-73926a19a744") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:28:34.113922 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:34.113610    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4vhkj"
Jul 27 17:28:34.113922 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:34.113745    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4vhkj" podUID="b96d56dc-19ef-41e7-a185-73926a19a744"
Jul 27 17:28:34.114477 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:34.114244    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:28:34.114477 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:34.114348    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzhnc" podUID="57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec"
Jul 27 17:28:35.113050 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:35.112522    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:28:35.113050 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:35.112664    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzcb2" podUID="8eac848d-fc37-4bc7-b58d-54eaf719c28f"
Jul 27 17:28:36.113885 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:36.113850    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4vhkj"
Jul 27 17:28:36.114321 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:36.113985    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4vhkj" podUID="b96d56dc-19ef-41e7-a185-73926a19a744"
Jul 27 17:28:36.114402 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:36.114373    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:28:36.114503 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:36.114482    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzhnc" podUID="57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec"
Jul 27 17:28:37.112829 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:37.112792    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:28:37.113015 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:37.112938    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzcb2" podUID="8eac848d-fc37-4bc7-b58d-54eaf719c28f"
Jul 27 17:28:37.623559 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:37.623470    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhn7z\" (UniqueName: \"kubernetes.io/projected/b96d56dc-19ef-41e7-a185-73926a19a744-kube-api-access-zhn7z\") pod \"network-check-target-4vhkj\" (UID: \"b96d56dc-19ef-41e7-a185-73926a19a744\") " pod="openshift-network-diagnostics/network-check-target-4vhkj"
Jul 27 17:28:37.623559 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:37.623546    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs\") pod \"network-metrics-daemon-mzhnc\" (UID: \"57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec\") " pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:28:37.624427 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:37.623580    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8eac848d-fc37-4bc7-b58d-54eaf719c28f-original-pull-secret\") pod \"global-pull-secret-syncer-zzcb2\" (UID: \"8eac848d-fc37-4bc7-b58d-54eaf719c28f\") " pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:28:37.624427 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:37.623715    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:28:37.624427 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:37.623798    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eac848d-fc37-4bc7-b58d-54eaf719c28f-original-pull-secret podName:8eac848d-fc37-4bc7-b58d-54eaf719c28f nodeName:}" failed. No retries permitted until 2026-07-27 17:28:45.62377884 +0000 UTC m=+18.110590106 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8eac848d-fc37-4bc7-b58d-54eaf719c28f-original-pull-secret") pod "global-pull-secret-syncer-zzcb2" (UID: "8eac848d-fc37-4bc7-b58d-54eaf719c28f") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:28:37.624427 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:37.623871    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:28:37.624427 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:37.623889    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:28:37.624427 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:37.623902    2557 projected.go:196] Error preparing data for projected volume kube-api-access-zhn7z for pod openshift-network-diagnostics/network-check-target-4vhkj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:28:37.624427 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:37.623945    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b96d56dc-19ef-41e7-a185-73926a19a744-kube-api-access-zhn7z podName:b96d56dc-19ef-41e7-a185-73926a19a744 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:45.623931676 +0000 UTC m=+18.110742945 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zhn7z" (UniqueName: "kubernetes.io/projected/b96d56dc-19ef-41e7-a185-73926a19a744-kube-api-access-zhn7z") pod "network-check-target-4vhkj" (UID: "b96d56dc-19ef-41e7-a185-73926a19a744") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:28:37.624427 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:37.624003    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:28:37.624427 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:37.624033    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs podName:57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec nodeName:}" failed. No retries permitted until 2026-07-27 17:28:45.624022642 +0000 UTC m=+18.110833928 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs") pod "network-metrics-daemon-mzhnc" (UID: "57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:28:38.114294 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:38.113627    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4vhkj"
Jul 27 17:28:38.114294 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:38.113744    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4vhkj" podUID="b96d56dc-19ef-41e7-a185-73926a19a744"
Jul 27 17:28:38.114294 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:38.114143    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:28:38.114294 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:38.114248    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzhnc" podUID="57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec"
Jul 27 17:28:39.113768 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:39.113042    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:28:39.113768 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:39.113409    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzcb2" podUID="8eac848d-fc37-4bc7-b58d-54eaf719c28f"
Jul 27 17:28:40.113033 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:40.112971    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4vhkj"
Jul 27 17:28:40.113222 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:40.113113    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4vhkj" podUID="b96d56dc-19ef-41e7-a185-73926a19a744"
Jul 27 17:28:40.113222 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:40.113190    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:28:40.113355 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:40.113323    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzhnc" podUID="57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec"
Jul 27 17:28:41.112526 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:41.112491    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:28:41.113009 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:41.112621    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzcb2" podUID="8eac848d-fc37-4bc7-b58d-54eaf719c28f"
Jul 27 17:28:42.112829 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:42.112787    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4vhkj"
Jul 27 17:28:42.112829 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:42.112816    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:28:42.113347 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:42.112919    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4vhkj" podUID="b96d56dc-19ef-41e7-a185-73926a19a744"
Jul 27 17:28:42.113347 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:42.113029    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzhnc" podUID="57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec"
Jul 27 17:28:43.112546 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:43.112495    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:28:43.112714 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:43.112624    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzcb2" podUID="8eac848d-fc37-4bc7-b58d-54eaf719c28f"
Jul 27 17:28:44.112480 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:44.112445    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4vhkj"
Jul 27 17:28:44.112992 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:44.112569    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4vhkj" podUID="b96d56dc-19ef-41e7-a185-73926a19a744"
Jul 27 17:28:44.112992 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:44.112627    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:28:44.112992 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:44.112779    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzhnc" podUID="57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec"
Jul 27 17:28:45.113451 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:45.113415    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:28:45.113911 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:45.113552    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzcb2" podUID="8eac848d-fc37-4bc7-b58d-54eaf719c28f"
Jul 27 17:28:45.682411 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:45.682373    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhn7z\" (UniqueName: \"kubernetes.io/projected/b96d56dc-19ef-41e7-a185-73926a19a744-kube-api-access-zhn7z\") pod \"network-check-target-4vhkj\" (UID: \"b96d56dc-19ef-41e7-a185-73926a19a744\") " pod="openshift-network-diagnostics/network-check-target-4vhkj"
Jul 27 17:28:45.682411 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:45.682427    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs\") pod \"network-metrics-daemon-mzhnc\" (UID: \"57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec\") " pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:28:45.682800 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:45.682451    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8eac848d-fc37-4bc7-b58d-54eaf719c28f-original-pull-secret\") pod \"global-pull-secret-syncer-zzcb2\" (UID: \"8eac848d-fc37-4bc7-b58d-54eaf719c28f\") " pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:28:45.682800 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:45.682576    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:28:45.682800 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:45.682611    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:28:45.682800 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:45.682625    2557 projected.go:196] Error preparing data for projected volume kube-api-access-zhn7z for pod openshift-network-diagnostics/network-check-target-4vhkj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:28:45.682800 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:45.682639    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:28:45.682800 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:45.682585    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:28:45.682800 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:45.682687    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b96d56dc-19ef-41e7-a185-73926a19a744-kube-api-access-zhn7z podName:b96d56dc-19ef-41e7-a185-73926a19a744 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:01.68266892 +0000 UTC m=+34.169480398 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zhn7z" (UniqueName: "kubernetes.io/projected/b96d56dc-19ef-41e7-a185-73926a19a744-kube-api-access-zhn7z") pod "network-check-target-4vhkj" (UID: "b96d56dc-19ef-41e7-a185-73926a19a744") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:28:45.682800 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:45.682705    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eac848d-fc37-4bc7-b58d-54eaf719c28f-original-pull-secret podName:8eac848d-fc37-4bc7-b58d-54eaf719c28f nodeName:}" failed. No retries permitted until 2026-07-27 17:29:01.682696532 +0000 UTC m=+34.169507797 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8eac848d-fc37-4bc7-b58d-54eaf719c28f-original-pull-secret") pod "global-pull-secret-syncer-zzcb2" (UID: "8eac848d-fc37-4bc7-b58d-54eaf719c28f") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:28:45.682800 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:45.682719    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs podName:57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec nodeName:}" failed. No retries permitted until 2026-07-27 17:29:01.682711321 +0000 UTC m=+34.169522590 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs") pod "network-metrics-daemon-mzhnc" (UID: "57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:28:46.112902 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:46.112738    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4vhkj"
Jul 27 17:28:46.112902 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:46.112741    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:28:46.112902 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:46.112863    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4vhkj" podUID="b96d56dc-19ef-41e7-a185-73926a19a744"
Jul 27 17:28:46.113173 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:46.112942    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzhnc" podUID="57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec"
Jul 27 17:28:47.112920 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:47.112885    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:28:47.113377 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:47.113009    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzcb2" podUID="8eac848d-fc37-4bc7-b58d-54eaf719c28f"
Jul 27 17:28:48.113109 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:48.113069    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4vhkj"
Jul 27 17:28:48.113541 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:48.113184    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4vhkj" podUID="b96d56dc-19ef-41e7-a185-73926a19a744"
Jul 27 17:28:48.113541 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:48.113231    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:28:48.113541 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:48.113314    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzhnc" podUID="57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec"
Jul 27 17:28:49.113310 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:49.113270    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:28:49.113675 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:49.113416    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzcb2" podUID="8eac848d-fc37-4bc7-b58d-54eaf719c28f"
Jul 27 17:28:50.113040 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:50.112868    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4vhkj"
Jul 27 17:28:50.113183 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:50.112871    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:28:50.113266 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:50.113244    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzhnc" podUID="57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec"
Jul 27 17:28:50.113311 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:50.113123    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4vhkj" podUID="b96d56dc-19ef-41e7-a185-73926a19a744"
Jul 27 17:28:50.199041 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:50.199015    2557 generic.go:356] "Generic (PLEG): container finished" podID="ee935ed2-91cd-41fb-a3c3-fc5c52f42882" containerID="d5c8e6bb784a1efe1a3d9ed8492cb34359f03d8c5a223293b17090a990c8431f" exitCode=0
Jul 27 17:28:50.199699 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:50.199095    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w7blt" event={"ID":"ee935ed2-91cd-41fb-a3c3-fc5c52f42882","Type":"ContainerDied","Data":"d5c8e6bb784a1efe1a3d9ed8492cb34359f03d8c5a223293b17090a990c8431f"}
Jul 27 17:28:50.203327 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:50.203305    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtrlp" event={"ID":"6652a856-18e7-4544-8fd4-9c616f32e854","Type":"ContainerStarted","Data":"43c36f4dda40f0823b3bb74c61a74c02dc1c7a19d937a1a6eda26d393acf995c"}
Jul 27 17:28:50.204814 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:50.204785    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k56kz" event={"ID":"17308a6d-e821-4aae-afe4-592c1f5f0175","Type":"ContainerStarted","Data":"964835b0cb576f565ce98abb49304370aeebf882f88191045ec157d707d288ac"}
Jul 27 17:28:50.206417 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:50.206393    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tr5lq" event={"ID":"3ffc168b-79b8-46ef-8f17-636a697cb4e7","Type":"ContainerStarted","Data":"f927bc9d584a8f8c8902837531d5dc104971c7624978c185f349b875be643fa2"}
Jul 27 17:28:50.207709 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:50.207687    2557 generic.go:356] "Generic (PLEG): container finished" podID="5572cbb25a4673cdd3eb29fa42ae078c" containerID="63b02a5e49bbdebeb33c53d141c6d033eac2aa8039ec9415a0e6281a371eebf4" exitCode=0
Jul 27 17:28:50.207810 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:50.207772    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-114.ec2.internal" event={"ID":"5572cbb25a4673cdd3eb29fa42ae078c","Type":"ContainerDied","Data":"63b02a5e49bbdebeb33c53d141c6d033eac2aa8039ec9415a0e6281a371eebf4"}
Jul 27 17:28:50.209053 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:50.209034    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-114.ec2.internal" event={"ID":"14088cbd064fda87605e72bf989bc1ec","Type":"ContainerStarted","Data":"6bac6c23b15f9bd562ccbf10f22e273702ed0f8d4118e6a04decf60232b5d3f5"}
Jul 27 17:28:50.210299 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:50.210278    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xqqgl" event={"ID":"9086a46f-a7bf-4e78-b049-1a1d363043a8","Type":"ContainerStarted","Data":"a1a7264614123453ed8a54ca8db364a07a0bf3a21355a4403e8b667425f7bcd0"}
Jul 27 17:28:50.211532 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:50.211512    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zgsmw" event={"ID":"5cac4e21-c3de-42a6-9265-bd6fb7de3b42","Type":"ContainerStarted","Data":"a3eedc2627807d4d98efc73500d29d18aa6841d327ce5a2a081145f848fb093b"}
Jul 27 17:28:50.212703 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:50.212686    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q8nkb" event={"ID":"231ae1f3-5af5-445f-92d1-2a8901275b9f","Type":"ContainerStarted","Data":"62bf67650f09d9236c0a648ff0bc3f4620f828e12fce02bf5942e3d298d486bb"}
Jul 27 17:28:50.214789 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:50.214750    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs" event={"ID":"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28","Type":"ContainerStarted","Data":"163564ad9a1db52b963b6a3532d8562f6f0168e714ffa9336b4a3c2ea3244d23"}
Jul 27 17:28:50.214854 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:50.214821    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs" event={"ID":"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28","Type":"ContainerStarted","Data":"c5f05eb4abb8a10bfc79d714e17c159138d31c78acf665ab53f8cb8384525b90"}
Jul 27 17:28:50.214854 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:50.214851    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs" event={"ID":"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28","Type":"ContainerStarted","Data":"1338bd54935ef2ac68195959aee19210fb76c6d398a99fd217b8b3e069e9aa40"}
Jul 27 17:28:50.214922 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:50.214880    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs" event={"ID":"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28","Type":"ContainerStarted","Data":"4cda50394c0be76ea61cc00fdc4bd7250a35628ce5adc066dda150461c1d0204"}
Jul 27 17:28:50.236743 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:50.236704    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-114.ec2.internal" podStartSLOduration=21.236687341 podStartE2EDuration="21.236687341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:50.236287377 +0000 UTC m=+22.723098665" watchObservedRunningTime="2026-07-27 17:28:50.236687341 +0000 UTC m=+22.723498628"
Jul 27 17:28:50.252294 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:50.252251    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-tr5lq" podStartSLOduration=3.025248486 podStartE2EDuration="22.252234267s" totalImagesPullingTime="19.226985781s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:50.251229259 +0000 UTC m=+22.738040547" watchObservedRunningTime="2026-07-27 17:28:50.252234267 +0000 UTC m=+22.739045554"
Jul 27 17:28:50.265372 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:50.265322    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-k56kz" podStartSLOduration=3.062826992 podStartE2EDuration="22.265305972s" totalImagesPullingTime="19.20247898s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:50.264869129 +0000 UTC m=+22.751680442" watchObservedRunningTime="2026-07-27 17:28:50.265305972 +0000 UTC m=+22.752117264"
Jul 27 17:28:50.277480 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:50.277443    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-q8nkb" podStartSLOduration=3.0823081549999998 podStartE2EDuration="22.277429855s" totalImagesPullingTime="19.1951217s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:50.276944256 +0000 UTC m=+22.763755544" watchObservedRunningTime="2026-07-27 17:28:50.277429855 +0000 UTC m=+22.764241142"
Jul 27 17:28:50.306410 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:50.306362    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zgsmw" podStartSLOduration=3.130954192 podStartE2EDuration="22.3063424s" totalImagesPullingTime="19.175388208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:50.306242809 +0000 UTC m=+22.793054114" watchObservedRunningTime="2026-07-27 17:28:50.3063424 +0000 UTC m=+22.793153688"
Jul 27 17:28:50.331721 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:50.331111    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xqqgl" podStartSLOduration=3.159862922 podStartE2EDuration="22.331089007s" totalImagesPullingTime="19.171226085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:50.330561813 +0000 UTC m=+22.817373129" watchObservedRunningTime="2026-07-27 17:28:50.331089007 +0000 UTC m=+22.817900294"
Jul 27 17:28:51.112694 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:51.112651    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:28:51.112894 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:51.112807    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzcb2" podUID="8eac848d-fc37-4bc7-b58d-54eaf719c28f"
Jul 27 17:28:51.220517 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:51.220491    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs" event={"ID":"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28","Type":"ContainerStarted","Data":"464008345df94986f4805dfd565e84aed95ad442db912c6c87f4bc35fea0e78d"}
Jul 27 17:28:51.220881 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:51.220565    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs" event={"ID":"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28","Type":"ContainerStarted","Data":"fda57217c3b54e1675c9854ee5be1629c03399084cb68d622ead90f6f8db1530"}
Jul 27 17:28:51.222226 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:51.222166    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pzwrj" event={"ID":"6ea8ac48-a2a4-4f67-a740-89a5e4032966","Type":"ContainerStarted","Data":"62c9586b80c4fc6efcf7d7ad4215d4621536983554e696d85b6b6ae209068215"}
Jul 27 17:28:51.224176 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:51.224124    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-114.ec2.internal" event={"ID":"5572cbb25a4673cdd3eb29fa42ae078c","Type":"ContainerStarted","Data":"455922ffd2607325be45f4b41f6535a564391bbc9735731993b55b396c4b1c42"}
Jul 27 17:28:51.236979 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:51.236937    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pzwrj" podStartSLOduration=4.022774782 podStartE2EDuration="23.236922922s" totalImagesPullingTime="19.21414814s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:51.23692084 +0000 UTC m=+23.723732149" watchObservedRunningTime="2026-07-27 17:28:51.236922922 +0000 UTC m=+23.723734209"
Jul 27 17:28:51.252835 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:51.252816    2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 27 17:28:51.253940 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:51.253900    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-114.ec2.internal" podStartSLOduration=22.253886287 podStartE2EDuration="22.253886287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:51.253517915 +0000 UTC m=+23.740329202" watchObservedRunningTime="2026-07-27 17:28:51.253886287 +0000 UTC m=+23.740697573"
Jul 27 17:28:52.022489 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:52.022371    2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T17:28:51.25283109Z","UUID":"685c08be-0d68-43d0-a29e-02fb9c861315","Handler":null,"Name":"","Endpoint":""}
Jul 27 17:28:52.024018 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:52.023991    2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 27 17:28:52.024145 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:52.024037    2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 27 17:28:52.113066 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:52.113032    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4vhkj"
Jul 27 17:28:52.113243 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:52.113151    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4vhkj" podUID="b96d56dc-19ef-41e7-a185-73926a19a744"
Jul 27 17:28:52.113243 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:52.113205    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:28:52.113371 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:52.113327    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzhnc" podUID="57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec"
Jul 27 17:28:52.229096 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:52.229062    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtrlp" event={"ID":"6652a856-18e7-4544-8fd4-9c616f32e854","Type":"ContainerStarted","Data":"3aa7fe62e4b840bd520243e5c2c78859fe24a9c80c4cc60146bb0e6dda23eaaa"}
Jul 27 17:28:53.112584 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:53.112558    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:28:53.112742 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:53.112653    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzcb2" podUID="8eac848d-fc37-4bc7-b58d-54eaf719c28f"
Jul 27 17:28:53.234927 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:53.234878    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs" event={"ID":"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28","Type":"ContainerStarted","Data":"fda4e56b87328086a4e6c8dc4bf25d340c5b1bd1c369989a8f60bbe93f72fa61"}
Jul 27 17:28:53.237215 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:53.237187    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtrlp" event={"ID":"6652a856-18e7-4544-8fd4-9c616f32e854","Type":"ContainerStarted","Data":"bd235e6a6f5bbc989b7696ab1cfc3148bcccf6714a709f728fda22a01215336d"}
Jul 27 17:28:53.253048 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:53.253008    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtrlp" podStartSLOduration=3.409837551 podStartE2EDuration="25.252993677s" totalImagesPullingTime="21.843156126s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:53.252619405 +0000 UTC m=+25.739430693" watchObservedRunningTime="2026-07-27 17:28:53.252993677 +0000 UTC m=+25.739805011"
Jul 27 17:28:54.113350 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:54.113131    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4vhkj"
Jul 27 17:28:54.113529 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:54.113176    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:28:54.113529 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:54.113458    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4vhkj" podUID="b96d56dc-19ef-41e7-a185-73926a19a744"
Jul 27 17:28:54.113648 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:54.113543    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzhnc" podUID="57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec"
Jul 27 17:28:55.113061 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:55.112862    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:28:55.113666 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:55.113094    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzcb2" podUID="8eac848d-fc37-4bc7-b58d-54eaf719c28f"
Jul 27 17:28:55.120336 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:55.120313    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xqqgl"
Jul 27 17:28:55.120799 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:55.120780    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xqqgl"
Jul 27 17:28:55.245601 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:55.245566    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs" event={"ID":"05ef0d6a-aef6-4dd6-9b09-a7d86a3d8a28","Type":"ContainerStarted","Data":"95c990d29c51ca532b4927b07720d8fb7053effc896f162f8af1db6166b3ee7c"}
Jul 27 17:28:55.245967 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:55.245947    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:55.247338 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:55.247316    2557 generic.go:356] "Generic (PLEG): container finished" podID="ee935ed2-91cd-41fb-a3c3-fc5c52f42882" containerID="c6cba499275445db2d8191760c1d8f1286d0ec48794a15310d74b4bbc087ee97" exitCode=0
Jul 27 17:28:55.247442 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:55.247396    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w7blt" event={"ID":"ee935ed2-91cd-41fb-a3c3-fc5c52f42882","Type":"ContainerDied","Data":"c6cba499275445db2d8191760c1d8f1286d0ec48794a15310d74b4bbc087ee97"}
Jul 27 17:28:55.262226 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:55.262210    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:55.272212 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:55.272172    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs" podStartSLOduration=7.931737238 podStartE2EDuration="27.272158041s" totalImagesPullingTime="19.340420803s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:55.271690131 +0000 UTC m=+27.758501417" watchObservedRunningTime="2026-07-27 17:28:55.272158041 +0000 UTC m=+27.758969327"
Jul 27 17:28:56.112586 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:56.112558    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4vhkj"
Jul 27 17:28:56.112779 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:56.112655    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4vhkj" podUID="b96d56dc-19ef-41e7-a185-73926a19a744"
Jul 27 17:28:56.112829 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:56.112774    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:28:56.112919 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:56.112897    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzhnc" podUID="57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec"
Jul 27 17:28:56.250571 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:56.250510    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:56.251070 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:56.250589    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:56.267790 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:56.267708    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:28:56.716325 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:56.716296    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mzhnc"]
Jul 27 17:28:56.716462 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:56.716398    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:28:56.716511 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:56.716493    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzhnc" podUID="57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec"
Jul 27 17:28:56.719368 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:56.719336    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4vhkj"]
Jul 27 17:28:56.719501 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:56.719427    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4vhkj"
Jul 27 17:28:56.719558 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:56.719519    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4vhkj" podUID="b96d56dc-19ef-41e7-a185-73926a19a744"
Jul 27 17:28:56.720203 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:56.720163    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zzcb2"]
Jul 27 17:28:56.720328 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:56.720269    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:28:56.720403 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:56.720363    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzcb2" podUID="8eac848d-fc37-4bc7-b58d-54eaf719c28f"
Jul 27 17:28:57.254498 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:57.254465    2557 generic.go:356] "Generic (PLEG): container finished" podID="ee935ed2-91cd-41fb-a3c3-fc5c52f42882" containerID="6dbe8a21e5c21e919a1b6df46d1bb84061695e3572e19dfdd1958c087f633868" exitCode=0
Jul 27 17:28:57.255037 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:57.254550    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w7blt" event={"ID":"ee935ed2-91cd-41fb-a3c3-fc5c52f42882","Type":"ContainerDied","Data":"6dbe8a21e5c21e919a1b6df46d1bb84061695e3572e19dfdd1958c087f633868"}
Jul 27 17:28:58.113263 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:58.113233    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4vhkj"
Jul 27 17:28:58.113450 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:58.113306    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:28:58.113450 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:58.113315    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4vhkj" podUID="b96d56dc-19ef-41e7-a185-73926a19a744"
Jul 27 17:28:58.113450 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:58.113402    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzhnc" podUID="57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec"
Jul 27 17:28:59.028395 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:59.028363    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xqqgl"
Jul 27 17:28:59.028823 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:59.028480    2557 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 27 17:28:59.028963 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:59.028945    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xqqgl"
Jul 27 17:28:59.112615 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:59.112545    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:28:59.112782 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:28:59.112654    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzcb2" podUID="8eac848d-fc37-4bc7-b58d-54eaf719c28f"
Jul 27 17:28:59.261993 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:59.261955    2557 generic.go:356] "Generic (PLEG): container finished" podID="ee935ed2-91cd-41fb-a3c3-fc5c52f42882" containerID="75a3a47d4aeb45604f5e934fad3ba1f3ab100e5df76114d6811b33e963f4b507" exitCode=0
Jul 27 17:28:59.262135 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:28:59.262029    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w7blt" event={"ID":"ee935ed2-91cd-41fb-a3c3-fc5c52f42882","Type":"ContainerDied","Data":"75a3a47d4aeb45604f5e934fad3ba1f3ab100e5df76114d6811b33e963f4b507"}
Jul 27 17:29:00.113186 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:00.113150    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4vhkj"
Jul 27 17:29:00.113648 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:00.113287    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4vhkj" podUID="b96d56dc-19ef-41e7-a185-73926a19a744"
Jul 27 17:29:00.113648 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:00.113347    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:29:00.113648 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:00.113482    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzhnc" podUID="57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec"
Jul 27 17:29:01.112991 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:01.112953    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:29:01.113146 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:01.113060    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zzcb2" podUID="8eac848d-fc37-4bc7-b58d-54eaf719c28f"
Jul 27 17:29:01.708996 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:01.708952    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhn7z\" (UniqueName: \"kubernetes.io/projected/b96d56dc-19ef-41e7-a185-73926a19a744-kube-api-access-zhn7z\") pod \"network-check-target-4vhkj\" (UID: \"b96d56dc-19ef-41e7-a185-73926a19a744\") " pod="openshift-network-diagnostics/network-check-target-4vhkj"
Jul 27 17:29:01.709434 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:01.709007    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs\") pod \"network-metrics-daemon-mzhnc\" (UID: \"57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec\") " pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:29:01.709434 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:01.709040    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8eac848d-fc37-4bc7-b58d-54eaf719c28f-original-pull-secret\") pod \"global-pull-secret-syncer-zzcb2\" (UID: \"8eac848d-fc37-4bc7-b58d-54eaf719c28f\") " pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:29:01.709434 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:01.709121    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:29:01.709434 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:01.709152    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:29:01.709434 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:01.709155    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:29:01.709434 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:01.709167    2557 projected.go:196] Error preparing data for projected volume kube-api-access-zhn7z for pod openshift-network-diagnostics/network-check-target-4vhkj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:29:01.709434 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:01.709222    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs podName:57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec nodeName:}" failed. No retries permitted until 2026-07-27 17:29:33.709203635 +0000 UTC m=+66.196014925 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs") pod "network-metrics-daemon-mzhnc" (UID: "57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:29:01.709434 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:01.709155    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:29:01.709434 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:01.709238    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b96d56dc-19ef-41e7-a185-73926a19a744-kube-api-access-zhn7z podName:b96d56dc-19ef-41e7-a185-73926a19a744 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:33.709231762 +0000 UTC m=+66.196043032 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zhn7z" (UniqueName: "kubernetes.io/projected/b96d56dc-19ef-41e7-a185-73926a19a744-kube-api-access-zhn7z") pod "network-check-target-4vhkj" (UID: "b96d56dc-19ef-41e7-a185-73926a19a744") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:29:01.709434 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:01.709305    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eac848d-fc37-4bc7-b58d-54eaf719c28f-original-pull-secret podName:8eac848d-fc37-4bc7-b58d-54eaf719c28f nodeName:}" failed. No retries permitted until 2026-07-27 17:29:33.709286205 +0000 UTC m=+66.196097488 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8eac848d-fc37-4bc7-b58d-54eaf719c28f-original-pull-secret") pod "global-pull-secret-syncer-zzcb2" (UID: "8eac848d-fc37-4bc7-b58d-54eaf719c28f") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:29:02.112921 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.112889    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4vhkj"
Jul 27 17:29:02.113102 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.112896    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:29:02.113102 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:02.113022    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4vhkj" podUID="b96d56dc-19ef-41e7-a185-73926a19a744"
Jul 27 17:29:02.113102 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:02.113083    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzhnc" podUID="57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec"
Jul 27 17:29:02.273246 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.273220    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-114.ec2.internal" event="NodeReady"
Jul 27 17:29:02.273500 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.273367    2557 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 27 17:29:02.314659 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.314580    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6b499f6dbb-gq96p"]
Jul 27 17:29:02.318364 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.318343    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:29:02.321386 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.321359    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-skq5m\""
Jul 27 17:29:02.321920 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.321660    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 27 17:29:02.321920 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.321782    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 27 17:29:02.321920 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.321795    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 27 17:29:02.328154 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.328131    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 27 17:29:02.331236 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.331216    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-k5llc"]
Jul 27 17:29:02.334143 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.334122    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-k5llc"
Jul 27 17:29:02.334433 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.334394    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-f7c9g"]
Jul 27 17:29:02.336680 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.336642    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-22498\""
Jul 27 17:29:02.336900 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.336879    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 27 17:29:02.336983 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.336881    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 27 17:29:02.337342 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.337323    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-f7c9g"
Jul 27 17:29:02.340766 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.340568    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6fr89\""
Jul 27 17:29:02.341520 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.341403    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 27 17:29:02.341520 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.341449    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 27 17:29:02.341653 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.341611    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 27 17:29:02.344396 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.344376    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b499f6dbb-gq96p"]
Jul 27 17:29:02.348343 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.348325    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-f7c9g"]
Jul 27 17:29:02.363165 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.363141    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-k5llc"]
Jul 27 17:29:02.413793 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.413729    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86ded04d-8b64-408e-8352-7ef795d848a0-installation-pull-secrets\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:29:02.413963 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.413849    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:29:02.414017 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.413967    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86ded04d-8b64-408e-8352-7ef795d848a0-image-registry-private-configuration\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:29:02.414017 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.414011    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt95m\" (UniqueName: \"kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-kube-api-access-xt95m\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:29:02.414104 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.414069    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86ded04d-8b64-408e-8352-7ef795d848a0-trusted-ca\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:29:02.414146 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.414124    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86ded04d-8b64-408e-8352-7ef795d848a0-ca-trust-extracted\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:29:02.414146 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.414140    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-bound-sa-token\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:29:02.414214 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.414169    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86ded04d-8b64-408e-8352-7ef795d848a0-registry-certificates\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:29:02.422605 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.422579    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pj6vq"]
Jul 27 17:29:02.425765 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.425735    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pj6vq"
Jul 27 17:29:02.427881 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.427858    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 27 17:29:02.427984 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.427888    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-x7wp7\""
Jul 27 17:29:02.427984 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.427867    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 27 17:29:02.434439 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.434384    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pj6vq"]
Jul 27 17:29:02.514639 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.514598    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d458fd4-bb2f-4907-9431-226cac065ce0-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-k5llc\" (UID: \"3d458fd4-bb2f-4907-9431-226cac065ce0\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-k5llc"
Jul 27 17:29:02.514836 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.514700    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86ded04d-8b64-408e-8352-7ef795d848a0-trusted-ca\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:29:02.514836 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.514766    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3d458fd4-bb2f-4907-9431-226cac065ce0-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-k5llc\" (UID: \"3d458fd4-bb2f-4907-9431-226cac065ce0\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-k5llc"
Jul 27 17:29:02.514836 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.514803    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c6446ac-f2c3-4897-a995-94176302c6b6-metrics-tls\") pod \"dns-default-pj6vq\" (UID: \"2c6446ac-f2c3-4897-a995-94176302c6b6\") " pod="openshift-dns/dns-default-pj6vq"
Jul 27 17:29:02.515055 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.514873    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86ded04d-8b64-408e-8352-7ef795d848a0-ca-trust-extracted\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:29:02.515055 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.514913    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-bound-sa-token\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:29:02.515055 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.514945    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkfvg\" (UniqueName: \"kubernetes.io/projected/2c6446ac-f2c3-4897-a995-94176302c6b6-kube-api-access-pkfvg\") pod \"dns-default-pj6vq\" (UID: \"2c6446ac-f2c3-4897-a995-94176302c6b6\") " pod="openshift-dns/dns-default-pj6vq"
Jul 27 17:29:02.515055 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.514981    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86ded04d-8b64-408e-8352-7ef795d848a0-registry-certificates\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:29:02.515055 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.515012    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f29005f-0e4f-4b5f-9e9a-8a2c3089fd08-cert\") pod \"ingress-canary-f7c9g\" (UID: \"5f29005f-0e4f-4b5f-9e9a-8a2c3089fd08\") " pod="openshift-ingress-canary/ingress-canary-f7c9g"
Jul 27 17:29:02.515055 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.515036    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86ded04d-8b64-408e-8352-7ef795d848a0-installation-pull-secrets\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:29:02.515315 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.515085    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2c6446ac-f2c3-4897-a995-94176302c6b6-config-volume\") pod \"dns-default-pj6vq\" (UID: \"2c6446ac-f2c3-4897-a995-94176302c6b6\") " pod="openshift-dns/dns-default-pj6vq"
Jul 27 17:29:02.515315 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.515146    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:29:02.515315 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.515179    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g27l4\" (UniqueName: \"kubernetes.io/projected/5f29005f-0e4f-4b5f-9e9a-8a2c3089fd08-kube-api-access-g27l4\") pod \"ingress-canary-f7c9g\" (UID: \"5f29005f-0e4f-4b5f-9e9a-8a2c3089fd08\") " pod="openshift-ingress-canary/ingress-canary-f7c9g"
Jul 27 17:29:02.515315 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.515217    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c6446ac-f2c3-4897-a995-94176302c6b6-tmp-dir\") pod \"dns-default-pj6vq\" (UID: \"2c6446ac-f2c3-4897-a995-94176302c6b6\") " pod="openshift-dns/dns-default-pj6vq"
Jul 27 17:29:02.515315 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.515258    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86ded04d-8b64-408e-8352-7ef795d848a0-image-registry-private-configuration\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:29:02.515315 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.515284    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86ded04d-8b64-408e-8352-7ef795d848a0-ca-trust-extracted\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:29:02.515315 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.515287    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt95m\" (UniqueName: \"kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-kube-api-access-xt95m\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:29:02.515638 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:02.515419    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:29:02.515638 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:02.515434    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b499f6dbb-gq96p: secret "image-registry-tls" not found
Jul 27 17:29:02.515638 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:02.515488    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls podName:86ded04d-8b64-408e-8352-7ef795d848a0 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:03.015468494 +0000 UTC m=+35.502279759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls") pod "image-registry-6b499f6dbb-gq96p" (UID: "86ded04d-8b64-408e-8352-7ef795d848a0") : secret "image-registry-tls" not found
Jul 27 17:29:02.515897 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.515872    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86ded04d-8b64-408e-8352-7ef795d848a0-trusted-ca\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:29:02.516331 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.516294    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86ded04d-8b64-408e-8352-7ef795d848a0-registry-certificates\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:29:02.520273 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.520249    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86ded04d-8b64-408e-8352-7ef795d848a0-image-registry-private-configuration\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:29:02.520365 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.520270    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86ded04d-8b64-408e-8352-7ef795d848a0-installation-pull-secrets\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:29:02.524067 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.523913    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-bound-sa-token\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:29:02.524167 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.524153    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt95m\" (UniqueName: \"kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-kube-api-access-xt95m\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:29:02.616341 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.616261    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkfvg\" (UniqueName: \"kubernetes.io/projected/2c6446ac-f2c3-4897-a995-94176302c6b6-kube-api-access-pkfvg\") pod \"dns-default-pj6vq\" (UID: \"2c6446ac-f2c3-4897-a995-94176302c6b6\") " pod="openshift-dns/dns-default-pj6vq"
Jul 27 17:29:02.616341 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.616303    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f29005f-0e4f-4b5f-9e9a-8a2c3089fd08-cert\") pod \"ingress-canary-f7c9g\" (UID: \"5f29005f-0e4f-4b5f-9e9a-8a2c3089fd08\") " pod="openshift-ingress-canary/ingress-canary-f7c9g"
Jul 27 17:29:02.616581 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.616345    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2c6446ac-f2c3-4897-a995-94176302c6b6-config-volume\") pod \"dns-default-pj6vq\" (UID: \"2c6446ac-f2c3-4897-a995-94176302c6b6\") " pod="openshift-dns/dns-default-pj6vq"
Jul 27 17:29:02.616581 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.616490    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g27l4\" (UniqueName: \"kubernetes.io/projected/5f29005f-0e4f-4b5f-9e9a-8a2c3089fd08-kube-api-access-g27l4\") pod \"ingress-canary-f7c9g\" (UID: \"5f29005f-0e4f-4b5f-9e9a-8a2c3089fd08\") " pod="openshift-ingress-canary/ingress-canary-f7c9g"
Jul 27 17:29:02.616581 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.616530    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c6446ac-f2c3-4897-a995-94176302c6b6-tmp-dir\") pod \"dns-default-pj6vq\" (UID: \"2c6446ac-f2c3-4897-a995-94176302c6b6\") " pod="openshift-dns/dns-default-pj6vq"
Jul 27 17:29:02.616581 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.616570    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d458fd4-bb2f-4907-9431-226cac065ce0-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-k5llc\" (UID: \"3d458fd4-bb2f-4907-9431-226cac065ce0\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-k5llc"
Jul 27 17:29:02.616833 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.616611    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3d458fd4-bb2f-4907-9431-226cac065ce0-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-k5llc\" (UID: \"3d458fd4-bb2f-4907-9431-226cac065ce0\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-k5llc"
Jul 27 17:29:02.616833 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.616640    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c6446ac-f2c3-4897-a995-94176302c6b6-metrics-tls\") pod \"dns-default-pj6vq\" (UID: \"2c6446ac-f2c3-4897-a995-94176302c6b6\") " pod="openshift-dns/dns-default-pj6vq"
Jul 27 17:29:02.616833 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:02.616708    2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 27 17:29:02.616833 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:02.616740    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:29:02.616833 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:02.616791    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d458fd4-bb2f-4907-9431-226cac065ce0-networking-console-plugin-cert podName:3d458fd4-bb2f-4907-9431-226cac065ce0 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:03.116772105 +0000 UTC m=+35.603583383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3d458fd4-bb2f-4907-9431-226cac065ce0-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-k5llc" (UID: "3d458fd4-bb2f-4907-9431-226cac065ce0") : secret "networking-console-plugin-cert" not found
Jul 27 17:29:02.616833 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:02.616814    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c6446ac-f2c3-4897-a995-94176302c6b6-metrics-tls podName:2c6446ac-f2c3-4897-a995-94176302c6b6 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:03.116802811 +0000 UTC m=+35.603614083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c6446ac-f2c3-4897-a995-94176302c6b6-metrics-tls") pod "dns-default-pj6vq" (UID: "2c6446ac-f2c3-4897-a995-94176302c6b6") : secret "dns-default-metrics-tls" not found
Jul 27 17:29:02.617138 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.616947    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2c6446ac-f2c3-4897-a995-94176302c6b6-config-volume\") pod \"dns-default-pj6vq\" (UID: \"2c6446ac-f2c3-4897-a995-94176302c6b6\") " pod="openshift-dns/dns-default-pj6vq"
Jul 27 17:29:02.617138 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.617071    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c6446ac-f2c3-4897-a995-94176302c6b6-tmp-dir\") pod \"dns-default-pj6vq\" (UID: \"2c6446ac-f2c3-4897-a995-94176302c6b6\") " pod="openshift-dns/dns-default-pj6vq"
Jul 27 17:29:02.617485 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.617462    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3d458fd4-bb2f-4907-9431-226cac065ce0-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-k5llc\" (UID: \"3d458fd4-bb2f-4907-9431-226cac065ce0\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-k5llc"
Jul 27 17:29:02.619050 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.619031    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f29005f-0e4f-4b5f-9e9a-8a2c3089fd08-cert\") pod \"ingress-canary-f7c9g\" (UID: \"5f29005f-0e4f-4b5f-9e9a-8a2c3089fd08\") " pod="openshift-ingress-canary/ingress-canary-f7c9g"
Jul 27 17:29:02.625326 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.625298    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkfvg\" (UniqueName: \"kubernetes.io/projected/2c6446ac-f2c3-4897-a995-94176302c6b6-kube-api-access-pkfvg\") pod \"dns-default-pj6vq\" (UID: \"2c6446ac-f2c3-4897-a995-94176302c6b6\") " pod="openshift-dns/dns-default-pj6vq"
Jul 27 17:29:02.625551 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.625534    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g27l4\" (UniqueName: \"kubernetes.io/projected/5f29005f-0e4f-4b5f-9e9a-8a2c3089fd08-kube-api-access-g27l4\") pod \"ingress-canary-f7c9g\" (UID: \"5f29005f-0e4f-4b5f-9e9a-8a2c3089fd08\") " pod="openshift-ingress-canary/ingress-canary-f7c9g"
Jul 27 17:29:02.652594 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:02.652563    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-f7c9g"
Jul 27 17:29:03.018885 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:03.018850    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:29:03.019425 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:03.018973    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:29:03.019425 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:03.018985    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b499f6dbb-gq96p: secret "image-registry-tls" not found
Jul 27 17:29:03.019425 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:03.019037    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls podName:86ded04d-8b64-408e-8352-7ef795d848a0 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:04.019022215 +0000 UTC m=+36.505833480 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls") pod "image-registry-6b499f6dbb-gq96p" (UID: "86ded04d-8b64-408e-8352-7ef795d848a0") : secret "image-registry-tls" not found
Jul 27 17:29:03.112648 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:03.112616    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:29:03.114941 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:03.114918    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 17:29:03.119401 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:03.119293    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d458fd4-bb2f-4907-9431-226cac065ce0-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-k5llc\" (UID: \"3d458fd4-bb2f-4907-9431-226cac065ce0\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-k5llc"
Jul 27 17:29:03.119401 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:03.119349    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c6446ac-f2c3-4897-a995-94176302c6b6-metrics-tls\") pod \"dns-default-pj6vq\" (UID: \"2c6446ac-f2c3-4897-a995-94176302c6b6\") " pod="openshift-dns/dns-default-pj6vq"
Jul 27 17:29:03.119572 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:03.119419    2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 27 17:29:03.119572 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:03.119468    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:29:03.119572 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:03.119488    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d458fd4-bb2f-4907-9431-226cac065ce0-networking-console-plugin-cert podName:3d458fd4-bb2f-4907-9431-226cac065ce0 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:04.119468611 +0000 UTC m=+36.606279877 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3d458fd4-bb2f-4907-9431-226cac065ce0-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-k5llc" (UID: "3d458fd4-bb2f-4907-9431-226cac065ce0") : secret "networking-console-plugin-cert" not found
Jul 27 17:29:03.119572 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:03.119524    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c6446ac-f2c3-4897-a995-94176302c6b6-metrics-tls podName:2c6446ac-f2c3-4897-a995-94176302c6b6 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:04.119507201 +0000 UTC m=+36.606318485 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c6446ac-f2c3-4897-a995-94176302c6b6-metrics-tls") pod "dns-default-pj6vq" (UID: "2c6446ac-f2c3-4897-a995-94176302c6b6") : secret "dns-default-metrics-tls" not found
Jul 27 17:29:03.360078 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:03.360049    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-f7c9g"]
Jul 27 17:29:03.368704 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:29:03.368680    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f29005f_0e4f_4b5f_9e9a_8a2c3089fd08.slice/crio-f834b0b9d17175bdad154ec57919e4a6ca604984d3ba34fadda2acc3c3acd220 WatchSource:0}: Error finding container f834b0b9d17175bdad154ec57919e4a6ca604984d3ba34fadda2acc3c3acd220: Status 404 returned error can't find the container with id f834b0b9d17175bdad154ec57919e4a6ca604984d3ba34fadda2acc3c3acd220
Jul 27 17:29:04.028412 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:04.028218    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:29:04.029158 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:04.028369    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:29:04.029158 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:04.028461    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b499f6dbb-gq96p: secret "image-registry-tls" not found
Jul 27 17:29:04.029158 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:04.028520    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls podName:86ded04d-8b64-408e-8352-7ef795d848a0 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:06.028504397 +0000 UTC m=+38.515315662 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls") pod "image-registry-6b499f6dbb-gq96p" (UID: "86ded04d-8b64-408e-8352-7ef795d848a0") : secret "image-registry-tls" not found
Jul 27 17:29:04.112688 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:04.112653    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:29:04.112688 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:04.112681    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4vhkj"
Jul 27 17:29:04.114937 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:04.114914    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 17:29:04.115391 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:04.115372    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 17:29:04.115508 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:04.115372    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 17:29:04.115508 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:04.115418    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qwvcb\""
Jul 27 17:29:04.115508 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:04.115448    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zkh28\""
Jul 27 17:29:04.129274 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:04.129254    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c6446ac-f2c3-4897-a995-94176302c6b6-metrics-tls\") pod \"dns-default-pj6vq\" (UID: \"2c6446ac-f2c3-4897-a995-94176302c6b6\") " pod="openshift-dns/dns-default-pj6vq"
Jul 27 17:29:04.129368 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:04.129333    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d458fd4-bb2f-4907-9431-226cac065ce0-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-k5llc\" (UID: \"3d458fd4-bb2f-4907-9431-226cac065ce0\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-k5llc"
Jul 27 17:29:04.129407 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:04.129377    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:29:04.129441 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:04.129406    2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 27 17:29:04.129441 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:04.129427    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c6446ac-f2c3-4897-a995-94176302c6b6-metrics-tls podName:2c6446ac-f2c3-4897-a995-94176302c6b6 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:06.129413 +0000 UTC m=+38.616224265 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c6446ac-f2c3-4897-a995-94176302c6b6-metrics-tls") pod "dns-default-pj6vq" (UID: "2c6446ac-f2c3-4897-a995-94176302c6b6") : secret "dns-default-metrics-tls" not found
Jul 27 17:29:04.129508 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:04.129450    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d458fd4-bb2f-4907-9431-226cac065ce0-networking-console-plugin-cert podName:3d458fd4-bb2f-4907-9431-226cac065ce0 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:06.129438776 +0000 UTC m=+38.616250041 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3d458fd4-bb2f-4907-9431-226cac065ce0-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-k5llc" (UID: "3d458fd4-bb2f-4907-9431-226cac065ce0") : secret "networking-console-plugin-cert" not found
Jul 27 17:29:04.274632 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:04.274595    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-f7c9g" event={"ID":"5f29005f-0e4f-4b5f-9e9a-8a2c3089fd08","Type":"ContainerStarted","Data":"f834b0b9d17175bdad154ec57919e4a6ca604984d3ba34fadda2acc3c3acd220"}
Jul 27 17:29:04.277099 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:04.277074    2557 generic.go:356] "Generic (PLEG): container finished" podID="ee935ed2-91cd-41fb-a3c3-fc5c52f42882" containerID="3682363650c35e83f759944008ee2e6488ec281c6374815633bfefab6353794f" exitCode=0
Jul 27 17:29:04.277233 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:04.277108    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w7blt" event={"ID":"ee935ed2-91cd-41fb-a3c3-fc5c52f42882","Type":"ContainerDied","Data":"3682363650c35e83f759944008ee2e6488ec281c6374815633bfefab6353794f"}
Jul 27 17:29:05.285219 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:05.285169    2557 generic.go:356] "Generic (PLEG): container finished" podID="ee935ed2-91cd-41fb-a3c3-fc5c52f42882" containerID="965aa09d49ec880e5a4427b779a8656ecc4b2cf18333de1de906324f16397853" exitCode=0
Jul 27 17:29:05.285635 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:05.285208    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w7blt" event={"ID":"ee935ed2-91cd-41fb-a3c3-fc5c52f42882","Type":"ContainerDied","Data":"965aa09d49ec880e5a4427b779a8656ecc4b2cf18333de1de906324f16397853"}
Jul 27 17:29:06.044942 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:06.044896    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:29:06.045111 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:06.045071    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:29:06.045111 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:06.045094    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b499f6dbb-gq96p: secret "image-registry-tls" not found
Jul 27 17:29:06.045214 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:06.045171    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls podName:86ded04d-8b64-408e-8352-7ef795d848a0 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:10.045148142 +0000 UTC m=+42.531959411 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls") pod "image-registry-6b499f6dbb-gq96p" (UID: "86ded04d-8b64-408e-8352-7ef795d848a0") : secret "image-registry-tls" not found
Jul 27 17:29:06.145711 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:06.145671    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d458fd4-bb2f-4907-9431-226cac065ce0-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-k5llc\" (UID: \"3d458fd4-bb2f-4907-9431-226cac065ce0\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-k5llc"
Jul 27 17:29:06.146008 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:06.145730    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c6446ac-f2c3-4897-a995-94176302c6b6-metrics-tls\") pod \"dns-default-pj6vq\" (UID: \"2c6446ac-f2c3-4897-a995-94176302c6b6\") " pod="openshift-dns/dns-default-pj6vq"
Jul 27 17:29:06.146008 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:06.145842    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:29:06.146008 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:06.145911    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c6446ac-f2c3-4897-a995-94176302c6b6-metrics-tls podName:2c6446ac-f2c3-4897-a995-94176302c6b6 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:10.145895099 +0000 UTC m=+42.632706364 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c6446ac-f2c3-4897-a995-94176302c6b6-metrics-tls") pod "dns-default-pj6vq" (UID: "2c6446ac-f2c3-4897-a995-94176302c6b6") : secret "dns-default-metrics-tls" not found
Jul 27 17:29:06.146008 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:06.145843    2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 27 17:29:06.146008 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:06.145982    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d458fd4-bb2f-4907-9431-226cac065ce0-networking-console-plugin-cert podName:3d458fd4-bb2f-4907-9431-226cac065ce0 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:10.145969663 +0000 UTC m=+42.632780928 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3d458fd4-bb2f-4907-9431-226cac065ce0-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-k5llc" (UID: "3d458fd4-bb2f-4907-9431-226cac065ce0") : secret "networking-console-plugin-cert" not found
Jul 27 17:29:06.291424 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:06.291390    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w7blt" event={"ID":"ee935ed2-91cd-41fb-a3c3-fc5c52f42882","Type":"ContainerStarted","Data":"340c15244247b3d58dab1194dc84b20cae8579ec3f3de5dbbb75513008d5b41d"}
Jul 27 17:29:06.292971 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:06.292950    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-f7c9g" event={"ID":"5f29005f-0e4f-4b5f-9e9a-8a2c3089fd08","Type":"ContainerStarted","Data":"ecdc6994332e6fe85fed03c83d5ccfd3a467800be83d76a37ffa7a1f88be6e80"}
Jul 27 17:29:06.311629 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:06.311537    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-w7blt" podStartSLOduration=8.062020512 podStartE2EDuration="38.311519214s" totalImagesPullingTime="30.249498702s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:06.310282786 +0000 UTC m=+38.797094075" watchObservedRunningTime="2026-07-27 17:29:06.311519214 +0000 UTC m=+38.798330503"
Jul 27 17:29:06.325445 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:06.325401    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-f7c9g" podStartSLOduration=1.543570565 podStartE2EDuration="4.325386113s" totalImagesPullingTime="2.781815548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:06.324705566 +0000 UTC m=+38.811516854" watchObservedRunningTime="2026-07-27 17:29:06.325386113 +0000 UTC m=+38.812197400"
Jul 27 17:29:10.074516 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:10.074319    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:29:10.074932 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:10.074480    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:29:10.074932 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:10.074560    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b499f6dbb-gq96p: secret "image-registry-tls" not found
Jul 27 17:29:10.074932 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:10.074613    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls podName:86ded04d-8b64-408e-8352-7ef795d848a0 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:18.074598725 +0000 UTC m=+50.561409991 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls") pod "image-registry-6b499f6dbb-gq96p" (UID: "86ded04d-8b64-408e-8352-7ef795d848a0") : secret "image-registry-tls" not found
Jul 27 17:29:10.175809 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:10.175768    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d458fd4-bb2f-4907-9431-226cac065ce0-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-k5llc\" (UID: \"3d458fd4-bb2f-4907-9431-226cac065ce0\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-k5llc"
Jul 27 17:29:10.175809 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:10.175818    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c6446ac-f2c3-4897-a995-94176302c6b6-metrics-tls\") pod \"dns-default-pj6vq\" (UID: \"2c6446ac-f2c3-4897-a995-94176302c6b6\") " pod="openshift-dns/dns-default-pj6vq"
Jul 27 17:29:10.176007 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:10.175920    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:29:10.176007 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:10.175974    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c6446ac-f2c3-4897-a995-94176302c6b6-metrics-tls podName:2c6446ac-f2c3-4897-a995-94176302c6b6 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:18.175961926 +0000 UTC m=+50.662773207 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c6446ac-f2c3-4897-a995-94176302c6b6-metrics-tls") pod "dns-default-pj6vq" (UID: "2c6446ac-f2c3-4897-a995-94176302c6b6") : secret "dns-default-metrics-tls" not found
Jul 27 17:29:10.176078 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:10.175921    2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 27 17:29:10.176078 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:10.176072    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d458fd4-bb2f-4907-9431-226cac065ce0-networking-console-plugin-cert podName:3d458fd4-bb2f-4907-9431-226cac065ce0 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:18.176056088 +0000 UTC m=+50.662867372 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3d458fd4-bb2f-4907-9431-226cac065ce0-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-k5llc" (UID: "3d458fd4-bb2f-4907-9431-226cac065ce0") : secret "networking-console-plugin-cert" not found
Jul 27 17:29:18.131701 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:18.131670    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:29:18.132150 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:18.131820    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:29:18.132150 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:18.131836    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b499f6dbb-gq96p: secret "image-registry-tls" not found
Jul 27 17:29:18.132150 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:18.131883    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls podName:86ded04d-8b64-408e-8352-7ef795d848a0 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:34.131869356 +0000 UTC m=+66.618680623 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls") pod "image-registry-6b499f6dbb-gq96p" (UID: "86ded04d-8b64-408e-8352-7ef795d848a0") : secret "image-registry-tls" not found
Jul 27 17:29:18.232201 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:18.232167    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c6446ac-f2c3-4897-a995-94176302c6b6-metrics-tls\") pod \"dns-default-pj6vq\" (UID: \"2c6446ac-f2c3-4897-a995-94176302c6b6\") " pod="openshift-dns/dns-default-pj6vq"
Jul 27 17:29:18.232371 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:18.232303    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:29:18.232371 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:18.232337    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d458fd4-bb2f-4907-9431-226cac065ce0-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-k5llc\" (UID: \"3d458fd4-bb2f-4907-9431-226cac065ce0\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-k5llc"
Jul 27 17:29:18.232371 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:18.232364    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c6446ac-f2c3-4897-a995-94176302c6b6-metrics-tls podName:2c6446ac-f2c3-4897-a995-94176302c6b6 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:34.232348482 +0000 UTC m=+66.719159747 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c6446ac-f2c3-4897-a995-94176302c6b6-metrics-tls") pod "dns-default-pj6vq" (UID: "2c6446ac-f2c3-4897-a995-94176302c6b6") : secret "dns-default-metrics-tls" not found
Jul 27 17:29:18.232495 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:18.232407    2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 27 17:29:18.232495 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:18.232461    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d458fd4-bb2f-4907-9431-226cac065ce0-networking-console-plugin-cert podName:3d458fd4-bb2f-4907-9431-226cac065ce0 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:34.232449482 +0000 UTC m=+66.719260746 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3d458fd4-bb2f-4907-9431-226cac065ce0-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-k5llc" (UID: "3d458fd4-bb2f-4907-9431-226cac065ce0") : secret "networking-console-plugin-cert" not found
Jul 27 17:29:19.689321 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:19.689290    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z"]
Jul 27 17:29:19.704249 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:19.704225    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z"]
Jul 27 17:29:19.704359 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:19.704316    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z"
Jul 27 17:29:19.706922 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:19.706897    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 27 17:29:19.707059 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:19.706953    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 27 17:29:19.707059 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:19.706952    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 27 17:29:19.707059 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:19.706901    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 27 17:29:19.744285 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:19.744260    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c44c2dc2-907f-4753-8afe-9b58dbf89308-tmp\") pod \"klusterlet-addon-workmgr-69665bc4b4-dlg7z\" (UID: \"c44c2dc2-907f-4753-8afe-9b58dbf89308\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z"
Jul 27 17:29:19.744380 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:19.744325    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c44c2dc2-907f-4753-8afe-9b58dbf89308-klusterlet-config\") pod \"klusterlet-addon-workmgr-69665bc4b4-dlg7z\" (UID: \"c44c2dc2-907f-4753-8afe-9b58dbf89308\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z"
Jul 27 17:29:19.744437 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:19.744383    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f9wj\" (UniqueName: \"kubernetes.io/projected/c44c2dc2-907f-4753-8afe-9b58dbf89308-kube-api-access-8f9wj\") pod \"klusterlet-addon-workmgr-69665bc4b4-dlg7z\" (UID: \"c44c2dc2-907f-4753-8afe-9b58dbf89308\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z"
Jul 27 17:29:19.848178 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:19.846255    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c44c2dc2-907f-4753-8afe-9b58dbf89308-klusterlet-config\") pod \"klusterlet-addon-workmgr-69665bc4b4-dlg7z\" (UID: \"c44c2dc2-907f-4753-8afe-9b58dbf89308\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z"
Jul 27 17:29:19.848178 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:19.846373    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f9wj\" (UniqueName: \"kubernetes.io/projected/c44c2dc2-907f-4753-8afe-9b58dbf89308-kube-api-access-8f9wj\") pod \"klusterlet-addon-workmgr-69665bc4b4-dlg7z\" (UID: \"c44c2dc2-907f-4753-8afe-9b58dbf89308\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z"
Jul 27 17:29:19.848178 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:19.846418    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c44c2dc2-907f-4753-8afe-9b58dbf89308-tmp\") pod \"klusterlet-addon-workmgr-69665bc4b4-dlg7z\" (UID: \"c44c2dc2-907f-4753-8afe-9b58dbf89308\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z"
Jul 27 17:29:19.848178 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:19.846873    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c44c2dc2-907f-4753-8afe-9b58dbf89308-tmp\") pod \"klusterlet-addon-workmgr-69665bc4b4-dlg7z\" (UID: \"c44c2dc2-907f-4753-8afe-9b58dbf89308\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z"
Jul 27 17:29:19.851102 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:19.851074    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c44c2dc2-907f-4753-8afe-9b58dbf89308-klusterlet-config\") pod \"klusterlet-addon-workmgr-69665bc4b4-dlg7z\" (UID: \"c44c2dc2-907f-4753-8afe-9b58dbf89308\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z"
Jul 27 17:29:19.856072 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:19.856047    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f9wj\" (UniqueName: \"kubernetes.io/projected/c44c2dc2-907f-4753-8afe-9b58dbf89308-kube-api-access-8f9wj\") pod \"klusterlet-addon-workmgr-69665bc4b4-dlg7z\" (UID: \"c44c2dc2-907f-4753-8afe-9b58dbf89308\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z"
Jul 27 17:29:20.013629 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:20.013566    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z"
Jul 27 17:29:20.144121 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:20.144092    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z"]
Jul 27 17:29:20.147497 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:29:20.147467    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc44c2dc2_907f_4753_8afe_9b58dbf89308.slice/crio-3269dde265fc71cdf487279a976bbba1e8676f238d4dc0351c00fb0459805ba0 WatchSource:0}: Error finding container 3269dde265fc71cdf487279a976bbba1e8676f238d4dc0351c00fb0459805ba0: Status 404 returned error can't find the container with id 3269dde265fc71cdf487279a976bbba1e8676f238d4dc0351c00fb0459805ba0
Jul 27 17:29:20.325889 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:20.325819    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z" event={"ID":"c44c2dc2-907f-4753-8afe-9b58dbf89308","Type":"ContainerStarted","Data":"3269dde265fc71cdf487279a976bbba1e8676f238d4dc0351c00fb0459805ba0"}
Jul 27 17:29:24.338354 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:24.338320    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z" event={"ID":"c44c2dc2-907f-4753-8afe-9b58dbf89308","Type":"ContainerStarted","Data":"66638458a8a61af91f91551c19860e8fae99877c74ff00e4f71400df95cf086a"}
Jul 27 17:29:24.338737 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:24.338547    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z"
Jul 27 17:29:24.340239 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:24.340220    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z"
Jul 27 17:29:24.353508 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:24.353462    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z" podStartSLOduration=1.643381357 podStartE2EDuration="5.353449551s" totalImagesPullingTime="3.710068194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:24.352618125 +0000 UTC m=+56.839429413" watchObservedRunningTime="2026-07-27 17:29:24.353449551 +0000 UTC m=+56.840260838"
Jul 27 17:29:28.275720 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:28.275689    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gwwvs"
Jul 27 17:29:33.757812 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:33.757773    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs\") pod \"network-metrics-daemon-mzhnc\" (UID: \"57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec\") " pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:29:33.757812 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:33.757818    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8eac848d-fc37-4bc7-b58d-54eaf719c28f-original-pull-secret\") pod \"global-pull-secret-syncer-zzcb2\" (UID: \"8eac848d-fc37-4bc7-b58d-54eaf719c28f\") " pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:29:33.758281 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:33.757900    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhn7z\" (UniqueName: \"kubernetes.io/projected/b96d56dc-19ef-41e7-a185-73926a19a744-kube-api-access-zhn7z\") pod \"network-check-target-4vhkj\" (UID: \"b96d56dc-19ef-41e7-a185-73926a19a744\") " pod="openshift-network-diagnostics/network-check-target-4vhkj"
Jul 27 17:29:33.760488 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:33.760465    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 17:29:33.760572 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:33.760497    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 17:29:33.760572 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:33.760507    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 17:29:33.768283 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:33.768268    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 27 17:29:33.768334 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:33.768322    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs podName:57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec nodeName:}" failed. No retries permitted until 2026-07-27 17:30:37.768307966 +0000 UTC m=+130.255119230 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs") pod "network-metrics-daemon-mzhnc" (UID: "57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec") : secret "metrics-daemon-secret" not found
Jul 27 17:29:33.770039 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:33.770024    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 17:29:33.770437 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:33.770415    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8eac848d-fc37-4bc7-b58d-54eaf719c28f-original-pull-secret\") pod \"global-pull-secret-syncer-zzcb2\" (UID: \"8eac848d-fc37-4bc7-b58d-54eaf719c28f\") " pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:29:33.780605 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:33.780582    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhn7z\" (UniqueName: \"kubernetes.io/projected/b96d56dc-19ef-41e7-a185-73926a19a744-kube-api-access-zhn7z\") pod \"network-check-target-4vhkj\" (UID: \"b96d56dc-19ef-41e7-a185-73926a19a744\") " pod="openshift-network-diagnostics/network-check-target-4vhkj"
Jul 27 17:29:33.829465 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:33.829447    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zkh28\""
Jul 27 17:29:33.838202 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:33.838185    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4vhkj"
Jul 27 17:29:33.955948 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:33.955920    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4vhkj"]
Jul 27 17:29:33.958982 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:29:33.958950    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96d56dc_19ef_41e7_a185_73926a19a744.slice/crio-feb96b324979e58762b359b98752c5b87a9f41bea6a6c9bc8e1a005d6b88fa33 WatchSource:0}: Error finding container feb96b324979e58762b359b98752c5b87a9f41bea6a6c9bc8e1a005d6b88fa33: Status 404 returned error can't find the container with id feb96b324979e58762b359b98752c5b87a9f41bea6a6c9bc8e1a005d6b88fa33
Jul 27 17:29:34.022713 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:34.022637    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zzcb2"
Jul 27 17:29:34.140111 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:34.140082    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zzcb2"]
Jul 27 17:29:34.142950 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:29:34.142921    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eac848d_fc37_4bc7_b58d_54eaf719c28f.slice/crio-62c842b75b902aa5aa2ffcd6431ad51cb3aa86dee8bb0ec0224ef317dfcdc26d WatchSource:0}: Error finding container 62c842b75b902aa5aa2ffcd6431ad51cb3aa86dee8bb0ec0224ef317dfcdc26d: Status 404 returned error can't find the container with id 62c842b75b902aa5aa2ffcd6431ad51cb3aa86dee8bb0ec0224ef317dfcdc26d
Jul 27 17:29:34.160616 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:34.160595    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:29:34.160767 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:34.160731    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:29:34.160767 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:34.160763    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b499f6dbb-gq96p: secret "image-registry-tls" not found
Jul 27 17:29:34.160855 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:34.160812    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls podName:86ded04d-8b64-408e-8352-7ef795d848a0 nodeName:}" failed. No retries permitted until 2026-07-27 17:30:06.160797189 +0000 UTC m=+98.647608454 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls") pod "image-registry-6b499f6dbb-gq96p" (UID: "86ded04d-8b64-408e-8352-7ef795d848a0") : secret "image-registry-tls" not found
Jul 27 17:29:34.261862 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:34.261823    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d458fd4-bb2f-4907-9431-226cac065ce0-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-k5llc\" (UID: \"3d458fd4-bb2f-4907-9431-226cac065ce0\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-k5llc"
Jul 27 17:29:34.262110 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:34.261875    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c6446ac-f2c3-4897-a995-94176302c6b6-metrics-tls\") pod \"dns-default-pj6vq\" (UID: \"2c6446ac-f2c3-4897-a995-94176302c6b6\") " pod="openshift-dns/dns-default-pj6vq"
Jul 27 17:29:34.262110 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:34.261982    2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 27 17:29:34.262110 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:34.262043    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:29:34.262110 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:34.262062    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d458fd4-bb2f-4907-9431-226cac065ce0-networking-console-plugin-cert podName:3d458fd4-bb2f-4907-9431-226cac065ce0 nodeName:}" failed. No retries permitted until 2026-07-27 17:30:06.262045527 +0000 UTC m=+98.748856792 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3d458fd4-bb2f-4907-9431-226cac065ce0-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-k5llc" (UID: "3d458fd4-bb2f-4907-9431-226cac065ce0") : secret "networking-console-plugin-cert" not found
Jul 27 17:29:34.262110 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:29:34.262096    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c6446ac-f2c3-4897-a995-94176302c6b6-metrics-tls podName:2c6446ac-f2c3-4897-a995-94176302c6b6 nodeName:}" failed. No retries permitted until 2026-07-27 17:30:06.26208071 +0000 UTC m=+98.748891978 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c6446ac-f2c3-4897-a995-94176302c6b6-metrics-tls") pod "dns-default-pj6vq" (UID: "2c6446ac-f2c3-4897-a995-94176302c6b6") : secret "dns-default-metrics-tls" not found
Jul 27 17:29:34.361469 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:34.361365    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4vhkj" event={"ID":"b96d56dc-19ef-41e7-a185-73926a19a744","Type":"ContainerStarted","Data":"feb96b324979e58762b359b98752c5b87a9f41bea6a6c9bc8e1a005d6b88fa33"}
Jul 27 17:29:34.362408 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:34.362376    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zzcb2" event={"ID":"8eac848d-fc37-4bc7-b58d-54eaf719c28f","Type":"ContainerStarted","Data":"62c842b75b902aa5aa2ffcd6431ad51cb3aa86dee8bb0ec0224ef317dfcdc26d"}
Jul 27 17:29:39.377958 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:39.377866    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4vhkj" event={"ID":"b96d56dc-19ef-41e7-a185-73926a19a744","Type":"ContainerStarted","Data":"62b4f0c6ddc0b5d40459792c640517ae604a5276bfa31352acb22f32ad4c555a"}
Jul 27 17:29:39.377958 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:39.377948    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4vhkj"
Jul 27 17:29:39.379160 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:39.379138    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zzcb2" event={"ID":"8eac848d-fc37-4bc7-b58d-54eaf719c28f","Type":"ContainerStarted","Data":"59ce7feb43176da366cf1312d7d253666280c7ed4b5a407dcb9b6e7f5b8061ec"}
Jul 27 17:29:39.392765 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:39.392715    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4vhkj" podStartSLOduration=66.38769378 podStartE2EDuration="1m11.392704671s" totalImagesPullingTime="5.005010891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:39.392677913 +0000 UTC m=+71.879489202" watchObservedRunningTime="2026-07-27 17:29:39.392704671 +0000 UTC m=+71.879515936"
Jul 27 17:29:39.406526 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:29:39.406489    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zzcb2" podStartSLOduration=66.585215547 podStartE2EDuration="1m11.406473579s" totalImagesPullingTime="4.821258032s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:39.40629826 +0000 UTC m=+71.893109549" watchObservedRunningTime="2026-07-27 17:29:39.406473579 +0000 UTC m=+71.893284896"
Jul 27 17:30:06.204822 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:30:06.204677    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:30:06.205289 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:30:06.204841    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:30:06.205289 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:30:06.204859    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b499f6dbb-gq96p: secret "image-registry-tls" not found
Jul 27 17:30:06.205289 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:30:06.204912    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls podName:86ded04d-8b64-408e-8352-7ef795d848a0 nodeName:}" failed. No retries permitted until 2026-07-27 17:31:10.204897738 +0000 UTC m=+162.691709003 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls") pod "image-registry-6b499f6dbb-gq96p" (UID: "86ded04d-8b64-408e-8352-7ef795d848a0") : secret "image-registry-tls" not found
Jul 27 17:30:06.305240 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:30:06.305209    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d458fd4-bb2f-4907-9431-226cac065ce0-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-k5llc\" (UID: \"3d458fd4-bb2f-4907-9431-226cac065ce0\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-k5llc"
Jul 27 17:30:06.305392 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:30:06.305254    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c6446ac-f2c3-4897-a995-94176302c6b6-metrics-tls\") pod \"dns-default-pj6vq\" (UID: \"2c6446ac-f2c3-4897-a995-94176302c6b6\") " pod="openshift-dns/dns-default-pj6vq"
Jul 27 17:30:06.305392 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:30:06.305349    2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 27 17:30:06.305392 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:30:06.305371    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:30:06.305495 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:30:06.305413    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d458fd4-bb2f-4907-9431-226cac065ce0-networking-console-plugin-cert podName:3d458fd4-bb2f-4907-9431-226cac065ce0 nodeName:}" failed. No retries permitted until 2026-07-27 17:31:10.305397001 +0000 UTC m=+162.792208265 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3d458fd4-bb2f-4907-9431-226cac065ce0-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-k5llc" (UID: "3d458fd4-bb2f-4907-9431-226cac065ce0") : secret "networking-console-plugin-cert" not found
Jul 27 17:30:06.305495 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:30:06.305429    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c6446ac-f2c3-4897-a995-94176302c6b6-metrics-tls podName:2c6446ac-f2c3-4897-a995-94176302c6b6 nodeName:}" failed. No retries permitted until 2026-07-27 17:31:10.305422282 +0000 UTC m=+162.792233548 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c6446ac-f2c3-4897-a995-94176302c6b6-metrics-tls") pod "dns-default-pj6vq" (UID: "2c6446ac-f2c3-4897-a995-94176302c6b6") : secret "dns-default-metrics-tls" not found
Jul 27 17:30:10.384095 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:30:10.384063    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4vhkj"
Jul 27 17:30:37.828829 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:30:37.828790    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs\") pod \"network-metrics-daemon-mzhnc\" (UID: \"57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec\") " pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:30:37.829304 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:30:37.828899    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 27 17:30:37.829304 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:30:37.828984    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs podName:57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec nodeName:}" failed. No retries permitted until 2026-07-27 17:32:39.828969896 +0000 UTC m=+252.315781161 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs") pod "network-metrics-daemon-mzhnc" (UID: "57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec") : secret "metrics-daemon-secret" not found
Jul 27 17:31:05.329770 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:31:05.329720    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p" podUID="86ded04d-8b64-408e-8352-7ef795d848a0"
Jul 27 17:31:05.345881 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:31:05.345856    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-74c9f884b7-k5llc" podUID="3d458fd4-bb2f-4907-9431-226cac065ce0"
Jul 27 17:31:05.436661 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:31:05.436612    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-pj6vq" podUID="2c6446ac-f2c3-4897-a995-94176302c6b6"
Jul 27 17:31:05.580016 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:31:05.579952    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pj6vq"
Jul 27 17:31:05.580137 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:31:05.579952    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:31:05.580137 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:31:05.579952    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-k5llc"
Jul 27 17:31:07.123376 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:31:07.123332    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-mzhnc" podUID="57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec"
Jul 27 17:31:10.263060 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:31:10.263029    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:31:10.263443 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:31:10.263179    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:31:10.263443 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:31:10.263196    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b499f6dbb-gq96p: secret "image-registry-tls" not found
Jul 27 17:31:10.263443 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:31:10.263262    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls podName:86ded04d-8b64-408e-8352-7ef795d848a0 nodeName:}" failed. No retries permitted until 2026-07-27 17:33:12.263244849 +0000 UTC m=+284.750056127 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls") pod "image-registry-6b499f6dbb-gq96p" (UID: "86ded04d-8b64-408e-8352-7ef795d848a0") : secret "image-registry-tls" not found
Jul 27 17:31:10.363400 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:31:10.363367    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d458fd4-bb2f-4907-9431-226cac065ce0-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-k5llc\" (UID: \"3d458fd4-bb2f-4907-9431-226cac065ce0\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-k5llc"
Jul 27 17:31:10.363400 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:31:10.363407    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c6446ac-f2c3-4897-a995-94176302c6b6-metrics-tls\") pod \"dns-default-pj6vq\" (UID: \"2c6446ac-f2c3-4897-a995-94176302c6b6\") " pod="openshift-dns/dns-default-pj6vq"
Jul 27 17:31:10.363594 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:31:10.363503    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:31:10.363594 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:31:10.363506    2557 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 27 17:31:10.363594 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:31:10.363559    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c6446ac-f2c3-4897-a995-94176302c6b6-metrics-tls podName:2c6446ac-f2c3-4897-a995-94176302c6b6 nodeName:}" failed. No retries permitted until 2026-07-27 17:33:12.363546535 +0000 UTC m=+284.850357800 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2c6446ac-f2c3-4897-a995-94176302c6b6-metrics-tls") pod "dns-default-pj6vq" (UID: "2c6446ac-f2c3-4897-a995-94176302c6b6") : secret "dns-default-metrics-tls" not found
Jul 27 17:31:10.363594 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:31:10.363573    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d458fd4-bb2f-4907-9431-226cac065ce0-networking-console-plugin-cert podName:3d458fd4-bb2f-4907-9431-226cac065ce0 nodeName:}" failed. No retries permitted until 2026-07-27 17:33:12.363565946 +0000 UTC m=+284.850377211 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3d458fd4-bb2f-4907-9431-226cac065ce0-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-k5llc" (UID: "3d458fd4-bb2f-4907-9431-226cac065ce0") : secret "networking-console-plugin-cert" not found
Jul 27 17:31:19.112611 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:31:19.112556    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:31:24.339881 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:31:24.339786    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z" podUID="c44c2dc2-907f-4753-8afe-9b58dbf89308" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.10:8000/readyz\": dial tcp 10.132.0.10:8000: connect: connection refused"
Jul 27 17:31:24.624887 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:31:24.624813    2557 generic.go:356] "Generic (PLEG): container finished" podID="c44c2dc2-907f-4753-8afe-9b58dbf89308" containerID="66638458a8a61af91f91551c19860e8fae99877c74ff00e4f71400df95cf086a" exitCode=1
Jul 27 17:31:24.624887 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:31:24.624852    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z" event={"ID":"c44c2dc2-907f-4753-8afe-9b58dbf89308","Type":"ContainerDied","Data":"66638458a8a61af91f91551c19860e8fae99877c74ff00e4f71400df95cf086a"}
Jul 27 17:31:24.625180 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:31:24.625166    2557 scope.go:117] "RemoveContainer" containerID="66638458a8a61af91f91551c19860e8fae99877c74ff00e4f71400df95cf086a"
Jul 27 17:31:25.628996 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:31:25.628964    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z" event={"ID":"c44c2dc2-907f-4753-8afe-9b58dbf89308","Type":"ContainerStarted","Data":"366d51b71e53734de7354d156e9b61fada91e50907d78de598bcefd3c81bb279"}
Jul 27 17:31:25.629382 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:31:25.629187    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z"
Jul 27 17:31:25.629814 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:31:25.629794    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z"
Jul 27 17:32:38.569384 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:38.569335    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t"]
Jul 27 17:32:38.572490 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:38.572471    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-8wzwg"]
Jul 27 17:32:38.572610 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:38.572593    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t"
Jul 27 17:32:38.574822 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:38.574797    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 27 17:32:38.574933 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:38.574839    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 27 17:32:38.575236 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:38.575218    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 27 17:32:38.575449 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:38.575434    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-wljxx\""
Jul 27 17:32:38.575542 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:38.575500    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 27 17:32:38.576024 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:38.576010    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-8wzwg"
Jul 27 17:32:38.577945 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:38.577924    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-grml9\""
Jul 27 17:32:38.584149 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:38.584128    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t"]
Jul 27 17:32:38.584866 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:38.584848    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-8wzwg"]
Jul 27 17:32:38.694915 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:38.694882    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a3ad2c9-8fab-442a-9ec6-f8eacd33bc22-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t\" (UID: \"0a3ad2c9-8fab-442a-9ec6-f8eacd33bc22\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t"
Jul 27 17:32:38.694915 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:38.694913    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0a3ad2c9-8fab-442a-9ec6-f8eacd33bc22-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t\" (UID: \"0a3ad2c9-8fab-442a-9ec6-f8eacd33bc22\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t"
Jul 27 17:32:38.695091 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:38.694938    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg25k\" (UniqueName: \"kubernetes.io/projected/37e14bdb-244c-41b3-8cbb-709c029dcca9-kube-api-access-jg25k\") pod \"network-check-source-7d88cb55d8-8wzwg\" (UID: \"37e14bdb-244c-41b3-8cbb-709c029dcca9\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-8wzwg"
Jul 27 17:32:38.695091 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:38.694968    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk27k\" (UniqueName: \"kubernetes.io/projected/0a3ad2c9-8fab-442a-9ec6-f8eacd33bc22-kube-api-access-fk27k\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t\" (UID: \"0a3ad2c9-8fab-442a-9ec6-f8eacd33bc22\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t"
Jul 27 17:32:38.796108 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:38.796082    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a3ad2c9-8fab-442a-9ec6-f8eacd33bc22-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t\" (UID: \"0a3ad2c9-8fab-442a-9ec6-f8eacd33bc22\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t"
Jul 27 17:32:38.796206 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:38.796114    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0a3ad2c9-8fab-442a-9ec6-f8eacd33bc22-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t\" (UID: \"0a3ad2c9-8fab-442a-9ec6-f8eacd33bc22\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t"
Jul 27 17:32:38.796206 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:38.796134    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg25k\" (UniqueName: \"kubernetes.io/projected/37e14bdb-244c-41b3-8cbb-709c029dcca9-kube-api-access-jg25k\") pod \"network-check-source-7d88cb55d8-8wzwg\" (UID: \"37e14bdb-244c-41b3-8cbb-709c029dcca9\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-8wzwg"
Jul 27 17:32:38.796206 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:38.796168    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk27k\" (UniqueName: \"kubernetes.io/projected/0a3ad2c9-8fab-442a-9ec6-f8eacd33bc22-kube-api-access-fk27k\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t\" (UID: \"0a3ad2c9-8fab-442a-9ec6-f8eacd33bc22\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t"
Jul 27 17:32:38.796604 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:38.796584    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0a3ad2c9-8fab-442a-9ec6-f8eacd33bc22-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t\" (UID: \"0a3ad2c9-8fab-442a-9ec6-f8eacd33bc22\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t"
Jul 27 17:32:38.798529 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:38.798510    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a3ad2c9-8fab-442a-9ec6-f8eacd33bc22-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t\" (UID: \"0a3ad2c9-8fab-442a-9ec6-f8eacd33bc22\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t"
Jul 27 17:32:38.804030 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:38.804008    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk27k\" (UniqueName: \"kubernetes.io/projected/0a3ad2c9-8fab-442a-9ec6-f8eacd33bc22-kube-api-access-fk27k\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t\" (UID: \"0a3ad2c9-8fab-442a-9ec6-f8eacd33bc22\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t"
Jul 27 17:32:38.804324 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:38.804307    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg25k\" (UniqueName: \"kubernetes.io/projected/37e14bdb-244c-41b3-8cbb-709c029dcca9-kube-api-access-jg25k\") pod \"network-check-source-7d88cb55d8-8wzwg\" (UID: \"37e14bdb-244c-41b3-8cbb-709c029dcca9\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-8wzwg"
Jul 27 17:32:38.883406 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:38.883356    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t"
Jul 27 17:32:38.888288 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:38.888268    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-8wzwg"
Jul 27 17:32:39.007841 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:39.007813    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t"]
Jul 27 17:32:39.010774 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:32:39.010736    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3ad2c9_8fab_442a_9ec6_f8eacd33bc22.slice/crio-50f2d45fd6937ee73ca1fc1c1c8017a6ae2339197a038baf2ee5ac7aacb19dc3 WatchSource:0}: Error finding container 50f2d45fd6937ee73ca1fc1c1c8017a6ae2339197a038baf2ee5ac7aacb19dc3: Status 404 returned error can't find the container with id 50f2d45fd6937ee73ca1fc1c1c8017a6ae2339197a038baf2ee5ac7aacb19dc3
Jul 27 17:32:39.021613 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:39.021590    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-8wzwg"]
Jul 27 17:32:39.025523 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:32:39.025500    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37e14bdb_244c_41b3_8cbb_709c029dcca9.slice/crio-65d9960684b513b4c93fbc6e19f978d18547ee44af846f83b31d068a283c8ae0 WatchSource:0}: Error finding container 65d9960684b513b4c93fbc6e19f978d18547ee44af846f83b31d068a283c8ae0: Status 404 returned error can't find the container with id 65d9960684b513b4c93fbc6e19f978d18547ee44af846f83b31d068a283c8ae0
Jul 27 17:32:39.807605 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:39.807569    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-8wzwg" event={"ID":"37e14bdb-244c-41b3-8cbb-709c029dcca9","Type":"ContainerStarted","Data":"df1074a9e3fa730a99f63c8fb36b3c384146f1ec458d8236d365222a726a0dbc"}
Jul 27 17:32:39.808097 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:39.807630    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-8wzwg" event={"ID":"37e14bdb-244c-41b3-8cbb-709c029dcca9","Type":"ContainerStarted","Data":"65d9960684b513b4c93fbc6e19f978d18547ee44af846f83b31d068a283c8ae0"}
Jul 27 17:32:39.809459 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:39.809431    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t" event={"ID":"0a3ad2c9-8fab-442a-9ec6-f8eacd33bc22","Type":"ContainerStarted","Data":"50f2d45fd6937ee73ca1fc1c1c8017a6ae2339197a038baf2ee5ac7aacb19dc3"}
Jul 27 17:32:39.821920 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:39.821881    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-8wzwg" podStartSLOduration=1.82186737 podStartE2EDuration="1.82186737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:32:39.821091605 +0000 UTC m=+252.307902904" watchObservedRunningTime="2026-07-27 17:32:39.82186737 +0000 UTC m=+252.308678657"
Jul 27 17:32:39.905014 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:39.904981    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs\") pod \"network-metrics-daemon-mzhnc\" (UID: \"57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec\") " pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:32:39.905149 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:32:39.905103    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 27 17:32:39.905194 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:32:39.905170    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs podName:57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec nodeName:}" failed. No retries permitted until 2026-07-27 17:34:41.905145969 +0000 UTC m=+374.391957235 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs") pod "network-metrics-daemon-mzhnc" (UID: "57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec") : secret "metrics-daemon-secret" not found
Jul 27 17:32:42.722089 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:42.722054    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cz9hj"]
Jul 27 17:32:42.725370 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:42.725349    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cz9hj"
Jul 27 17:32:42.731475 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:42.731346    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 27 17:32:42.731475 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:42.731381    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 27 17:32:42.731919 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:42.731900    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 27 17:32:42.731919 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:42.731907    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 27 17:32:42.732089 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:42.731913    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8r9mw\""
Jul 27 17:32:42.741054 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:42.741031    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cz9hj"]
Jul 27 17:32:42.818373 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:42.818339    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t" event={"ID":"0a3ad2c9-8fab-442a-9ec6-f8eacd33bc22","Type":"ContainerStarted","Data":"a689a000ec8ddd50e1140b8956751687699539fbeea87134cac75b397060447a"}
Jul 27 17:32:42.826552 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:42.826529    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5a1f84b3-6930-4b20-95e9-75d46cddb36b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cz9hj\" (UID: \"5a1f84b3-6930-4b20-95e9-75d46cddb36b\") " pod="openshift-insights/insights-runtime-extractor-cz9hj"
Jul 27 17:32:42.826646 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:42.826576    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5vb4\" (UniqueName: \"kubernetes.io/projected/5a1f84b3-6930-4b20-95e9-75d46cddb36b-kube-api-access-k5vb4\") pod \"insights-runtime-extractor-cz9hj\" (UID: \"5a1f84b3-6930-4b20-95e9-75d46cddb36b\") " pod="openshift-insights/insights-runtime-extractor-cz9hj"
Jul 27 17:32:42.826684 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:42.826653    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5a1f84b3-6930-4b20-95e9-75d46cddb36b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cz9hj\" (UID: \"5a1f84b3-6930-4b20-95e9-75d46cddb36b\") " pod="openshift-insights/insights-runtime-extractor-cz9hj"
Jul 27 17:32:42.826726 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:42.826682    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5a1f84b3-6930-4b20-95e9-75d46cddb36b-data-volume\") pod \"insights-runtime-extractor-cz9hj\" (UID: \"5a1f84b3-6930-4b20-95e9-75d46cddb36b\") " pod="openshift-insights/insights-runtime-extractor-cz9hj"
Jul 27 17:32:42.826726 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:42.826710    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5a1f84b3-6930-4b20-95e9-75d46cddb36b-crio-socket\") pod \"insights-runtime-extractor-cz9hj\" (UID: \"5a1f84b3-6930-4b20-95e9-75d46cddb36b\") " pod="openshift-insights/insights-runtime-extractor-cz9hj"
Jul 27 17:32:42.833953 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:42.833907    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t" podStartSLOduration=1.491948042 podStartE2EDuration="4.833892381s" totalImagesPullingTime="3.341944339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:32:42.833291442 +0000 UTC m=+255.320102730" watchObservedRunningTime="2026-07-27 17:32:42.833892381 +0000 UTC m=+255.320703707"
Jul 27 17:32:42.927225 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:42.927189    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5a1f84b3-6930-4b20-95e9-75d46cddb36b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cz9hj\" (UID: \"5a1f84b3-6930-4b20-95e9-75d46cddb36b\") " pod="openshift-insights/insights-runtime-extractor-cz9hj"
Jul 27 17:32:42.927432 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:32:42.927394    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 17:32:42.927567 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:32:42.927482    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a1f84b3-6930-4b20-95e9-75d46cddb36b-insights-runtime-extractor-tls podName:5a1f84b3-6930-4b20-95e9-75d46cddb36b nodeName:}" failed. No retries permitted until 2026-07-27 17:32:43.427459975 +0000 UTC m=+255.914271257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5a1f84b3-6930-4b20-95e9-75d46cddb36b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cz9hj" (UID: "5a1f84b3-6930-4b20-95e9-75d46cddb36b") : secret "insights-runtime-extractor-tls" not found
Jul 27 17:32:42.927567 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:42.927529    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5vb4\" (UniqueName: \"kubernetes.io/projected/5a1f84b3-6930-4b20-95e9-75d46cddb36b-kube-api-access-k5vb4\") pod \"insights-runtime-extractor-cz9hj\" (UID: \"5a1f84b3-6930-4b20-95e9-75d46cddb36b\") " pod="openshift-insights/insights-runtime-extractor-cz9hj"
Jul 27 17:32:42.927685 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:42.927603    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5a1f84b3-6930-4b20-95e9-75d46cddb36b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cz9hj\" (UID: \"5a1f84b3-6930-4b20-95e9-75d46cddb36b\") " pod="openshift-insights/insights-runtime-extractor-cz9hj"
Jul 27 17:32:42.927685 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:42.927637    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5a1f84b3-6930-4b20-95e9-75d46cddb36b-data-volume\") pod \"insights-runtime-extractor-cz9hj\" (UID: \"5a1f84b3-6930-4b20-95e9-75d46cddb36b\") " pod="openshift-insights/insights-runtime-extractor-cz9hj"
Jul 27 17:32:42.927685 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:32:42.927674    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 17:32:42.927849 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:42.927696    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5a1f84b3-6930-4b20-95e9-75d46cddb36b-crio-socket\") pod \"insights-runtime-extractor-cz9hj\" (UID: \"5a1f84b3-6930-4b20-95e9-75d46cddb36b\") " pod="openshift-insights/insights-runtime-extractor-cz9hj"
Jul 27 17:32:42.927849 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:32:42.927742    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5a1f84b3-6930-4b20-95e9-75d46cddb36b-kube-rbac-proxy-cm podName:5a1f84b3-6930-4b20-95e9-75d46cddb36b nodeName:}" failed. No retries permitted until 2026-07-27 17:32:43.427724886 +0000 UTC m=+255.914536172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5a1f84b3-6930-4b20-95e9-75d46cddb36b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cz9hj" (UID: "5a1f84b3-6930-4b20-95e9-75d46cddb36b") : configmap "kube-rbac-proxy" not found
Jul 27 17:32:42.927957 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:42.927841    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5a1f84b3-6930-4b20-95e9-75d46cddb36b-crio-socket\") pod \"insights-runtime-extractor-cz9hj\" (UID: \"5a1f84b3-6930-4b20-95e9-75d46cddb36b\") " pod="openshift-insights/insights-runtime-extractor-cz9hj"
Jul 27 17:32:42.927994 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:42.927956    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5a1f84b3-6930-4b20-95e9-75d46cddb36b-data-volume\") pod \"insights-runtime-extractor-cz9hj\" (UID: \"5a1f84b3-6930-4b20-95e9-75d46cddb36b\") " pod="openshift-insights/insights-runtime-extractor-cz9hj"
Jul 27 17:32:42.935306 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:42.935285    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5vb4\" (UniqueName: \"kubernetes.io/projected/5a1f84b3-6930-4b20-95e9-75d46cddb36b-kube-api-access-k5vb4\") pod \"insights-runtime-extractor-cz9hj\" (UID: \"5a1f84b3-6930-4b20-95e9-75d46cddb36b\") " pod="openshift-insights/insights-runtime-extractor-cz9hj"
Jul 27 17:32:43.432060 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:43.432021    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5a1f84b3-6930-4b20-95e9-75d46cddb36b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cz9hj\" (UID: \"5a1f84b3-6930-4b20-95e9-75d46cddb36b\") " pod="openshift-insights/insights-runtime-extractor-cz9hj"
Jul 27 17:32:43.432268 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:43.432084    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5a1f84b3-6930-4b20-95e9-75d46cddb36b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cz9hj\" (UID: \"5a1f84b3-6930-4b20-95e9-75d46cddb36b\") " pod="openshift-insights/insights-runtime-extractor-cz9hj"
Jul 27 17:32:43.432268 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:32:43.432167    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 17:32:43.432268 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:32:43.432211    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 17:32:43.432268 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:32:43.432251    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5a1f84b3-6930-4b20-95e9-75d46cddb36b-kube-rbac-proxy-cm podName:5a1f84b3-6930-4b20-95e9-75d46cddb36b nodeName:}" failed. No retries permitted until 2026-07-27 17:32:44.432229641 +0000 UTC m=+256.919040915 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5a1f84b3-6930-4b20-95e9-75d46cddb36b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cz9hj" (UID: "5a1f84b3-6930-4b20-95e9-75d46cddb36b") : configmap "kube-rbac-proxy" not found
Jul 27 17:32:43.432268 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:32:43.432270    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a1f84b3-6930-4b20-95e9-75d46cddb36b-insights-runtime-extractor-tls podName:5a1f84b3-6930-4b20-95e9-75d46cddb36b nodeName:}" failed. No retries permitted until 2026-07-27 17:32:44.43226031 +0000 UTC m=+256.919071576 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5a1f84b3-6930-4b20-95e9-75d46cddb36b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cz9hj" (UID: "5a1f84b3-6930-4b20-95e9-75d46cddb36b") : secret "insights-runtime-extractor-tls" not found
Jul 27 17:32:44.112233 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:44.112199    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-wc267"]
Jul 27 17:32:44.115344 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:44.115316    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wc267"
Jul 27 17:32:44.117908 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:44.117884    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 27 17:32:44.118064 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:44.117885    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 27 17:32:44.118399 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:44.118378    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-q2drp\""
Jul 27 17:32:44.123367 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:44.123344    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-wc267"]
Jul 27 17:32:44.240402 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:44.240359    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmzc5\" (UniqueName: \"kubernetes.io/projected/f3577d8f-d13f-4cb9-948f-3d09033f62d9-kube-api-access-fmzc5\") pod \"migrator-6865c9967-wc267\" (UID: \"f3577d8f-d13f-4cb9-948f-3d09033f62d9\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wc267"
Jul 27 17:32:44.341607 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:44.341551    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmzc5\" (UniqueName: \"kubernetes.io/projected/f3577d8f-d13f-4cb9-948f-3d09033f62d9-kube-api-access-fmzc5\") pod \"migrator-6865c9967-wc267\" (UID: \"f3577d8f-d13f-4cb9-948f-3d09033f62d9\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wc267"
Jul 27 17:32:44.349056 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:44.349025    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmzc5\" (UniqueName: \"kubernetes.io/projected/f3577d8f-d13f-4cb9-948f-3d09033f62d9-kube-api-access-fmzc5\") pod \"migrator-6865c9967-wc267\" (UID: \"f3577d8f-d13f-4cb9-948f-3d09033f62d9\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wc267"
Jul 27 17:32:44.425925 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:44.425893    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wc267"
Jul 27 17:32:44.442868 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:44.442844    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5a1f84b3-6930-4b20-95e9-75d46cddb36b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cz9hj\" (UID: \"5a1f84b3-6930-4b20-95e9-75d46cddb36b\") " pod="openshift-insights/insights-runtime-extractor-cz9hj"
Jul 27 17:32:44.442991 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:44.442916    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5a1f84b3-6930-4b20-95e9-75d46cddb36b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cz9hj\" (UID: \"5a1f84b3-6930-4b20-95e9-75d46cddb36b\") " pod="openshift-insights/insights-runtime-extractor-cz9hj"
Jul 27 17:32:44.443040 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:32:44.442996    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 17:32:44.443040 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:32:44.443019    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 17:32:44.443114 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:32:44.443047    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5a1f84b3-6930-4b20-95e9-75d46cddb36b-kube-rbac-proxy-cm podName:5a1f84b3-6930-4b20-95e9-75d46cddb36b nodeName:}" failed. No retries permitted until 2026-07-27 17:32:46.443033493 +0000 UTC m=+258.929844759 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5a1f84b3-6930-4b20-95e9-75d46cddb36b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cz9hj" (UID: "5a1f84b3-6930-4b20-95e9-75d46cddb36b") : configmap "kube-rbac-proxy" not found
Jul 27 17:32:44.443114 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:32:44.443086    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a1f84b3-6930-4b20-95e9-75d46cddb36b-insights-runtime-extractor-tls podName:5a1f84b3-6930-4b20-95e9-75d46cddb36b nodeName:}" failed. No retries permitted until 2026-07-27 17:32:46.44306666 +0000 UTC m=+258.929877940 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5a1f84b3-6930-4b20-95e9-75d46cddb36b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cz9hj" (UID: "5a1f84b3-6930-4b20-95e9-75d46cddb36b") : secret "insights-runtime-extractor-tls" not found
Jul 27 17:32:44.548632 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:44.548601    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-wc267"]
Jul 27 17:32:44.551471 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:32:44.551443    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3577d8f_d13f_4cb9_948f_3d09033f62d9.slice/crio-482cea3a0cddce5fcf47bd280730328f61e20d795cb92c7bfa26a33ba1c8888d WatchSource:0}: Error finding container 482cea3a0cddce5fcf47bd280730328f61e20d795cb92c7bfa26a33ba1c8888d: Status 404 returned error can't find the container with id 482cea3a0cddce5fcf47bd280730328f61e20d795cb92c7bfa26a33ba1c8888d
Jul 27 17:32:44.706842 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:44.706773    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k56kz_17308a6d-e821-4aae-afe4-592c1f5f0175/dns-node-resolver/0.log"
Jul 27 17:32:44.826300 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:44.826257    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wc267" event={"ID":"f3577d8f-d13f-4cb9-948f-3d09033f62d9","Type":"ContainerStarted","Data":"482cea3a0cddce5fcf47bd280730328f61e20d795cb92c7bfa26a33ba1c8888d"}
Jul 27 17:32:45.906078 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:45.906049    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q8nkb_231ae1f3-5af5-445f-92d1-2a8901275b9f/node-ca/0.log"
Jul 27 17:32:46.458882 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:46.458840    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5a1f84b3-6930-4b20-95e9-75d46cddb36b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cz9hj\" (UID: \"5a1f84b3-6930-4b20-95e9-75d46cddb36b\") " pod="openshift-insights/insights-runtime-extractor-cz9hj"
Jul 27 17:32:46.459074 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:46.458922    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5a1f84b3-6930-4b20-95e9-75d46cddb36b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cz9hj\" (UID: \"5a1f84b3-6930-4b20-95e9-75d46cddb36b\") " pod="openshift-insights/insights-runtime-extractor-cz9hj"
Jul 27 17:32:46.459074 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:32:46.458995    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 17:32:46.459074 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:32:46.458998    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 17:32:46.459074 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:32:46.459046    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5a1f84b3-6930-4b20-95e9-75d46cddb36b-kube-rbac-proxy-cm podName:5a1f84b3-6930-4b20-95e9-75d46cddb36b nodeName:}" failed. No retries permitted until 2026-07-27 17:32:50.45903295 +0000 UTC m=+262.945844214 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5a1f84b3-6930-4b20-95e9-75d46cddb36b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cz9hj" (UID: "5a1f84b3-6930-4b20-95e9-75d46cddb36b") : configmap "kube-rbac-proxy" not found
Jul 27 17:32:46.459074 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:32:46.459061    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a1f84b3-6930-4b20-95e9-75d46cddb36b-insights-runtime-extractor-tls podName:5a1f84b3-6930-4b20-95e9-75d46cddb36b nodeName:}" failed. No retries permitted until 2026-07-27 17:32:50.45905503 +0000 UTC m=+262.945866294 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5a1f84b3-6930-4b20-95e9-75d46cddb36b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cz9hj" (UID: "5a1f84b3-6930-4b20-95e9-75d46cddb36b") : secret "insights-runtime-extractor-tls" not found
Jul 27 17:32:46.706085 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:46.706056    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-f7c9g_5f29005f-0e4f-4b5f-9e9a-8a2c3089fd08/serve-healthcheck-canary/0.log"
Jul 27 17:32:47.837281 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:47.837238    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wc267" event={"ID":"f3577d8f-d13f-4cb9-948f-3d09033f62d9","Type":"ContainerStarted","Data":"8f9ebd1a0f0fdb936f6b78f3db8badace6080d066f1af79b280c04b22e115786"}
Jul 27 17:32:47.837675 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:47.837342    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wc267" event={"ID":"f3577d8f-d13f-4cb9-948f-3d09033f62d9","Type":"ContainerStarted","Data":"f29ee25da876d8acf98a3d0fc568130581cf0cb5f305dc171a255d8fa0586510"}
Jul 27 17:32:47.852359 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:47.852308    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wc267" podStartSLOduration=1.020106804 podStartE2EDuration="3.852293263s" totalImagesPullingTime="2.832186459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:32:47.851787227 +0000 UTC m=+260.338598515" watchObservedRunningTime="2026-07-27 17:32:47.852293263 +0000 UTC m=+260.339104550"
Jul 27 17:32:50.491396 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:50.491304    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5a1f84b3-6930-4b20-95e9-75d46cddb36b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cz9hj\" (UID: \"5a1f84b3-6930-4b20-95e9-75d46cddb36b\") " pod="openshift-insights/insights-runtime-extractor-cz9hj"
Jul 27 17:32:50.491396 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:50.491380    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5a1f84b3-6930-4b20-95e9-75d46cddb36b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cz9hj\" (UID: \"5a1f84b3-6930-4b20-95e9-75d46cddb36b\") " pod="openshift-insights/insights-runtime-extractor-cz9hj"
Jul 27 17:32:50.491828 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:32:50.491451    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 17:32:50.491828 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:32:50.491491    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 17:32:50.491828 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:32:50.491504    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5a1f84b3-6930-4b20-95e9-75d46cddb36b-kube-rbac-proxy-cm podName:5a1f84b3-6930-4b20-95e9-75d46cddb36b nodeName:}" failed. No retries permitted until 2026-07-27 17:32:58.491491407 +0000 UTC m=+270.978302672 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5a1f84b3-6930-4b20-95e9-75d46cddb36b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cz9hj" (UID: "5a1f84b3-6930-4b20-95e9-75d46cddb36b") : configmap "kube-rbac-proxy" not found
Jul 27 17:32:50.491828 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:32:50.491560    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a1f84b3-6930-4b20-95e9-75d46cddb36b-insights-runtime-extractor-tls podName:5a1f84b3-6930-4b20-95e9-75d46cddb36b nodeName:}" failed. No retries permitted until 2026-07-27 17:32:58.491546617 +0000 UTC m=+270.978357882 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5a1f84b3-6930-4b20-95e9-75d46cddb36b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cz9hj" (UID: "5a1f84b3-6930-4b20-95e9-75d46cddb36b") : secret "insights-runtime-extractor-tls" not found
Jul 27 17:32:58.557627 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:58.557592    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5a1f84b3-6930-4b20-95e9-75d46cddb36b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cz9hj\" (UID: \"5a1f84b3-6930-4b20-95e9-75d46cddb36b\") " pod="openshift-insights/insights-runtime-extractor-cz9hj"
Jul 27 17:32:58.558043 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:58.557741    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5a1f84b3-6930-4b20-95e9-75d46cddb36b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cz9hj\" (UID: \"5a1f84b3-6930-4b20-95e9-75d46cddb36b\") " pod="openshift-insights/insights-runtime-extractor-cz9hj"
Jul 27 17:32:58.558317 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:58.558285    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5a1f84b3-6930-4b20-95e9-75d46cddb36b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cz9hj\" (UID: \"5a1f84b3-6930-4b20-95e9-75d46cddb36b\") " pod="openshift-insights/insights-runtime-extractor-cz9hj"
Jul 27 17:32:58.560276 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:58.560256    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5a1f84b3-6930-4b20-95e9-75d46cddb36b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cz9hj\" (UID: \"5a1f84b3-6930-4b20-95e9-75d46cddb36b\") " pod="openshift-insights/insights-runtime-extractor-cz9hj"
Jul 27 17:32:58.633976 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:58.633943    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cz9hj"
Jul 27 17:32:58.754256 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:58.754225    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cz9hj"]
Jul 27 17:32:58.757585 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:32:58.757558    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a1f84b3_6930_4b20_95e9_75d46cddb36b.slice/crio-6e2b232c42e5d3b0afaf6296db85a1fe68710cecbd0973f7e002c6fc36b5b6be WatchSource:0}: Error finding container 6e2b232c42e5d3b0afaf6296db85a1fe68710cecbd0973f7e002c6fc36b5b6be: Status 404 returned error can't find the container with id 6e2b232c42e5d3b0afaf6296db85a1fe68710cecbd0973f7e002c6fc36b5b6be
Jul 27 17:32:58.869600 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:32:58.869513    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cz9hj" event={"ID":"5a1f84b3-6930-4b20-95e9-75d46cddb36b","Type":"ContainerStarted","Data":"6e2b232c42e5d3b0afaf6296db85a1fe68710cecbd0973f7e002c6fc36b5b6be"}
Jul 27 17:33:03.886842 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:03.886799    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cz9hj" event={"ID":"5a1f84b3-6930-4b20-95e9-75d46cddb36b","Type":"ContainerStarted","Data":"bc8dc9b6a9ccc82a040f27871c753f0db8b95d81feb3f62ad22bfff0c5fe6647"}
Jul 27 17:33:06.593192 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:06.593160    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-s7hbx"]
Jul 27 17:33:06.620147 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:06.620111    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-s7hbx"]
Jul 27 17:33:06.620301 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:06.620219    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-s7hbx"
Jul 27 17:33:06.622493 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:06.622468    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-t4k5z\""
Jul 27 17:33:06.622635 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:06.622535    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 27 17:33:06.622635 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:06.622625    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 27 17:33:06.626868 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:06.626843    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm8qq\" (UniqueName: \"kubernetes.io/projected/823de302-a6b7-4e92-8cea-80322d2507ce-kube-api-access-zm8qq\") pod \"downloads-7c97dc4555-s7hbx\" (UID: \"823de302-a6b7-4e92-8cea-80322d2507ce\") " pod="openshift-console/downloads-7c97dc4555-s7hbx"
Jul 27 17:33:06.727788 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:06.727737    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm8qq\" (UniqueName: \"kubernetes.io/projected/823de302-a6b7-4e92-8cea-80322d2507ce-kube-api-access-zm8qq\") pod \"downloads-7c97dc4555-s7hbx\" (UID: \"823de302-a6b7-4e92-8cea-80322d2507ce\") " pod="openshift-console/downloads-7c97dc4555-s7hbx"
Jul 27 17:33:06.739147 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:06.739122    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm8qq\" (UniqueName: \"kubernetes.io/projected/823de302-a6b7-4e92-8cea-80322d2507ce-kube-api-access-zm8qq\") pod \"downloads-7c97dc4555-s7hbx\" (UID: \"823de302-a6b7-4e92-8cea-80322d2507ce\") " pod="openshift-console/downloads-7c97dc4555-s7hbx"
Jul 27 17:33:06.929897 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:06.929863    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-s7hbx"
Jul 27 17:33:07.467401 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:07.467375    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-s7hbx"]
Jul 27 17:33:07.470814 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:33:07.470786    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823de302_a6b7_4e92_8cea_80322d2507ce.slice/crio-e34122620e87a07b58c5ec9a1e0ef7dc6af49431a29119d8bd8f5bf968890767 WatchSource:0}: Error finding container e34122620e87a07b58c5ec9a1e0ef7dc6af49431a29119d8bd8f5bf968890767: Status 404 returned error can't find the container with id e34122620e87a07b58c5ec9a1e0ef7dc6af49431a29119d8bd8f5bf968890767
Jul 27 17:33:07.903223 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:07.903181    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cz9hj" event={"ID":"5a1f84b3-6930-4b20-95e9-75d46cddb36b","Type":"ContainerStarted","Data":"e813fe62644d1bbc64f7eb1a5c0e4b62595309a76921ffa1bedd33447b4b80c9"}
Jul 27 17:33:07.904488 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:07.904456    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-s7hbx" event={"ID":"823de302-a6b7-4e92-8cea-80322d2507ce","Type":"ContainerStarted","Data":"e34122620e87a07b58c5ec9a1e0ef7dc6af49431a29119d8bd8f5bf968890767"}
Jul 27 17:33:08.580567 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:33:08.580491    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p" podUID="86ded04d-8b64-408e-8352-7ef795d848a0"
Jul 27 17:33:08.580805 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:33:08.580614    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-pj6vq" podUID="2c6446ac-f2c3-4897-a995-94176302c6b6"
Jul 27 17:33:08.580805 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:33:08.580617    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-74c9f884b7-k5llc" podUID="3d458fd4-bb2f-4907-9431-226cac065ce0"
Jul 27 17:33:08.908273 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:08.908235    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-k5llc"
Jul 27 17:33:08.908740 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:08.908239    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pj6vq"
Jul 27 17:33:08.908740 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:08.908239    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:33:10.728394 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:10.728362    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64bb7f674d-lsdcl"]
Jul 27 17:33:10.746583 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:10.746521    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64bb7f674d-lsdcl"]
Jul 27 17:33:10.746730 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:10.746655    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64bb7f674d-lsdcl"
Jul 27 17:33:10.749416 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:10.749188    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-stmxj\""
Jul 27 17:33:10.749416 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:10.749239    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 27 17:33:10.749416 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:10.749264    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 27 17:33:10.749416 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:10.749285    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 27 17:33:10.750291 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:10.750036    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 27 17:33:10.750291 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:10.750273    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 27 17:33:10.762696 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:10.762622    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e5e65078-6a76-4778-b0b0-0d8fc032a406-console-config\") pod \"console-64bb7f674d-lsdcl\" (UID: \"e5e65078-6a76-4778-b0b0-0d8fc032a406\") " pod="openshift-console/console-64bb7f674d-lsdcl"
Jul 27 17:33:10.762696 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:10.762659    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e5e65078-6a76-4778-b0b0-0d8fc032a406-console-oauth-config\") pod \"console-64bb7f674d-lsdcl\" (UID: \"e5e65078-6a76-4778-b0b0-0d8fc032a406\") " pod="openshift-console/console-64bb7f674d-lsdcl"
Jul 27 17:33:10.763157 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:10.762703    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e5e65078-6a76-4778-b0b0-0d8fc032a406-oauth-serving-cert\") pod \"console-64bb7f674d-lsdcl\" (UID: \"e5e65078-6a76-4778-b0b0-0d8fc032a406\") " pod="openshift-console/console-64bb7f674d-lsdcl"
Jul 27 17:33:10.763157 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:10.762727    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e65078-6a76-4778-b0b0-0d8fc032a406-service-ca\") pod \"console-64bb7f674d-lsdcl\" (UID: \"e5e65078-6a76-4778-b0b0-0d8fc032a406\") " pod="openshift-console/console-64bb7f674d-lsdcl"
Jul 27 17:33:10.763157 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:10.762796    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbmpk\" (UniqueName: \"kubernetes.io/projected/e5e65078-6a76-4778-b0b0-0d8fc032a406-kube-api-access-rbmpk\") pod \"console-64bb7f674d-lsdcl\" (UID: \"e5e65078-6a76-4778-b0b0-0d8fc032a406\") " pod="openshift-console/console-64bb7f674d-lsdcl"
Jul 27 17:33:10.763157 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:10.762921    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e5e65078-6a76-4778-b0b0-0d8fc032a406-console-serving-cert\") pod \"console-64bb7f674d-lsdcl\" (UID: \"e5e65078-6a76-4778-b0b0-0d8fc032a406\") " pod="openshift-console/console-64bb7f674d-lsdcl"
Jul 27 17:33:10.864200 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:10.864169    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e5e65078-6a76-4778-b0b0-0d8fc032a406-console-serving-cert\") pod \"console-64bb7f674d-lsdcl\" (UID: \"e5e65078-6a76-4778-b0b0-0d8fc032a406\") " pod="openshift-console/console-64bb7f674d-lsdcl"
Jul 27 17:33:10.864391 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:10.864225    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e5e65078-6a76-4778-b0b0-0d8fc032a406-console-config\") pod \"console-64bb7f674d-lsdcl\" (UID: \"e5e65078-6a76-4778-b0b0-0d8fc032a406\") " pod="openshift-console/console-64bb7f674d-lsdcl"
Jul 27 17:33:10.864391 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:10.864253    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e5e65078-6a76-4778-b0b0-0d8fc032a406-console-oauth-config\") pod \"console-64bb7f674d-lsdcl\" (UID: \"e5e65078-6a76-4778-b0b0-0d8fc032a406\") " pod="openshift-console/console-64bb7f674d-lsdcl"
Jul 27 17:33:10.864391 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:10.864291    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e5e65078-6a76-4778-b0b0-0d8fc032a406-oauth-serving-cert\") pod \"console-64bb7f674d-lsdcl\" (UID: \"e5e65078-6a76-4778-b0b0-0d8fc032a406\") " pod="openshift-console/console-64bb7f674d-lsdcl"
Jul 27 17:33:10.864391 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:10.864315    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e65078-6a76-4778-b0b0-0d8fc032a406-service-ca\") pod \"console-64bb7f674d-lsdcl\" (UID: \"e5e65078-6a76-4778-b0b0-0d8fc032a406\") " pod="openshift-console/console-64bb7f674d-lsdcl"
Jul 27 17:33:10.864391 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:10.864365    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbmpk\" (UniqueName: \"kubernetes.io/projected/e5e65078-6a76-4778-b0b0-0d8fc032a406-kube-api-access-rbmpk\") pod \"console-64bb7f674d-lsdcl\" (UID: \"e5e65078-6a76-4778-b0b0-0d8fc032a406\") " pod="openshift-console/console-64bb7f674d-lsdcl"
Jul 27 17:33:10.865290 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:10.865262    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e5e65078-6a76-4778-b0b0-0d8fc032a406-oauth-serving-cert\") pod \"console-64bb7f674d-lsdcl\" (UID: \"e5e65078-6a76-4778-b0b0-0d8fc032a406\") " pod="openshift-console/console-64bb7f674d-lsdcl"
Jul 27 17:33:10.865290 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:10.865276    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e65078-6a76-4778-b0b0-0d8fc032a406-service-ca\") pod \"console-64bb7f674d-lsdcl\" (UID: \"e5e65078-6a76-4778-b0b0-0d8fc032a406\") " pod="openshift-console/console-64bb7f674d-lsdcl"
Jul 27 17:33:10.865444 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:10.865334    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e5e65078-6a76-4778-b0b0-0d8fc032a406-console-config\") pod \"console-64bb7f674d-lsdcl\" (UID: \"e5e65078-6a76-4778-b0b0-0d8fc032a406\") " pod="openshift-console/console-64bb7f674d-lsdcl"
Jul 27 17:33:10.867070 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:10.867028    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e5e65078-6a76-4778-b0b0-0d8fc032a406-console-serving-cert\") pod \"console-64bb7f674d-lsdcl\" (UID: \"e5e65078-6a76-4778-b0b0-0d8fc032a406\") " pod="openshift-console/console-64bb7f674d-lsdcl"
Jul 27 17:33:10.867771 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:10.867729    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e5e65078-6a76-4778-b0b0-0d8fc032a406-console-oauth-config\") pod \"console-64bb7f674d-lsdcl\" (UID: \"e5e65078-6a76-4778-b0b0-0d8fc032a406\") " pod="openshift-console/console-64bb7f674d-lsdcl"
Jul 27 17:33:10.872783 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:10.872744    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbmpk\" (UniqueName: \"kubernetes.io/projected/e5e65078-6a76-4778-b0b0-0d8fc032a406-kube-api-access-rbmpk\") pod \"console-64bb7f674d-lsdcl\" (UID: \"e5e65078-6a76-4778-b0b0-0d8fc032a406\") " pod="openshift-console/console-64bb7f674d-lsdcl"
Jul 27 17:33:11.059822 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:11.059724    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64bb7f674d-lsdcl"
Jul 27 17:33:11.611068 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:11.611043    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64bb7f674d-lsdcl"]
Jul 27 17:33:11.614112 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:33:11.614076    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e65078_6a76_4778_b0b0_0d8fc032a406.slice/crio-08614bf9aec5f28fb9ad5410e897c65f3998f8a4701b826ad478d4b2ada9e352 WatchSource:0}: Error finding container 08614bf9aec5f28fb9ad5410e897c65f3998f8a4701b826ad478d4b2ada9e352: Status 404 returned error can't find the container with id 08614bf9aec5f28fb9ad5410e897c65f3998f8a4701b826ad478d4b2ada9e352
Jul 27 17:33:11.919196 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:11.919159    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64bb7f674d-lsdcl" event={"ID":"e5e65078-6a76-4778-b0b0-0d8fc032a406","Type":"ContainerStarted","Data":"08614bf9aec5f28fb9ad5410e897c65f3998f8a4701b826ad478d4b2ada9e352"}
Jul 27 17:33:11.921414 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:11.921384    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cz9hj" event={"ID":"5a1f84b3-6930-4b20-95e9-75d46cddb36b","Type":"ContainerStarted","Data":"b18bd50b79caf6d10b2e9aabbef39e5cf6ff44befbac32fbfb4434d141e01ff1"}
Jul 27 17:33:12.277350 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:12.277258    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:33:12.279925 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:12.279901    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls\") pod \"image-registry-6b499f6dbb-gq96p\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") " pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:33:12.378244 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:12.378205    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c6446ac-f2c3-4897-a995-94176302c6b6-metrics-tls\") pod \"dns-default-pj6vq\" (UID: \"2c6446ac-f2c3-4897-a995-94176302c6b6\") " pod="openshift-dns/dns-default-pj6vq"
Jul 27 17:33:12.378429 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:12.378298    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d458fd4-bb2f-4907-9431-226cac065ce0-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-k5llc\" (UID: \"3d458fd4-bb2f-4907-9431-226cac065ce0\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-k5llc"
Jul 27 17:33:12.380884 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:12.380852    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3d458fd4-bb2f-4907-9431-226cac065ce0-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-k5llc\" (UID: \"3d458fd4-bb2f-4907-9431-226cac065ce0\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-k5llc"
Jul 27 17:33:12.380976 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:12.380908    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c6446ac-f2c3-4897-a995-94176302c6b6-metrics-tls\") pod \"dns-default-pj6vq\" (UID: \"2c6446ac-f2c3-4897-a995-94176302c6b6\") " pod="openshift-dns/dns-default-pj6vq"
Jul 27 17:33:12.511598 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:12.511563    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-skq5m\""
Jul 27 17:33:12.511748 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:12.511622    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-22498\""
Jul 27 17:33:12.511748 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:12.511620    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-x7wp7\""
Jul 27 17:33:12.519345 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:12.519323    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pj6vq"
Jul 27 17:33:12.519345 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:12.519333    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-k5llc"
Jul 27 17:33:12.519531 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:12.519487    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:33:12.690552 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:12.690493    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-cz9hj" podStartSLOduration=18.06712298 podStartE2EDuration="30.69046852s" totalImagesPullingTime="12.62334554s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:32:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:33:11.939311029 +0000 UTC m=+284.426122345" watchObservedRunningTime="2026-07-27 17:33:12.69046852 +0000 UTC m=+285.177279808"
Jul 27 17:33:12.691677 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:12.691004    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pj6vq"]
Jul 27 17:33:12.697707 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:33:12.697675    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c6446ac_f2c3_4897_a995_94176302c6b6.slice/crio-e613db20da70b07cc723c1b2220d7984caa623957a83fdaaeda2405a131b5f84 WatchSource:0}: Error finding container e613db20da70b07cc723c1b2220d7984caa623957a83fdaaeda2405a131b5f84: Status 404 returned error can't find the container with id e613db20da70b07cc723c1b2220d7984caa623957a83fdaaeda2405a131b5f84
Jul 27 17:33:12.721944 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:12.721811    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-k5llc"]
Jul 27 17:33:12.724441 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:12.724417    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b499f6dbb-gq96p"]
Jul 27 17:33:12.725225 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:33:12.725192    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d458fd4_bb2f_4907_9431_226cac065ce0.slice/crio-b1077a7e21fd022117d30824f854dc0e1804651740be116390d5f351f177ca42 WatchSource:0}: Error finding container b1077a7e21fd022117d30824f854dc0e1804651740be116390d5f351f177ca42: Status 404 returned error can't find the container with id b1077a7e21fd022117d30824f854dc0e1804651740be116390d5f351f177ca42
Jul 27 17:33:12.727687 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:33:12.727660    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86ded04d_8b64_408e_8352_7ef795d848a0.slice/crio-918687f15de67659601d118746c5c2cd030e6f13a9d91386b29f20a9b85124eb WatchSource:0}: Error finding container 918687f15de67659601d118746c5c2cd030e6f13a9d91386b29f20a9b85124eb: Status 404 returned error can't find the container with id 918687f15de67659601d118746c5c2cd030e6f13a9d91386b29f20a9b85124eb
Jul 27 17:33:12.926407 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:12.926366    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pj6vq" event={"ID":"2c6446ac-f2c3-4897-a995-94176302c6b6","Type":"ContainerStarted","Data":"e613db20da70b07cc723c1b2220d7984caa623957a83fdaaeda2405a131b5f84"}
Jul 27 17:33:12.927857 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:12.927829    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-k5llc" event={"ID":"3d458fd4-bb2f-4907-9431-226cac065ce0","Type":"ContainerStarted","Data":"b1077a7e21fd022117d30824f854dc0e1804651740be116390d5f351f177ca42"}
Jul 27 17:33:12.930793 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:12.930749    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p" event={"ID":"86ded04d-8b64-408e-8352-7ef795d848a0","Type":"ContainerStarted","Data":"8302b9599b0b6809178eb547234adaa9ac34326bb97ed963963ebf7aada8a34d"}
Jul 27 17:33:12.930882 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:12.930813    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p" event={"ID":"86ded04d-8b64-408e-8352-7ef795d848a0","Type":"ContainerStarted","Data":"918687f15de67659601d118746c5c2cd030e6f13a9d91386b29f20a9b85124eb"}
Jul 27 17:33:12.930882 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:12.930856    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:33:12.949372 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:12.949324    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p" podStartSLOduration=284.949307718 podStartE2EDuration="4m44.949307718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:33:12.94815565 +0000 UTC m=+285.434966995" watchObservedRunningTime="2026-07-27 17:33:12.949307718 +0000 UTC m=+285.436119005"
Jul 27 17:33:14.940941 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:14.940892    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-k5llc" event={"ID":"3d458fd4-bb2f-4907-9431-226cac065ce0","Type":"ContainerStarted","Data":"22d63af07a3f22c2fcecc233b0d9b67c7bfdd97ad0c22564ff340603b96778c9"}
Jul 27 17:33:14.956413 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:14.956316    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-k5llc" podStartSLOduration=262.560117211 podStartE2EDuration="4m23.956297196s" totalImagesPullingTime="1.396179985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:33:14.955929375 +0000 UTC m=+287.442740687" watchObservedRunningTime="2026-07-27 17:33:14.956297196 +0000 UTC m=+287.443108484"
Jul 27 17:33:16.954469 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:16.954421    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64bb7f674d-lsdcl" event={"ID":"e5e65078-6a76-4778-b0b0-0d8fc032a406","Type":"ContainerStarted","Data":"d09ed4a23c5ef6123cabd76ecedfbe1c46b3b69a93f4cc50717dd21ad5f8cd68"}
Jul 27 17:33:16.970533 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:16.969986    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64bb7f674d-lsdcl" podStartSLOduration=2.283262548 podStartE2EDuration="6.969965303s" totalImagesPullingTime="4.686702755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:33:16.968852183 +0000 UTC m=+289.455663486" watchObservedRunningTime="2026-07-27 17:33:16.969965303 +0000 UTC m=+289.456776592"
Jul 27 17:33:17.959793 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:17.959731    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pj6vq" event={"ID":"2c6446ac-f2c3-4897-a995-94176302c6b6","Type":"ContainerStarted","Data":"5470c63272934626a4f89aae30b4cda8ceab0e15a2771073e7642e85378a911d"}
Jul 27 17:33:17.960262 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:17.959903    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pj6vq"
Jul 27 17:33:17.960262 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:17.959964    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pj6vq" event={"ID":"2c6446ac-f2c3-4897-a995-94176302c6b6","Type":"ContainerStarted","Data":"52717bc7ac6b1e6151ff7094bd7dce53dea030a55930706c91d339099b647189"}
Jul 27 17:33:17.977276 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:17.977223    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pj6vq" podStartSLOduration=251.073299763 podStartE2EDuration="4m15.977206796s" totalImagesPullingTime="4.903907033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:33:17.976530755 +0000 UTC m=+290.463342064" watchObservedRunningTime="2026-07-27 17:33:17.977206796 +0000 UTC m=+290.464018083"
Jul 27 17:33:19.806597 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.806563    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-xsxnf"]
Jul 27 17:33:19.810593 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.810569    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-xsxnf"
Jul 27 17:33:19.812777 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.812738    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 27 17:33:19.812910 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.812872    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 27 17:33:19.813669 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.813504    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 27 17:33:19.813669 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.813509    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-qnwh5\""
Jul 27 17:33:19.813669 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.813554    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 27 17:33:19.813669 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.813512    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 27 17:33:19.819498 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.819480    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-xsxnf"]
Jul 27 17:33:19.822948 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.822923    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-pxsf7"]
Jul 27 17:33:19.826065 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.826030    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pxsf7"
Jul 27 17:33:19.827981 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.827951    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 27 17:33:19.828100 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.827999    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 27 17:33:19.828455 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.828435    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 27 17:33:19.828553 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.828491    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-kvhjz\""
Jul 27 17:33:19.837409 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.837381    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-pxsf7"]
Jul 27 17:33:19.841177 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.841156    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ct2bc"]
Jul 27 17:33:19.844714 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.844691    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ct2bc"
Jul 27 17:33:19.844995 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.844967    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m82q\" (UniqueName: \"kubernetes.io/projected/a2877fd1-e9e9-400c-b9b2-e90d0590e13c-kube-api-access-8m82q\") pod \"openshift-state-metrics-b8f49f5c6-xsxnf\" (UID: \"a2877fd1-e9e9-400c-b9b2-e90d0590e13c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-xsxnf"
Jul 27 17:33:19.845175 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.845031    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2877fd1-e9e9-400c-b9b2-e90d0590e13c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-xsxnf\" (UID: \"a2877fd1-e9e9-400c-b9b2-e90d0590e13c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-xsxnf"
Jul 27 17:33:19.845175 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.845059    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2877fd1-e9e9-400c-b9b2-e90d0590e13c-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-xsxnf\" (UID: \"a2877fd1-e9e9-400c-b9b2-e90d0590e13c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-xsxnf"
Jul 27 17:33:19.845175 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.845128    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2877fd1-e9e9-400c-b9b2-e90d0590e13c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-xsxnf\" (UID: \"a2877fd1-e9e9-400c-b9b2-e90d0590e13c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-xsxnf"
Jul 27 17:33:19.846664 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.846647    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jrhwj\""
Jul 27 17:33:19.846804 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.846667    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 27 17:33:19.846804 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.846683    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 27 17:33:19.847165 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.847133    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 27 17:33:19.945693 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.945657    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ccf95fe-63a3-4237-be84-db12771f35cb-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-pxsf7\" (UID: \"3ccf95fe-63a3-4237-be84-db12771f35cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pxsf7"
Jul 27 17:33:19.945887 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.945703    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4e5e4d7c-9bef-4d41-a9b9-22885fe0213b-node-exporter-tls\") pod \"node-exporter-ct2bc\" (UID: \"4e5e4d7c-9bef-4d41-a9b9-22885fe0213b\") " pod="openshift-monitoring/node-exporter-ct2bc"
Jul 27 17:33:19.945887 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.945730    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4e5e4d7c-9bef-4d41-a9b9-22885fe0213b-node-exporter-accelerators-collector-config\") pod \"node-exporter-ct2bc\" (UID: \"4e5e4d7c-9bef-4d41-a9b9-22885fe0213b\") " pod="openshift-monitoring/node-exporter-ct2bc"
Jul 27 17:33:19.945887 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.945771    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dvcs\" (UniqueName: \"kubernetes.io/projected/4e5e4d7c-9bef-4d41-a9b9-22885fe0213b-kube-api-access-2dvcs\") pod \"node-exporter-ct2bc\" (UID: \"4e5e4d7c-9bef-4d41-a9b9-22885fe0213b\") " pod="openshift-monitoring/node-exporter-ct2bc"
Jul 27 17:33:19.945887 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.945799    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m82q\" (UniqueName: \"kubernetes.io/projected/a2877fd1-e9e9-400c-b9b2-e90d0590e13c-kube-api-access-8m82q\") pod \"openshift-state-metrics-b8f49f5c6-xsxnf\" (UID: \"a2877fd1-e9e9-400c-b9b2-e90d0590e13c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-xsxnf"
Jul 27 17:33:19.945887 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.945860    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e5e4d7c-9bef-4d41-a9b9-22885fe0213b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ct2bc\" (UID: \"4e5e4d7c-9bef-4d41-a9b9-22885fe0213b\") " pod="openshift-monitoring/node-exporter-ct2bc"
Jul 27 17:33:19.946123 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.945899    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4e5e4d7c-9bef-4d41-a9b9-22885fe0213b-node-exporter-textfile\") pod \"node-exporter-ct2bc\" (UID: \"4e5e4d7c-9bef-4d41-a9b9-22885fe0213b\") " pod="openshift-monitoring/node-exporter-ct2bc"
Jul 27 17:33:19.946123 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.945937    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e5e4d7c-9bef-4d41-a9b9-22885fe0213b-metrics-client-ca\") pod \"node-exporter-ct2bc\" (UID: \"4e5e4d7c-9bef-4d41-a9b9-22885fe0213b\") " pod="openshift-monitoring/node-exporter-ct2bc"
Jul 27 17:33:19.946123 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.945966    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2877fd1-e9e9-400c-b9b2-e90d0590e13c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-xsxnf\" (UID: \"a2877fd1-e9e9-400c-b9b2-e90d0590e13c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-xsxnf"
Jul 27 17:33:19.946123 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.945994    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2877fd1-e9e9-400c-b9b2-e90d0590e13c-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-xsxnf\" (UID: \"a2877fd1-e9e9-400c-b9b2-e90d0590e13c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-xsxnf"
Jul 27 17:33:19.946123 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.946033    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4e5e4d7c-9bef-4d41-a9b9-22885fe0213b-root\") pod \"node-exporter-ct2bc\" (UID: \"4e5e4d7c-9bef-4d41-a9b9-22885fe0213b\") " pod="openshift-monitoring/node-exporter-ct2bc"
Jul 27 17:33:19.946123 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.946065    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2877fd1-e9e9-400c-b9b2-e90d0590e13c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-xsxnf\" (UID: \"a2877fd1-e9e9-400c-b9b2-e90d0590e13c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-xsxnf"
Jul 27 17:33:19.946123 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.946091    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3ccf95fe-63a3-4237-be84-db12771f35cb-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-pxsf7\" (UID: \"3ccf95fe-63a3-4237-be84-db12771f35cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pxsf7"
Jul 27 17:33:19.946123 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.946107    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ccf95fe-63a3-4237-be84-db12771f35cb-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-pxsf7\" (UID: \"3ccf95fe-63a3-4237-be84-db12771f35cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pxsf7"
Jul 27 17:33:19.946463 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.946129    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4e5e4d7c-9bef-4d41-a9b9-22885fe0213b-node-exporter-wtmp\") pod \"node-exporter-ct2bc\" (UID: \"4e5e4d7c-9bef-4d41-a9b9-22885fe0213b\") " pod="openshift-monitoring/node-exporter-ct2bc"
Jul 27 17:33:19.946463 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.946159    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3ccf95fe-63a3-4237-be84-db12771f35cb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-pxsf7\" (UID: \"3ccf95fe-63a3-4237-be84-db12771f35cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pxsf7"
Jul 27 17:33:19.946463 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.946204    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzfdh\" (UniqueName: \"kubernetes.io/projected/3ccf95fe-63a3-4237-be84-db12771f35cb-kube-api-access-xzfdh\") pod \"kube-state-metrics-56d9f944b4-pxsf7\" (UID: \"3ccf95fe-63a3-4237-be84-db12771f35cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pxsf7"
Jul 27 17:33:19.946463 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.946226    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e5e4d7c-9bef-4d41-a9b9-22885fe0213b-sys\") pod \"node-exporter-ct2bc\" (UID: \"4e5e4d7c-9bef-4d41-a9b9-22885fe0213b\") " pod="openshift-monitoring/node-exporter-ct2bc"
Jul 27 17:33:19.946463 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.946267    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ccf95fe-63a3-4237-be84-db12771f35cb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-pxsf7\" (UID: \"3ccf95fe-63a3-4237-be84-db12771f35cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pxsf7"
Jul 27 17:33:19.946463 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:33:19.946293    2557 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 27 17:33:19.946463 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:33:19.946361    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2877fd1-e9e9-400c-b9b2-e90d0590e13c-openshift-state-metrics-tls podName:a2877fd1-e9e9-400c-b9b2-e90d0590e13c nodeName:}" failed. No retries permitted until 2026-07-27 17:33:20.44634066 +0000 UTC m=+292.933151939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/a2877fd1-e9e9-400c-b9b2-e90d0590e13c-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-xsxnf" (UID: "a2877fd1-e9e9-400c-b9b2-e90d0590e13c") : secret "openshift-state-metrics-tls" not found
Jul 27 17:33:19.946891 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.946869    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2877fd1-e9e9-400c-b9b2-e90d0590e13c-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-xsxnf\" (UID: \"a2877fd1-e9e9-400c-b9b2-e90d0590e13c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-xsxnf"
Jul 27 17:33:19.948630 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.948610    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2877fd1-e9e9-400c-b9b2-e90d0590e13c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-xsxnf\" (UID: \"a2877fd1-e9e9-400c-b9b2-e90d0590e13c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-xsxnf"
Jul 27 17:33:19.957052 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:19.954792    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m82q\" (UniqueName: \"kubernetes.io/projected/a2877fd1-e9e9-400c-b9b2-e90d0590e13c-kube-api-access-8m82q\") pod \"openshift-state-metrics-b8f49f5c6-xsxnf\" (UID: \"a2877fd1-e9e9-400c-b9b2-e90d0590e13c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-xsxnf"
Jul 27 17:33:20.048364 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.047085    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e5e4d7c-9bef-4d41-a9b9-22885fe0213b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ct2bc\" (UID: \"4e5e4d7c-9bef-4d41-a9b9-22885fe0213b\") " pod="openshift-monitoring/node-exporter-ct2bc"
Jul 27 17:33:20.048364 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.047135    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4e5e4d7c-9bef-4d41-a9b9-22885fe0213b-node-exporter-textfile\") pod \"node-exporter-ct2bc\" (UID: \"4e5e4d7c-9bef-4d41-a9b9-22885fe0213b\") " pod="openshift-monitoring/node-exporter-ct2bc"
Jul 27 17:33:20.048364 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.047168    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e5e4d7c-9bef-4d41-a9b9-22885fe0213b-metrics-client-ca\") pod \"node-exporter-ct2bc\" (UID: \"4e5e4d7c-9bef-4d41-a9b9-22885fe0213b\") " pod="openshift-monitoring/node-exporter-ct2bc"
Jul 27 17:33:20.048364 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.047207    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4e5e4d7c-9bef-4d41-a9b9-22885fe0213b-root\") pod \"node-exporter-ct2bc\" (UID: \"4e5e4d7c-9bef-4d41-a9b9-22885fe0213b\") " pod="openshift-monitoring/node-exporter-ct2bc"
Jul 27 17:33:20.048364 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.047251    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3ccf95fe-63a3-4237-be84-db12771f35cb-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-pxsf7\" (UID: \"3ccf95fe-63a3-4237-be84-db12771f35cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pxsf7"
Jul 27 17:33:20.048364 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.047274    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ccf95fe-63a3-4237-be84-db12771f35cb-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-pxsf7\" (UID: \"3ccf95fe-63a3-4237-be84-db12771f35cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pxsf7"
Jul 27 17:33:20.048364 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.047296    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4e5e4d7c-9bef-4d41-a9b9-22885fe0213b-node-exporter-wtmp\") pod \"node-exporter-ct2bc\" (UID: \"4e5e4d7c-9bef-4d41-a9b9-22885fe0213b\") " pod="openshift-monitoring/node-exporter-ct2bc"
Jul 27 17:33:20.048364 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.047319    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3ccf95fe-63a3-4237-be84-db12771f35cb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-pxsf7\" (UID: \"3ccf95fe-63a3-4237-be84-db12771f35cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pxsf7"
Jul 27 17:33:20.048364 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.047348    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzfdh\" (UniqueName: \"kubernetes.io/projected/3ccf95fe-63a3-4237-be84-db12771f35cb-kube-api-access-xzfdh\") pod \"kube-state-metrics-56d9f944b4-pxsf7\" (UID: \"3ccf95fe-63a3-4237-be84-db12771f35cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pxsf7"
Jul 27 17:33:20.048364 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.047346    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4e5e4d7c-9bef-4d41-a9b9-22885fe0213b-root\") pod \"node-exporter-ct2bc\" (UID: \"4e5e4d7c-9bef-4d41-a9b9-22885fe0213b\") " pod="openshift-monitoring/node-exporter-ct2bc"
Jul 27 17:33:20.048364 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.047373    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e5e4d7c-9bef-4d41-a9b9-22885fe0213b-sys\") pod \"node-exporter-ct2bc\" (UID: \"4e5e4d7c-9bef-4d41-a9b9-22885fe0213b\") " pod="openshift-monitoring/node-exporter-ct2bc"
Jul 27 17:33:20.048364 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.047418    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e5e4d7c-9bef-4d41-a9b9-22885fe0213b-sys\") pod \"node-exporter-ct2bc\" (UID: \"4e5e4d7c-9bef-4d41-a9b9-22885fe0213b\") " pod="openshift-monitoring/node-exporter-ct2bc"
Jul 27 17:33:20.048364 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.047442    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ccf95fe-63a3-4237-be84-db12771f35cb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-pxsf7\" (UID: \"3ccf95fe-63a3-4237-be84-db12771f35cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pxsf7"
Jul 27 17:33:20.048364 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.047498    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ccf95fe-63a3-4237-be84-db12771f35cb-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-pxsf7\" (UID: \"3ccf95fe-63a3-4237-be84-db12771f35cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pxsf7"
Jul 27 17:33:20.048364 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.047529    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4e5e4d7c-9bef-4d41-a9b9-22885fe0213b-node-exporter-tls\") pod \"node-exporter-ct2bc\" (UID: \"4e5e4d7c-9bef-4d41-a9b9-22885fe0213b\") " pod="openshift-monitoring/node-exporter-ct2bc"
Jul 27 17:33:20.048364 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.047553    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4e5e4d7c-9bef-4d41-a9b9-22885fe0213b-node-exporter-accelerators-collector-config\") pod \"node-exporter-ct2bc\" (UID: \"4e5e4d7c-9bef-4d41-a9b9-22885fe0213b\") " pod="openshift-monitoring/node-exporter-ct2bc"
Jul 27 17:33:20.049287 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.047581    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dvcs\" (UniqueName: \"kubernetes.io/projected/4e5e4d7c-9bef-4d41-a9b9-22885fe0213b-kube-api-access-2dvcs\") pod \"node-exporter-ct2bc\" (UID: \"4e5e4d7c-9bef-4d41-a9b9-22885fe0213b\") " pod="openshift-monitoring/node-exporter-ct2bc"
Jul 27 17:33:20.049287 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.047641    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4e5e4d7c-9bef-4d41-a9b9-22885fe0213b-node-exporter-textfile\") pod \"node-exporter-ct2bc\" (UID: \"4e5e4d7c-9bef-4d41-a9b9-22885fe0213b\") " pod="openshift-monitoring/node-exporter-ct2bc"
Jul 27 17:33:20.049287 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.047700    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4e5e4d7c-9bef-4d41-a9b9-22885fe0213b-node-exporter-wtmp\") pod \"node-exporter-ct2bc\" (UID: \"4e5e4d7c-9bef-4d41-a9b9-22885fe0213b\") " pod="openshift-monitoring/node-exporter-ct2bc"
Jul 27 17:33:20.049287 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.047764    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3ccf95fe-63a3-4237-be84-db12771f35cb-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-pxsf7\" (UID: \"3ccf95fe-63a3-4237-be84-db12771f35cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pxsf7"
Jul 27 17:33:20.049287 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:33:20.048087    2557 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 27 17:33:20.049287 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:33:20.048173    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e5e4d7c-9bef-4d41-a9b9-22885fe0213b-node-exporter-tls podName:4e5e4d7c-9bef-4d41-a9b9-22885fe0213b nodeName:}" failed. No retries permitted until 2026-07-27 17:33:20.548154279 +0000 UTC m=+293.034965548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/4e5e4d7c-9bef-4d41-a9b9-22885fe0213b-node-exporter-tls") pod "node-exporter-ct2bc" (UID: "4e5e4d7c-9bef-4d41-a9b9-22885fe0213b") : secret "node-exporter-tls" not found
Jul 27 17:33:20.049287 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.048204    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e5e4d7c-9bef-4d41-a9b9-22885fe0213b-metrics-client-ca\") pod \"node-exporter-ct2bc\" (UID: \"4e5e4d7c-9bef-4d41-a9b9-22885fe0213b\") " pod="openshift-monitoring/node-exporter-ct2bc"
Jul 27 17:33:20.049287 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.048353    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3ccf95fe-63a3-4237-be84-db12771f35cb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-pxsf7\" (UID: \"3ccf95fe-63a3-4237-be84-db12771f35cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pxsf7"
Jul 27 17:33:20.049287 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.049041    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4e5e4d7c-9bef-4d41-a9b9-22885fe0213b-node-exporter-accelerators-collector-config\") pod \"node-exporter-ct2bc\" (UID: \"4e5e4d7c-9bef-4d41-a9b9-22885fe0213b\") " pod="openshift-monitoring/node-exporter-ct2bc"
Jul 27 17:33:20.049287 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.049112    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ccf95fe-63a3-4237-be84-db12771f35cb-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-pxsf7\" (UID: \"3ccf95fe-63a3-4237-be84-db12771f35cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pxsf7"
Jul 27 17:33:20.050923 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.050862    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e5e4d7c-9bef-4d41-a9b9-22885fe0213b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ct2bc\" (UID: \"4e5e4d7c-9bef-4d41-a9b9-22885fe0213b\") " pod="openshift-monitoring/node-exporter-ct2bc"
Jul 27 17:33:20.051311 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.051286    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ccf95fe-63a3-4237-be84-db12771f35cb-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-pxsf7\" (UID: \"3ccf95fe-63a3-4237-be84-db12771f35cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pxsf7"
Jul 27 17:33:20.051399 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.051293    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ccf95fe-63a3-4237-be84-db12771f35cb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-pxsf7\" (UID: \"3ccf95fe-63a3-4237-be84-db12771f35cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pxsf7"
Jul 27 17:33:20.057613 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.057560    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzfdh\" (UniqueName: \"kubernetes.io/projected/3ccf95fe-63a3-4237-be84-db12771f35cb-kube-api-access-xzfdh\") pod \"kube-state-metrics-56d9f944b4-pxsf7\" (UID: \"3ccf95fe-63a3-4237-be84-db12771f35cb\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pxsf7"
Jul 27 17:33:20.058231 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.058209    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dvcs\" (UniqueName: \"kubernetes.io/projected/4e5e4d7c-9bef-4d41-a9b9-22885fe0213b-kube-api-access-2dvcs\") pod \"node-exporter-ct2bc\" (UID: \"4e5e4d7c-9bef-4d41-a9b9-22885fe0213b\") " pod="openshift-monitoring/node-exporter-ct2bc"
Jul 27 17:33:20.137274 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.137248    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pxsf7"
Jul 27 17:33:20.451282 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.451231    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2877fd1-e9e9-400c-b9b2-e90d0590e13c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-xsxnf\" (UID: \"a2877fd1-e9e9-400c-b9b2-e90d0590e13c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-xsxnf"
Jul 27 17:33:20.453945 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.453922    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2877fd1-e9e9-400c-b9b2-e90d0590e13c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-xsxnf\" (UID: \"a2877fd1-e9e9-400c-b9b2-e90d0590e13c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-xsxnf"
Jul 27 17:33:20.552299 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.552257    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4e5e4d7c-9bef-4d41-a9b9-22885fe0213b-node-exporter-tls\") pod \"node-exporter-ct2bc\" (UID: \"4e5e4d7c-9bef-4d41-a9b9-22885fe0213b\") " pod="openshift-monitoring/node-exporter-ct2bc"
Jul 27 17:33:20.554892 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.554864    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4e5e4d7c-9bef-4d41-a9b9-22885fe0213b-node-exporter-tls\") pod \"node-exporter-ct2bc\" (UID: \"4e5e4d7c-9bef-4d41-a9b9-22885fe0213b\") " pod="openshift-monitoring/node-exporter-ct2bc"
Jul 27 17:33:20.722844 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.722767    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-xsxnf"
Jul 27 17:33:20.754919 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.754883    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ct2bc"
Jul 27 17:33:20.890998 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.890955    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 17:33:20.895946 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.895924    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:20.898785 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.898745    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 27 17:33:20.898971 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.898923    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 27 17:33:20.899433 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.899413    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 27 17:33:20.899542 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.899520    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 27 17:33:20.899810 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.899790    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wf69w\""
Jul 27 17:33:20.900386 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.900366    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 27 17:33:20.900488 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.900409    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 27 17:33:20.900561 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.900408    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 27 17:33:20.901163 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.901105    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 27 17:33:20.909717 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.909682    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 27 17:33:20.928823 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.928793    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 17:33:20.955432 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.955400    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b06529a0-2a71-4cfa-a22f-096a4c144227-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:20.955581 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.955445    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b06529a0-2a71-4cfa-a22f-096a4c144227-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:20.955581 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.955478    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b06529a0-2a71-4cfa-a22f-096a4c144227-config-volume\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:20.955581 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.955512    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b06529a0-2a71-4cfa-a22f-096a4c144227-config-out\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:20.955581 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.955540    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b06529a0-2a71-4cfa-a22f-096a4c144227-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:20.955581 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.955576    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b06529a0-2a71-4cfa-a22f-096a4c144227-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:20.955822 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.955634    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b06529a0-2a71-4cfa-a22f-096a4c144227-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:20.955822 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.955674    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b06529a0-2a71-4cfa-a22f-096a4c144227-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:20.955822 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.955698    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b06529a0-2a71-4cfa-a22f-096a4c144227-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:20.955822 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.955784    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b06529a0-2a71-4cfa-a22f-096a4c144227-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:20.955822 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.955819    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w256k\" (UniqueName: \"kubernetes.io/projected/b06529a0-2a71-4cfa-a22f-096a4c144227-kube-api-access-w256k\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:20.956027 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.955862    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b06529a0-2a71-4cfa-a22f-096a4c144227-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:20.956027 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:20.955883    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b06529a0-2a71-4cfa-a22f-096a4c144227-web-config\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:21.056846 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:21.056735    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w256k\" (UniqueName: \"kubernetes.io/projected/b06529a0-2a71-4cfa-a22f-096a4c144227-kube-api-access-w256k\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:21.056846 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:21.056814    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b06529a0-2a71-4cfa-a22f-096a4c144227-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:21.056846 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:21.056848    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b06529a0-2a71-4cfa-a22f-096a4c144227-web-config\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:21.057154 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:21.056872    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b06529a0-2a71-4cfa-a22f-096a4c144227-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:21.057154 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:21.056899    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b06529a0-2a71-4cfa-a22f-096a4c144227-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:21.057154 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:21.056925    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b06529a0-2a71-4cfa-a22f-096a4c144227-config-volume\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:21.057154 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:21.056956    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b06529a0-2a71-4cfa-a22f-096a4c144227-config-out\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:21.057154 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:21.056983    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b06529a0-2a71-4cfa-a22f-096a4c144227-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:21.057154 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:21.057019    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b06529a0-2a71-4cfa-a22f-096a4c144227-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:21.057154 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:21.057048    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b06529a0-2a71-4cfa-a22f-096a4c144227-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:21.057154 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:21.057070    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b06529a0-2a71-4cfa-a22f-096a4c144227-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:21.057154 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:21.057093    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b06529a0-2a71-4cfa-a22f-096a4c144227-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:21.057610 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:21.057161    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b06529a0-2a71-4cfa-a22f-096a4c144227-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:21.057610 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:21.057479    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b06529a0-2a71-4cfa-a22f-096a4c144227-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:21.058483 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:21.057925    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b06529a0-2a71-4cfa-a22f-096a4c144227-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:21.058770 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:21.058736    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b06529a0-2a71-4cfa-a22f-096a4c144227-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:21.061151 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:21.061132    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64bb7f674d-lsdcl"
Jul 27 17:33:21.061426 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:21.061392    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64bb7f674d-lsdcl"
Jul 27 17:33:21.061628 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:21.061503    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b06529a0-2a71-4cfa-a22f-096a4c144227-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:21.061878 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:21.061653    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b06529a0-2a71-4cfa-a22f-096a4c144227-web-config\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:21.062041 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:21.061730    2557 patch_prober.go:28] interesting pod/console-64bb7f674d-lsdcl container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.16:8443/health\": dial tcp 10.132.0.16:8443: connect: connection refused" start-of-body=
Jul 27 17:33:21.062276 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:21.062252    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-64bb7f674d-lsdcl" podUID="e5e65078-6a76-4778-b0b0-0d8fc032a406" containerName="console" probeResult="failure" output="Get \"https://10.132.0.16:8443/health\": dial tcp 10.132.0.16:8443: connect: connection refused"
Jul 27 17:33:21.062427 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:21.062009    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b06529a0-2a71-4cfa-a22f-096a4c144227-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:21.062749 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:21.062706    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b06529a0-2a71-4cfa-a22f-096a4c144227-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:21.063267 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:21.062987    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b06529a0-2a71-4cfa-a22f-096a4c144227-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:21.063267 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:21.063020    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b06529a0-2a71-4cfa-a22f-096a4c144227-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:21.063414 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:21.063290    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b06529a0-2a71-4cfa-a22f-096a4c144227-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:21.063473 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:21.063432    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b06529a0-2a71-4cfa-a22f-096a4c144227-config-volume\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:21.063535 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:21.063509    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b06529a0-2a71-4cfa-a22f-096a4c144227-config-out\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:21.065584 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:21.065557    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w256k\" (UniqueName: \"kubernetes.io/projected/b06529a0-2a71-4cfa-a22f-096a4c144227-kube-api-access-w256k\") pod \"alertmanager-main-0\" (UID: \"b06529a0-2a71-4cfa-a22f-096a4c144227\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:21.208964 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:21.208920    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:33:22.114131 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:33:22.114087    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-mzhnc" podUID="57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec"
Jul 27 17:33:24.884203 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:33:24.884152    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e5e4d7c_9bef_4d41_a9b9_22885fe0213b.slice/crio-3c42658f14089d1e2ffece8416bbe4aa280e61c808faa8a42f6fc48a55b9d258 WatchSource:0}: Error finding container 3c42658f14089d1e2ffece8416bbe4aa280e61c808faa8a42f6fc48a55b9d258: Status 404 returned error can't find the container with id 3c42658f14089d1e2ffece8416bbe4aa280e61c808faa8a42f6fc48a55b9d258
Jul 27 17:33:24.988909 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:24.988770    2557 generic.go:356] "Generic (PLEG): container finished" podID="c44c2dc2-907f-4753-8afe-9b58dbf89308" containerID="366d51b71e53734de7354d156e9b61fada91e50907d78de598bcefd3c81bb279" exitCode=1
Jul 27 17:33:24.988909 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:24.988808    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z" event={"ID":"c44c2dc2-907f-4753-8afe-9b58dbf89308","Type":"ContainerDied","Data":"366d51b71e53734de7354d156e9b61fada91e50907d78de598bcefd3c81bb279"}
Jul 27 17:33:24.988909 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:24.988887    2557 scope.go:117] "RemoveContainer" containerID="66638458a8a61af91f91551c19860e8fae99877c74ff00e4f71400df95cf086a"
Jul 27 17:33:24.989425 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:24.989210    2557 scope.go:117] "RemoveContainer" containerID="366d51b71e53734de7354d156e9b61fada91e50907d78de598bcefd3c81bb279"
Jul 27 17:33:24.989555 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:33:24.989447    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-69665bc4b4-dlg7z_open-cluster-management-agent-addon(c44c2dc2-907f-4753-8afe-9b58dbf89308)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z" podUID="c44c2dc2-907f-4753-8afe-9b58dbf89308"
Jul 27 17:33:24.991318 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:24.991265    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ct2bc" event={"ID":"4e5e4d7c-9bef-4d41-a9b9-22885fe0213b","Type":"ContainerStarted","Data":"3c42658f14089d1e2ffece8416bbe4aa280e61c808faa8a42f6fc48a55b9d258"}
Jul 27 17:33:25.005804 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:25.005780    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 17:33:25.017716 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:25.017694    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-pxsf7"]
Jul 27 17:33:25.035440 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:25.035418    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-xsxnf"]
Jul 27 17:33:25.087226 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:33:25.087189    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb06529a0_2a71_4cfa_a22f_096a4c144227.slice/crio-de30c8c16f35ecbe086f63580d92dd0ccc120257aa13e901e7ef12d0cde2d649 WatchSource:0}: Error finding container de30c8c16f35ecbe086f63580d92dd0ccc120257aa13e901e7ef12d0cde2d649: Status 404 returned error can't find the container with id de30c8c16f35ecbe086f63580d92dd0ccc120257aa13e901e7ef12d0cde2d649
Jul 27 17:33:25.087981 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:33:25.087959    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ccf95fe_63a3_4237_be84_db12771f35cb.slice/crio-71b25e8e8748943136fb7f5fc08ebc53b0ed579af0133e33078f5d5c921381b7 WatchSource:0}: Error finding container 71b25e8e8748943136fb7f5fc08ebc53b0ed579af0133e33078f5d5c921381b7: Status 404 returned error can't find the container with id 71b25e8e8748943136fb7f5fc08ebc53b0ed579af0133e33078f5d5c921381b7
Jul 27 17:33:25.088991 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:33:25.088832    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2877fd1_e9e9_400c_b9b2_e90d0590e13c.slice/crio-f6c391b56ecd7de82caa89e123182d0221c9627a7603cab657e1652218a1ce61 WatchSource:0}: Error finding container f6c391b56ecd7de82caa89e123182d0221c9627a7603cab657e1652218a1ce61: Status 404 returned error can't find the container with id f6c391b56ecd7de82caa89e123182d0221c9627a7603cab657e1652218a1ce61
Jul 27 17:33:25.320593 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:25.320560    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64bb7f674d-lsdcl"]
Jul 27 17:33:25.629941 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:25.629855    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z"
Jul 27 17:33:25.953287 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:25.953249    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 17:33:25.958891 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:25.958868    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:25.961810 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:25.961560    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 27 17:33:25.962772 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:25.962421    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 27 17:33:25.964166 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:25.962644    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 27 17:33:25.964166 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:25.963077    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-stqsh\""
Jul 27 17:33:25.964166 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:25.963143    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 27 17:33:25.964166 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:25.963263    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 27 17:33:25.964166 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:25.963317    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 27 17:33:25.964797 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:25.963423    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 27 17:33:25.965093 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:25.963511    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 27 17:33:25.966723 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:25.965627    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 27 17:33:25.970508 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:25.970004    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-b621lv4oqcmq1\""
Jul 27 17:33:25.970508 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:25.970212    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 27 17:33:25.970508 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:25.970470    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 27 17:33:25.973964 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:25.971341    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 27 17:33:25.973964 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:25.971422    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 27 17:33:25.973964 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:25.971723    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 27 17:33:25.974431 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:25.974412    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 27 17:33:25.978900 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:25.978811    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 17:33:25.999553 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:25.999521    2557 scope.go:117] "RemoveContainer" containerID="366d51b71e53734de7354d156e9b61fada91e50907d78de598bcefd3c81bb279"
Jul 27 17:33:25.999996 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:33:25.999969    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-69665bc4b4-dlg7z_open-cluster-management-agent-addon(c44c2dc2-907f-4753-8afe-9b58dbf89308)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z" podUID="c44c2dc2-907f-4753-8afe-9b58dbf89308"
Jul 27 17:33:26.003852 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.003807    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db45ee80-77ff-49f5-a4ab-698f2ed851e8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.004095 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.004081    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/db45ee80-77ff-49f5-a4ab-698f2ed851e8-config\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.004317 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.004275    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/db45ee80-77ff-49f5-a4ab-698f2ed851e8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.004456 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.004441    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/db45ee80-77ff-49f5-a4ab-698f2ed851e8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.004585 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.004574    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db45ee80-77ff-49f5-a4ab-698f2ed851e8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.004909 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.004890    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/db45ee80-77ff-49f5-a4ab-698f2ed851e8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.005145 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.005113    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/db45ee80-77ff-49f5-a4ab-698f2ed851e8-web-config\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.005831 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.005532    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/db45ee80-77ff-49f5-a4ab-698f2ed851e8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.006593 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.006362    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/db45ee80-77ff-49f5-a4ab-698f2ed851e8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.006593 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.005032    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b06529a0-2a71-4cfa-a22f-096a4c144227","Type":"ContainerStarted","Data":"de30c8c16f35ecbe086f63580d92dd0ccc120257aa13e901e7ef12d0cde2d649"}
Jul 27 17:33:26.006593 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.006408    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db45ee80-77ff-49f5-a4ab-698f2ed851e8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.006593 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.006467    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db45ee80-77ff-49f5-a4ab-698f2ed851e8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.006593 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.006553    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/db45ee80-77ff-49f5-a4ab-698f2ed851e8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.006593 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.006594    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/db45ee80-77ff-49f5-a4ab-698f2ed851e8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.006964 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.006620    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94qkh\" (UniqueName: \"kubernetes.io/projected/db45ee80-77ff-49f5-a4ab-698f2ed851e8-kube-api-access-94qkh\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.006964 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.006649    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/db45ee80-77ff-49f5-a4ab-698f2ed851e8-config-out\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.006964 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.006688    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/db45ee80-77ff-49f5-a4ab-698f2ed851e8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.006964 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.006714    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db45ee80-77ff-49f5-a4ab-698f2ed851e8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.006964 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.006738    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db45ee80-77ff-49f5-a4ab-698f2ed851e8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.006964 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.006811    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/db45ee80-77ff-49f5-a4ab-698f2ed851e8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.006964 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.006841    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/db45ee80-77ff-49f5-a4ab-698f2ed851e8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.008087 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.008065    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-s7hbx" event={"ID":"823de302-a6b7-4e92-8cea-80322d2507ce","Type":"ContainerStarted","Data":"e7b4038d8d970629cf2b684cb912f6fdf698542120159eaefbb6a9d21d26e9fa"}
Jul 27 17:33:26.008577 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.008559    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-s7hbx"
Jul 27 17:33:26.011381 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.011321    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pxsf7" event={"ID":"3ccf95fe-63a3-4237-be84-db12771f35cb","Type":"ContainerStarted","Data":"71b25e8e8748943136fb7f5fc08ebc53b0ed579af0133e33078f5d5c921381b7"}
Jul 27 17:33:26.014876 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.014833    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-xsxnf" event={"ID":"a2877fd1-e9e9-400c-b9b2-e90d0590e13c","Type":"ContainerStarted","Data":"d3293a7cafba174d57950fe68d5e851b9bad2e28e366e826a46b83f57918b741"}
Jul 27 17:33:26.015153 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.014900    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-xsxnf" event={"ID":"a2877fd1-e9e9-400c-b9b2-e90d0590e13c","Type":"ContainerStarted","Data":"6cf4656aabcce127cc541ae5b968b2e03c4a8035228829a6f78c00c4c9deb680"}
Jul 27 17:33:26.015153 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.014933    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-xsxnf" event={"ID":"a2877fd1-e9e9-400c-b9b2-e90d0590e13c","Type":"ContainerStarted","Data":"f6c391b56ecd7de82caa89e123182d0221c9627a7603cab657e1652218a1ce61"}
Jul 27 17:33:26.020973 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.020924    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-s7hbx"
Jul 27 17:33:26.080139 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.080054    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-s7hbx" podStartSLOduration=2.39628724 podStartE2EDuration="20.080031374s" totalImagesPullingTime="17.683744134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:33:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:33:26.078964341 +0000 UTC m=+298.565775628" watchObservedRunningTime="2026-07-27 17:33:26.080031374 +0000 UTC m=+298.566842661"
Jul 27 17:33:26.108853 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.107659    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/db45ee80-77ff-49f5-a4ab-698f2ed851e8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.108853 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.107716    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/db45ee80-77ff-49f5-a4ab-698f2ed851e8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.108853 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.107742    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94qkh\" (UniqueName: \"kubernetes.io/projected/db45ee80-77ff-49f5-a4ab-698f2ed851e8-kube-api-access-94qkh\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.108853 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.107794    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/db45ee80-77ff-49f5-a4ab-698f2ed851e8-config-out\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.108853 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.107838    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/db45ee80-77ff-49f5-a4ab-698f2ed851e8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.108853 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.107879    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db45ee80-77ff-49f5-a4ab-698f2ed851e8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.108853 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.107904    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db45ee80-77ff-49f5-a4ab-698f2ed851e8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.108853 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.107951    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/db45ee80-77ff-49f5-a4ab-698f2ed851e8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.108853 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.107979    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/db45ee80-77ff-49f5-a4ab-698f2ed851e8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.108853 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.108003    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db45ee80-77ff-49f5-a4ab-698f2ed851e8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.108853 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.108031    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/db45ee80-77ff-49f5-a4ab-698f2ed851e8-config\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.108853 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.108075    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/db45ee80-77ff-49f5-a4ab-698f2ed851e8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.108853 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.108104    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/db45ee80-77ff-49f5-a4ab-698f2ed851e8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.108853 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.108129    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db45ee80-77ff-49f5-a4ab-698f2ed851e8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.108853 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.108173    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/db45ee80-77ff-49f5-a4ab-698f2ed851e8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.108853 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.108209    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/db45ee80-77ff-49f5-a4ab-698f2ed851e8-web-config\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.108853 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.108242    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/db45ee80-77ff-49f5-a4ab-698f2ed851e8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.115299 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.108276    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/db45ee80-77ff-49f5-a4ab-698f2ed851e8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.115299 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.108304    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db45ee80-77ff-49f5-a4ab-698f2ed851e8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.115299 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.108351    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db45ee80-77ff-49f5-a4ab-698f2ed851e8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.115299 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.110413    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/db45ee80-77ff-49f5-a4ab-698f2ed851e8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.115299 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.112466    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/db45ee80-77ff-49f5-a4ab-698f2ed851e8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.115299 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.112658    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/db45ee80-77ff-49f5-a4ab-698f2ed851e8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.115299 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.114297    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db45ee80-77ff-49f5-a4ab-698f2ed851e8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.119148 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.116010    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/db45ee80-77ff-49f5-a4ab-698f2ed851e8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.119148 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.117491    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db45ee80-77ff-49f5-a4ab-698f2ed851e8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.119148 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.117853    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/db45ee80-77ff-49f5-a4ab-698f2ed851e8-config-out\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.119148 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.118269    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db45ee80-77ff-49f5-a4ab-698f2ed851e8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.119148 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.118912    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db45ee80-77ff-49f5-a4ab-698f2ed851e8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.135165 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.135073    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/db45ee80-77ff-49f5-a4ab-698f2ed851e8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.142636 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.141651    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/db45ee80-77ff-49f5-a4ab-698f2ed851e8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.142636 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.142169    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/db45ee80-77ff-49f5-a4ab-698f2ed851e8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.142636 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.142583    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/db45ee80-77ff-49f5-a4ab-698f2ed851e8-web-config\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.147798 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.144151    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/db45ee80-77ff-49f5-a4ab-698f2ed851e8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.158563 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.158535    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94qkh\" (UniqueName: \"kubernetes.io/projected/db45ee80-77ff-49f5-a4ab-698f2ed851e8-kube-api-access-94qkh\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.159424 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.159361    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/db45ee80-77ff-49f5-a4ab-698f2ed851e8-config\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.161555 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.161496    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/db45ee80-77ff-49f5-a4ab-698f2ed851e8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.166812 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.165744    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/db45ee80-77ff-49f5-a4ab-698f2ed851e8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.166812 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.165883    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/db45ee80-77ff-49f5-a4ab-698f2ed851e8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.167945 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.167897    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/db45ee80-77ff-49f5-a4ab-698f2ed851e8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"db45ee80-77ff-49f5-a4ab-698f2ed851e8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.288854 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.288769    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:26.521473 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:26.521088    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 17:33:27.407923 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:33:27.407883    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb45ee80_77ff_49f5_a4ab_698f2ed851e8.slice/crio-07aad0742e2406acd33454027f803d5684ce1127717179f2cfffa8c94afb9c7a WatchSource:0}: Error finding container 07aad0742e2406acd33454027f803d5684ce1127717179f2cfffa8c94afb9c7a: Status 404 returned error can't find the container with id 07aad0742e2406acd33454027f803d5684ce1127717179f2cfffa8c94afb9c7a
Jul 27 17:33:27.966367 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:27.966336    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pj6vq"
Jul 27 17:33:28.027419 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:28.027282    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db45ee80-77ff-49f5-a4ab-698f2ed851e8","Type":"ContainerStarted","Data":"07aad0742e2406acd33454027f803d5684ce1127717179f2cfffa8c94afb9c7a"}
Jul 27 17:33:28.206855 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:28.206772    2557 kubelet.go:1648] "Image garbage collection succeeded"
Jul 27 17:33:29.033963 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:29.033921    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-xsxnf" event={"ID":"a2877fd1-e9e9-400c-b9b2-e90d0590e13c","Type":"ContainerStarted","Data":"11f70524a86488986171a501c1d65ea31c4534d655ab68dadf2516103b4c0c75"}
Jul 27 17:33:29.035994 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:29.035964    2557 generic.go:356] "Generic (PLEG): container finished" podID="4e5e4d7c-9bef-4d41-a9b9-22885fe0213b" containerID="d7a7b1c6a98db7bbc8873199442ea2fea5f048f948f0d70fb22ee8478d463264" exitCode=0
Jul 27 17:33:29.036130 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:29.035993    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ct2bc" event={"ID":"4e5e4d7c-9bef-4d41-a9b9-22885fe0213b","Type":"ContainerDied","Data":"d7a7b1c6a98db7bbc8873199442ea2fea5f048f948f0d70fb22ee8478d463264"}
Jul 27 17:33:29.037985 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:29.037956    2557 generic.go:356] "Generic (PLEG): container finished" podID="b06529a0-2a71-4cfa-a22f-096a4c144227" containerID="b8ca52661a93b3335176b7e899732911e0139943904725ce70c9719d1fff5ebe" exitCode=0
Jul 27 17:33:29.038125 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:29.037992    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b06529a0-2a71-4cfa-a22f-096a4c144227","Type":"ContainerDied","Data":"b8ca52661a93b3335176b7e899732911e0139943904725ce70c9719d1fff5ebe"}
Jul 27 17:33:29.040597 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:29.040572    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pxsf7" event={"ID":"3ccf95fe-63a3-4237-be84-db12771f35cb","Type":"ContainerStarted","Data":"e54f9be999615ccfe00675230555359621ce31642b6a9ab5c56c206e3f054df3"}
Jul 27 17:33:29.040699 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:29.040627    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pxsf7" event={"ID":"3ccf95fe-63a3-4237-be84-db12771f35cb","Type":"ContainerStarted","Data":"431b5a56bc0657d3d3049470851c98d33279e54810352f28222308be335a381b"}
Jul 27 17:33:29.040699 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:29.040666    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pxsf7" event={"ID":"3ccf95fe-63a3-4237-be84-db12771f35cb","Type":"ContainerStarted","Data":"f58fbfe23abb54afd1ba37bd1d6c7eff269452d66b22a68ec6a5ab047ca5d4c3"}
Jul 27 17:33:29.052838 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:29.052783    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-xsxnf" podStartSLOduration=7.522102476 podStartE2EDuration="10.052749169s" totalImagesPullingTime="2.530646693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:33:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:33:29.050489921 +0000 UTC m=+301.537301209" watchObservedRunningTime="2026-07-27 17:33:29.052749169 +0000 UTC m=+301.539560460"
Jul 27 17:33:29.067234 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:29.067174    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-pxsf7" podStartSLOduration=7.406857394 podStartE2EDuration="10.067156905s" totalImagesPullingTime="2.660299511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:33:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:33:29.065436154 +0000 UTC m=+301.552247466" watchObservedRunningTime="2026-07-27 17:33:29.067156905 +0000 UTC m=+301.553968191"
Jul 27 17:33:29.328899 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:29.328859    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b499f6dbb-gq96p"]
Jul 27 17:33:29.334717 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:29.334674    2557 patch_prober.go:28] interesting pod/image-registry-6b499f6dbb-gq96p container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 27 17:33:29.334912 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:29.334736    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p" podUID="86ded04d-8b64-408e-8352-7ef795d848a0" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:33:30.013877 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:30.013842    2557 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z"
Jul 27 17:33:30.014387 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:30.014368    2557 scope.go:117] "RemoveContainer" containerID="366d51b71e53734de7354d156e9b61fada91e50907d78de598bcefd3c81bb279"
Jul 27 17:33:30.014621 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:33:30.014592    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-69665bc4b4-dlg7z_open-cluster-management-agent-addon(c44c2dc2-907f-4753-8afe-9b58dbf89308)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z" podUID="c44c2dc2-907f-4753-8afe-9b58dbf89308"
Jul 27 17:33:30.052263 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:30.052231    2557 generic.go:356] "Generic (PLEG): container finished" podID="db45ee80-77ff-49f5-a4ab-698f2ed851e8" containerID="4e0e44cfb50475bba18958e44957b7d820fb27b4ed26c7a6f6c9a98fbbd38c53" exitCode=0
Jul 27 17:33:30.052697 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:30.052335    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db45ee80-77ff-49f5-a4ab-698f2ed851e8","Type":"ContainerDied","Data":"4e0e44cfb50475bba18958e44957b7d820fb27b4ed26c7a6f6c9a98fbbd38c53"}
Jul 27 17:33:30.053934 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:30.053908    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:33:30.058928 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:30.058555    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ct2bc" event={"ID":"4e5e4d7c-9bef-4d41-a9b9-22885fe0213b","Type":"ContainerStarted","Data":"8e37e959e7dadb2168eacc9ca3ca4334bf31e7dac852aef59001b9d9e7443eef"}
Jul 27 17:33:30.058928 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:30.058617    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ct2bc" event={"ID":"4e5e4d7c-9bef-4d41-a9b9-22885fe0213b","Type":"ContainerStarted","Data":"e198244341dfc1b461cf28aca7f00da62b4bda99be05e9e54ce41f591008a7f6"}
Jul 27 17:33:30.102476 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:30.102410    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ct2bc" podStartSLOduration=8.101133727 podStartE2EDuration="11.102377328s" totalImagesPullingTime="2.88115039s" totalInitContainerRuntime="120.093211ms" isStatefulPod=true podCreationTimestamp="2026-07-27 17:33:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:33:30.100682301 +0000 UTC m=+302.587493605" watchObservedRunningTime="2026-07-27 17:33:30.102377328 +0000 UTC m=+302.589188616"
Jul 27 17:33:32.073387 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:32.073309    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b06529a0-2a71-4cfa-a22f-096a4c144227","Type":"ContainerStarted","Data":"c790195147ae906623232c6991bc7c0683e792751ad27e48c4330234e4eeb683"}
Jul 27 17:33:32.073749 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:32.073496    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b06529a0-2a71-4cfa-a22f-096a4c144227","Type":"ContainerStarted","Data":"969934dc9eae3bdd90a738ecd8a0f45b8a5ed90cc38b7dc1c4893868a7e59cf4"}
Jul 27 17:33:32.073749 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:32.073547    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b06529a0-2a71-4cfa-a22f-096a4c144227","Type":"ContainerStarted","Data":"75b9bbb1c94e0576bc4812244a65bf853398ca5ceecbbbb271f9f651dea2175c"}
Jul 27 17:33:33.083654 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:33.083614    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b06529a0-2a71-4cfa-a22f-096a4c144227","Type":"ContainerStarted","Data":"69d822e6a2620245798298edb075f86a6e436d5e12de6cc1372815562873d969"}
Jul 27 17:33:33.084131 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:33.083692    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b06529a0-2a71-4cfa-a22f-096a4c144227","Type":"ContainerStarted","Data":"42cef633bca4a8838d22ca4974719e6ff222a5bd4619f82eae822c0294ea2fa7"}
Jul 27 17:33:33.113388 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:33.113345    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:33:35.096366 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:35.096322    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db45ee80-77ff-49f5-a4ab-698f2ed851e8","Type":"ContainerStarted","Data":"4494f88b04ca891bbce3ce9f9b28e02fe5bb1162b14e839b168a015530c7d8ba"}
Jul 27 17:33:35.096952 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:35.096419    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db45ee80-77ff-49f5-a4ab-698f2ed851e8","Type":"ContainerStarted","Data":"48ef9c07553282c4a40160d2a1c3852d5d32a4a046ee0e6ac1847dce995a1159"}
Jul 27 17:33:35.100570 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:35.100524    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b06529a0-2a71-4cfa-a22f-096a4c144227","Type":"ContainerStarted","Data":"7084d5d945c751bfa9546555af36e1d839bc1657d27be1e33f8f97eb8ba7eece"}
Jul 27 17:33:35.123707 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:35.123648    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=6.550538657 podStartE2EDuration="15.123629136s" totalImagesPullingTime="8.555559972s" totalInitContainerRuntime="17.530507ms" isStatefulPod=false podCreationTimestamp="2026-07-27 17:33:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:33:35.122988479 +0000 UTC m=+307.609799791" watchObservedRunningTime="2026-07-27 17:33:35.123629136 +0000 UTC m=+307.610440423"
Jul 27 17:33:38.118545 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:38.118506    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db45ee80-77ff-49f5-a4ab-698f2ed851e8","Type":"ContainerStarted","Data":"afcad5769c1735aba8d8814fdecda2ea2551f393641da12e9947fd4185c18f88"}
Jul 27 17:33:38.119019 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:38.118581    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db45ee80-77ff-49f5-a4ab-698f2ed851e8","Type":"ContainerStarted","Data":"3927e9a9d72a0607fde42b2fee84b441fd4f3e47770c36891719b7e6b0d0657d"}
Jul 27 17:33:38.119019 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:38.118619    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db45ee80-77ff-49f5-a4ab-698f2ed851e8","Type":"ContainerStarted","Data":"4a9c284ca9670b31d03ddc04ddc090895598a29081797a21146f3cba96dfbfdc"}
Jul 27 17:33:38.119019 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:38.118658    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"db45ee80-77ff-49f5-a4ab-698f2ed851e8","Type":"ContainerStarted","Data":"df4f87d32b9fb71a60d5249a0c14c08206d0fc8b5f4ca3ed316a3ce89e55a017"}
Jul 27 17:33:38.167303 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:38.167249    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.745182188 podStartE2EDuration="13.167228064s" totalImagesPullingTime="8.401994476s" totalInitContainerRuntime="20.0514ms" isStatefulPod=false podCreationTimestamp="2026-07-27 17:33:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:33:38.166197845 +0000 UTC m=+310.653009133" watchObservedRunningTime="2026-07-27 17:33:38.167228064 +0000 UTC m=+310.654039351"
Jul 27 17:33:39.333161 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:39.333133    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:33:41.112992 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:41.112957    2557 scope.go:117] "RemoveContainer" containerID="366d51b71e53734de7354d156e9b61fada91e50907d78de598bcefd3c81bb279"
Jul 27 17:33:41.289904 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:41.289867    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:33:42.135364 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:42.135331    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z" event={"ID":"c44c2dc2-907f-4753-8afe-9b58dbf89308","Type":"ContainerStarted","Data":"59c2e66db81d91650c8209cbaba615daac0d6f7dd3b802c54f46b5fceee8d345"}
Jul 27 17:33:42.135774 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:42.135606    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z"
Jul 27 17:33:42.137146 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:42.137124    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-69665bc4b4-dlg7z"
Jul 27 17:33:50.341981 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:50.341912    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-64bb7f674d-lsdcl" podUID="e5e65078-6a76-4778-b0b0-0d8fc032a406" containerName="console" containerID="cri-o://d09ed4a23c5ef6123cabd76ecedfbe1c46b3b69a93f4cc50717dd21ad5f8cd68" gracePeriod=15
Jul 27 17:33:50.623115 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:50.623081    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64bb7f674d-lsdcl"
Jul 27 17:33:50.651710 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:50.651671    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e65078-6a76-4778-b0b0-0d8fc032a406-service-ca\") pod \"e5e65078-6a76-4778-b0b0-0d8fc032a406\" (UID: \"e5e65078-6a76-4778-b0b0-0d8fc032a406\") "
Jul 27 17:33:50.651710 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:50.651711    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e5e65078-6a76-4778-b0b0-0d8fc032a406-console-serving-cert\") pod \"e5e65078-6a76-4778-b0b0-0d8fc032a406\" (UID: \"e5e65078-6a76-4778-b0b0-0d8fc032a406\") "
Jul 27 17:33:50.651972 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:50.651736    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e5e65078-6a76-4778-b0b0-0d8fc032a406-console-config\") pod \"e5e65078-6a76-4778-b0b0-0d8fc032a406\" (UID: \"e5e65078-6a76-4778-b0b0-0d8fc032a406\") "
Jul 27 17:33:50.651972 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:50.651764    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e5e65078-6a76-4778-b0b0-0d8fc032a406-oauth-serving-cert\") pod \"e5e65078-6a76-4778-b0b0-0d8fc032a406\" (UID: \"e5e65078-6a76-4778-b0b0-0d8fc032a406\") "
Jul 27 17:33:50.651972 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:50.651796    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e5e65078-6a76-4778-b0b0-0d8fc032a406-console-oauth-config\") pod \"e5e65078-6a76-4778-b0b0-0d8fc032a406\" (UID: \"e5e65078-6a76-4778-b0b0-0d8fc032a406\") "
Jul 27 17:33:50.651972 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:50.651824    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbmpk\" (UniqueName: \"kubernetes.io/projected/e5e65078-6a76-4778-b0b0-0d8fc032a406-kube-api-access-rbmpk\") pod \"e5e65078-6a76-4778-b0b0-0d8fc032a406\" (UID: \"e5e65078-6a76-4778-b0b0-0d8fc032a406\") "
Jul 27 17:33:50.652492 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:50.652132    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5e65078-6a76-4778-b0b0-0d8fc032a406-console-config" (OuterVolumeSpecName: "console-config") pod "e5e65078-6a76-4778-b0b0-0d8fc032a406" (UID: "e5e65078-6a76-4778-b0b0-0d8fc032a406"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:33:50.652492 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:50.652146    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5e65078-6a76-4778-b0b0-0d8fc032a406-service-ca" (OuterVolumeSpecName: "service-ca") pod "e5e65078-6a76-4778-b0b0-0d8fc032a406" (UID: "e5e65078-6a76-4778-b0b0-0d8fc032a406"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:33:50.652492 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:50.652180    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5e65078-6a76-4778-b0b0-0d8fc032a406-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e5e65078-6a76-4778-b0b0-0d8fc032a406" (UID: "e5e65078-6a76-4778-b0b0-0d8fc032a406"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:33:50.654167 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:50.654133    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e65078-6a76-4778-b0b0-0d8fc032a406-kube-api-access-rbmpk" (OuterVolumeSpecName: "kube-api-access-rbmpk") pod "e5e65078-6a76-4778-b0b0-0d8fc032a406" (UID: "e5e65078-6a76-4778-b0b0-0d8fc032a406"). InnerVolumeSpecName "kube-api-access-rbmpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:33:50.654264 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:50.654211    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5e65078-6a76-4778-b0b0-0d8fc032a406-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e5e65078-6a76-4778-b0b0-0d8fc032a406" (UID: "e5e65078-6a76-4778-b0b0-0d8fc032a406"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:33:50.654410 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:50.654386    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5e65078-6a76-4778-b0b0-0d8fc032a406-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e5e65078-6a76-4778-b0b0-0d8fc032a406" (UID: "e5e65078-6a76-4778-b0b0-0d8fc032a406"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:33:50.753099 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:50.753065    2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e5e65078-6a76-4778-b0b0-0d8fc032a406-service-ca\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:33:50.753099 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:50.753093    2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e5e65078-6a76-4778-b0b0-0d8fc032a406-console-serving-cert\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:33:50.753099 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:50.753103    2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e5e65078-6a76-4778-b0b0-0d8fc032a406-console-config\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:33:50.753318 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:50.753113    2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e5e65078-6a76-4778-b0b0-0d8fc032a406-oauth-serving-cert\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:33:50.753318 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:50.753122    2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e5e65078-6a76-4778-b0b0-0d8fc032a406-console-oauth-config\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:33:50.753318 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:50.753130    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbmpk\" (UniqueName: \"kubernetes.io/projected/e5e65078-6a76-4778-b0b0-0d8fc032a406-kube-api-access-rbmpk\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:33:51.169652 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:51.169618    2557 generic.go:356] "Generic (PLEG): container finished" podID="e5e65078-6a76-4778-b0b0-0d8fc032a406" containerID="d09ed4a23c5ef6123cabd76ecedfbe1c46b3b69a93f4cc50717dd21ad5f8cd68" exitCode=0
Jul 27 17:33:51.169847 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:51.169660    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64bb7f674d-lsdcl"
Jul 27 17:33:51.169847 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:51.169706    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64bb7f674d-lsdcl" event={"ID":"e5e65078-6a76-4778-b0b0-0d8fc032a406","Type":"ContainerDied","Data":"d09ed4a23c5ef6123cabd76ecedfbe1c46b3b69a93f4cc50717dd21ad5f8cd68"}
Jul 27 17:33:51.169847 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:51.169773    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64bb7f674d-lsdcl" event={"ID":"e5e65078-6a76-4778-b0b0-0d8fc032a406","Type":"ContainerDied","Data":"08614bf9aec5f28fb9ad5410e897c65f3998f8a4701b826ad478d4b2ada9e352"}
Jul 27 17:33:51.169847 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:51.169801    2557 scope.go:117] "RemoveContainer" containerID="d09ed4a23c5ef6123cabd76ecedfbe1c46b3b69a93f4cc50717dd21ad5f8cd68"
Jul 27 17:33:51.179146 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:51.179131    2557 scope.go:117] "RemoveContainer" containerID="d09ed4a23c5ef6123cabd76ecedfbe1c46b3b69a93f4cc50717dd21ad5f8cd68"
Jul 27 17:33:51.179431 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:33:51.179411    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d09ed4a23c5ef6123cabd76ecedfbe1c46b3b69a93f4cc50717dd21ad5f8cd68\": container with ID starting with d09ed4a23c5ef6123cabd76ecedfbe1c46b3b69a93f4cc50717dd21ad5f8cd68 not found: ID does not exist" containerID="d09ed4a23c5ef6123cabd76ecedfbe1c46b3b69a93f4cc50717dd21ad5f8cd68"
Jul 27 17:33:51.179507 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:51.179441    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d09ed4a23c5ef6123cabd76ecedfbe1c46b3b69a93f4cc50717dd21ad5f8cd68"} err="failed to get container status \"d09ed4a23c5ef6123cabd76ecedfbe1c46b3b69a93f4cc50717dd21ad5f8cd68\": rpc error: code = NotFound desc = could not find container \"d09ed4a23c5ef6123cabd76ecedfbe1c46b3b69a93f4cc50717dd21ad5f8cd68\": container with ID starting with d09ed4a23c5ef6123cabd76ecedfbe1c46b3b69a93f4cc50717dd21ad5f8cd68 not found: ID does not exist"
Jul 27 17:33:51.190971 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:51.190941    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64bb7f674d-lsdcl"]
Jul 27 17:33:51.192469 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:51.192445    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64bb7f674d-lsdcl"]
Jul 27 17:33:52.117670 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:52.117638    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5e65078-6a76-4778-b0b0-0d8fc032a406" path="/var/lib/kubelet/pods/e5e65078-6a76-4778-b0b0-0d8fc032a406/volumes"
Jul 27 17:33:54.352983 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:54.352939    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p" podUID="86ded04d-8b64-408e-8352-7ef795d848a0" containerName="registry" containerID="cri-o://8302b9599b0b6809178eb547234adaa9ac34326bb97ed963963ebf7aada8a34d" gracePeriod=30
Jul 27 17:33:54.635481 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:54.635438    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:33:54.688404 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:54.688366    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86ded04d-8b64-408e-8352-7ef795d848a0-installation-pull-secrets\") pod \"86ded04d-8b64-408e-8352-7ef795d848a0\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") "
Jul 27 17:33:54.688570 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:54.688426    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86ded04d-8b64-408e-8352-7ef795d848a0-registry-certificates\") pod \"86ded04d-8b64-408e-8352-7ef795d848a0\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") "
Jul 27 17:33:54.688570 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:54.688459    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls\") pod \"86ded04d-8b64-408e-8352-7ef795d848a0\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") "
Jul 27 17:33:54.688570 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:54.688480    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86ded04d-8b64-408e-8352-7ef795d848a0-ca-trust-extracted\") pod \"86ded04d-8b64-408e-8352-7ef795d848a0\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") "
Jul 27 17:33:54.688570 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:54.688531    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86ded04d-8b64-408e-8352-7ef795d848a0-image-registry-private-configuration\") pod \"86ded04d-8b64-408e-8352-7ef795d848a0\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") "
Jul 27 17:33:54.688570 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:54.688565    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xt95m\" (UniqueName: \"kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-kube-api-access-xt95m\") pod \"86ded04d-8b64-408e-8352-7ef795d848a0\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") "
Jul 27 17:33:54.688873 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:54.688589    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86ded04d-8b64-408e-8352-7ef795d848a0-trusted-ca\") pod \"86ded04d-8b64-408e-8352-7ef795d848a0\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") "
Jul 27 17:33:54.688873 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:54.688609    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-bound-sa-token\") pod \"86ded04d-8b64-408e-8352-7ef795d848a0\" (UID: \"86ded04d-8b64-408e-8352-7ef795d848a0\") "
Jul 27 17:33:54.688978 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:54.688895    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86ded04d-8b64-408e-8352-7ef795d848a0-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "86ded04d-8b64-408e-8352-7ef795d848a0" (UID: "86ded04d-8b64-408e-8352-7ef795d848a0"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:33:54.689188 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:54.689159    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86ded04d-8b64-408e-8352-7ef795d848a0-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "86ded04d-8b64-408e-8352-7ef795d848a0" (UID: "86ded04d-8b64-408e-8352-7ef795d848a0"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:33:54.691519 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:54.691492    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86ded04d-8b64-408e-8352-7ef795d848a0-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "86ded04d-8b64-408e-8352-7ef795d848a0" (UID: "86ded04d-8b64-408e-8352-7ef795d848a0"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:33:54.691646 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:54.691607    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86ded04d-8b64-408e-8352-7ef795d848a0-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "86ded04d-8b64-408e-8352-7ef795d848a0" (UID: "86ded04d-8b64-408e-8352-7ef795d848a0"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:33:54.691646 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:54.691639    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "86ded04d-8b64-408e-8352-7ef795d848a0" (UID: "86ded04d-8b64-408e-8352-7ef795d848a0"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:33:54.691788 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:54.691734    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "86ded04d-8b64-408e-8352-7ef795d848a0" (UID: "86ded04d-8b64-408e-8352-7ef795d848a0"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:33:54.691788 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:54.691738    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-kube-api-access-xt95m" (OuterVolumeSpecName: "kube-api-access-xt95m") pod "86ded04d-8b64-408e-8352-7ef795d848a0" (UID: "86ded04d-8b64-408e-8352-7ef795d848a0"). InnerVolumeSpecName "kube-api-access-xt95m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:33:54.697658 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:54.697629    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ded04d-8b64-408e-8352-7ef795d848a0-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "86ded04d-8b64-408e-8352-7ef795d848a0" (UID: "86ded04d-8b64-408e-8352-7ef795d848a0"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:33:54.790074 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:54.790028    2557 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86ded04d-8b64-408e-8352-7ef795d848a0-registry-certificates\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:33:54.790074 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:54.790067    2557 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-registry-tls\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:33:54.790074 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:54.790080    2557 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86ded04d-8b64-408e-8352-7ef795d848a0-ca-trust-extracted\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:33:54.790326 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:54.790093    2557 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86ded04d-8b64-408e-8352-7ef795d848a0-image-registry-private-configuration\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:33:54.790326 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:54.790107    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt95m\" (UniqueName: \"kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-kube-api-access-xt95m\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:33:54.790326 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:54.790117    2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86ded04d-8b64-408e-8352-7ef795d848a0-trusted-ca\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:33:54.790326 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:54.790128    2557 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86ded04d-8b64-408e-8352-7ef795d848a0-bound-sa-token\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:33:54.790326 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:54.790140    2557 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86ded04d-8b64-408e-8352-7ef795d848a0-installation-pull-secrets\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:33:55.186219 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:55.186186    2557 generic.go:356] "Generic (PLEG): container finished" podID="86ded04d-8b64-408e-8352-7ef795d848a0" containerID="8302b9599b0b6809178eb547234adaa9ac34326bb97ed963963ebf7aada8a34d" exitCode=0
Jul 27 17:33:55.186430 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:55.186225    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p"
Jul 27 17:33:55.186430 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:55.186255    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p" event={"ID":"86ded04d-8b64-408e-8352-7ef795d848a0","Type":"ContainerDied","Data":"8302b9599b0b6809178eb547234adaa9ac34326bb97ed963963ebf7aada8a34d"}
Jul 27 17:33:55.186430 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:55.186306    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b499f6dbb-gq96p" event={"ID":"86ded04d-8b64-408e-8352-7ef795d848a0","Type":"ContainerDied","Data":"918687f15de67659601d118746c5c2cd030e6f13a9d91386b29f20a9b85124eb"}
Jul 27 17:33:55.186430 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:55.186332    2557 scope.go:117] "RemoveContainer" containerID="8302b9599b0b6809178eb547234adaa9ac34326bb97ed963963ebf7aada8a34d"
Jul 27 17:33:55.195326 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:55.195308    2557 scope.go:117] "RemoveContainer" containerID="8302b9599b0b6809178eb547234adaa9ac34326bb97ed963963ebf7aada8a34d"
Jul 27 17:33:55.195577 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:33:55.195562    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8302b9599b0b6809178eb547234adaa9ac34326bb97ed963963ebf7aada8a34d\": container with ID starting with 8302b9599b0b6809178eb547234adaa9ac34326bb97ed963963ebf7aada8a34d not found: ID does not exist" containerID="8302b9599b0b6809178eb547234adaa9ac34326bb97ed963963ebf7aada8a34d"
Jul 27 17:33:55.195632 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:55.195585    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8302b9599b0b6809178eb547234adaa9ac34326bb97ed963963ebf7aada8a34d"} err="failed to get container status \"8302b9599b0b6809178eb547234adaa9ac34326bb97ed963963ebf7aada8a34d\": rpc error: code = NotFound desc = could not find container \"8302b9599b0b6809178eb547234adaa9ac34326bb97ed963963ebf7aada8a34d\": container with ID starting with 8302b9599b0b6809178eb547234adaa9ac34326bb97ed963963ebf7aada8a34d not found: ID does not exist"
Jul 27 17:33:55.206436 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:55.206409    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b499f6dbb-gq96p"]
Jul 27 17:33:55.211445 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:55.211420    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6b499f6dbb-gq96p"]
Jul 27 17:33:56.118233 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:33:56.118198    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86ded04d-8b64-408e-8352-7ef795d848a0" path="/var/lib/kubelet/pods/86ded04d-8b64-408e-8352-7ef795d848a0/volumes"
Jul 27 17:34:03.220804 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:34:03.220773    2557 generic.go:356] "Generic (PLEG): container finished" podID="0a3ad2c9-8fab-442a-9ec6-f8eacd33bc22" containerID="a689a000ec8ddd50e1140b8956751687699539fbeea87134cac75b397060447a" exitCode=0
Jul 27 17:34:03.220804 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:34:03.220792    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t" event={"ID":"0a3ad2c9-8fab-442a-9ec6-f8eacd33bc22","Type":"ContainerDied","Data":"a689a000ec8ddd50e1140b8956751687699539fbeea87134cac75b397060447a"}
Jul 27 17:34:03.221233 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:34:03.221124    2557 scope.go:117] "RemoveContainer" containerID="a689a000ec8ddd50e1140b8956751687699539fbeea87134cac75b397060447a"
Jul 27 17:34:04.226535 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:34:04.226498    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t" event={"ID":"0a3ad2c9-8fab-442a-9ec6-f8eacd33bc22","Type":"ContainerStarted","Data":"fb2d6725b386e79b49cc327ed389e351e16b2e45a517eb7e4a671c032cad65b4"}
Jul 27 17:34:26.289769 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:34:26.289679    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:34:26.308072 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:34:26.308049    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:34:26.322788 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:34:26.322769    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 17:34:41.981296 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:34:41.981251    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs\") pod \"network-metrics-daemon-mzhnc\" (UID: \"57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec\") " pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:34:41.983630 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:34:41.983608    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec-metrics-certs\") pod \"network-metrics-daemon-mzhnc\" (UID: \"57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec\") " pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:34:42.116151 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:34:42.116115    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qwvcb\""
Jul 27 17:34:42.124741 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:34:42.124723    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzhnc"
Jul 27 17:34:42.253211 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:34:42.253114    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mzhnc"]
Jul 27 17:34:42.259015 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:34:42.258984    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ad8e2d_11d3_4f9d_bc88_78eec6fc0eec.slice/crio-50dc004d6761b6232b686e9561a7638280408d8cff54c006b20bd6936e947c30 WatchSource:0}: Error finding container 50dc004d6761b6232b686e9561a7638280408d8cff54c006b20bd6936e947c30: Status 404 returned error can't find the container with id 50dc004d6761b6232b686e9561a7638280408d8cff54c006b20bd6936e947c30
Jul 27 17:34:42.369981 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:34:42.369948    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mzhnc" event={"ID":"57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec","Type":"ContainerStarted","Data":"50dc004d6761b6232b686e9561a7638280408d8cff54c006b20bd6936e947c30"}
Jul 27 17:34:44.379219 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:34:44.379189    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mzhnc" event={"ID":"57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec","Type":"ContainerStarted","Data":"21a1a53d76d9759b3055c1c3a4bc15094a1924256bebfca48670ece42f144a47"}
Jul 27 17:34:45.384407 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:34:45.384372    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mzhnc" event={"ID":"57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec","Type":"ContainerStarted","Data":"d37fb8307291c2015e28fd9302cc35cb6b0db9faf0fbbd0e09494a304abacd4a"}
Jul 27 17:34:45.398563 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:34:45.398519    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mzhnc" podStartSLOduration=375.39913794 podStartE2EDuration="6m17.398499276s" totalImagesPullingTime="1.999361336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:34:45.397905374 +0000 UTC m=+377.884716705" watchObservedRunningTime="2026-07-27 17:34:45.398499276 +0000 UTC m=+377.885310562"
Jul 27 17:39:25.849089 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:25.849051    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-h64fd"]
Jul 27 17:39:25.849597 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:25.849353    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86ded04d-8b64-408e-8352-7ef795d848a0" containerName="registry"
Jul 27 17:39:25.849597 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:25.849363    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="86ded04d-8b64-408e-8352-7ef795d848a0" containerName="registry"
Jul 27 17:39:25.849597 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:25.849385    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5e65078-6a76-4778-b0b0-0d8fc032a406" containerName="console"
Jul 27 17:39:25.849597 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:25.849390    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5e65078-6a76-4778-b0b0-0d8fc032a406" containerName="console"
Jul 27 17:39:25.849597 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:25.849450    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5e65078-6a76-4778-b0b0-0d8fc032a406" containerName="console"
Jul 27 17:39:25.849597 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:25.849458    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="86ded04d-8b64-408e-8352-7ef795d848a0" containerName="registry"
Jul 27 17:39:25.852357 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:25.852338    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-h64fd"
Jul 27 17:39:25.854469 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:25.854449    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 27 17:39:25.854946 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:25.854925    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 27 17:39:25.855041 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:25.854930    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-bz584\""
Jul 27 17:39:25.859999 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:25.859978    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-h64fd"]
Jul 27 17:39:25.952445 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:25.952407    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59969\" (UniqueName: \"kubernetes.io/projected/b3ed0734-0f0b-4669-b06a-51110deda353-kube-api-access-59969\") pod \"cert-manager-webhook-6486ffd997-h64fd\" (UID: \"b3ed0734-0f0b-4669-b06a-51110deda353\") " pod="cert-manager/cert-manager-webhook-6486ffd997-h64fd"
Jul 27 17:39:25.952609 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:25.952455    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3ed0734-0f0b-4669-b06a-51110deda353-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-h64fd\" (UID: \"b3ed0734-0f0b-4669-b06a-51110deda353\") " pod="cert-manager/cert-manager-webhook-6486ffd997-h64fd"
Jul 27 17:39:26.053508 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:26.053480    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3ed0734-0f0b-4669-b06a-51110deda353-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-h64fd\" (UID: \"b3ed0734-0f0b-4669-b06a-51110deda353\") " pod="cert-manager/cert-manager-webhook-6486ffd997-h64fd"
Jul 27 17:39:26.053651 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:26.053545    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59969\" (UniqueName: \"kubernetes.io/projected/b3ed0734-0f0b-4669-b06a-51110deda353-kube-api-access-59969\") pod \"cert-manager-webhook-6486ffd997-h64fd\" (UID: \"b3ed0734-0f0b-4669-b06a-51110deda353\") " pod="cert-manager/cert-manager-webhook-6486ffd997-h64fd"
Jul 27 17:39:26.067940 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:26.067914    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3ed0734-0f0b-4669-b06a-51110deda353-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-h64fd\" (UID: \"b3ed0734-0f0b-4669-b06a-51110deda353\") " pod="cert-manager/cert-manager-webhook-6486ffd997-h64fd"
Jul 27 17:39:26.068055 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:26.067915    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59969\" (UniqueName: \"kubernetes.io/projected/b3ed0734-0f0b-4669-b06a-51110deda353-kube-api-access-59969\") pod \"cert-manager-webhook-6486ffd997-h64fd\" (UID: \"b3ed0734-0f0b-4669-b06a-51110deda353\") " pod="cert-manager/cert-manager-webhook-6486ffd997-h64fd"
Jul 27 17:39:26.169048 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:26.169010    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-h64fd"
Jul 27 17:39:26.308724 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:26.308581    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-h64fd"]
Jul 27 17:39:26.311297 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:39:26.311263    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3ed0734_0f0b_4669_b06a_51110deda353.slice/crio-c973aa45ea41e9b6698e0331bcf9d9e283802f56c9023e95b9a93dfb6a575e77 WatchSource:0}: Error finding container c973aa45ea41e9b6698e0331bcf9d9e283802f56c9023e95b9a93dfb6a575e77: Status 404 returned error can't find the container with id c973aa45ea41e9b6698e0331bcf9d9e283802f56c9023e95b9a93dfb6a575e77
Jul 27 17:39:26.313407 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:26.313388    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:39:26.449845 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:26.449771    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-h64fd" event={"ID":"b3ed0734-0f0b-4669-b06a-51110deda353","Type":"ContainerStarted","Data":"c973aa45ea41e9b6698e0331bcf9d9e283802f56c9023e95b9a93dfb6a575e77"}
Jul 27 17:39:30.471283 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:30.471243    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-h64fd" event={"ID":"b3ed0734-0f0b-4669-b06a-51110deda353","Type":"ContainerStarted","Data":"2e004d157cf30751165d8b6898b72cff3fc7ac08c1ebc9eae02915efa34c89d5"}
Jul 27 17:39:30.471675 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:30.471431    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-h64fd"
Jul 27 17:39:30.487394 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:30.487348    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-h64fd" podStartSLOduration=2.201506626 podStartE2EDuration="5.487334721s" totalImagesPullingTime="3.285828095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:39:30.486565397 +0000 UTC m=+662.973376685" watchObservedRunningTime="2026-07-27 17:39:30.487334721 +0000 UTC m=+662.974146008"
Jul 27 17:39:36.478403 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:36.478372    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-h64fd"
Jul 27 17:39:40.138823 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:40.138790    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-hf2pd"]
Jul 27 17:39:40.142105 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:40.142087    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-hf2pd"
Jul 27 17:39:40.144371 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:40.144346    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-5b27q\""
Jul 27 17:39:40.150013 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:40.149987    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-hf2pd"]
Jul 27 17:39:40.274449 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:40.274415    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9rbq\" (UniqueName: \"kubernetes.io/projected/1c971a8d-d9e2-4816-8f09-3e4ee6233026-kube-api-access-p9rbq\") pod \"cert-manager-55b4d954d5-hf2pd\" (UID: \"1c971a8d-d9e2-4816-8f09-3e4ee6233026\") " pod="cert-manager/cert-manager-55b4d954d5-hf2pd"
Jul 27 17:39:40.274614 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:40.274479    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1c971a8d-d9e2-4816-8f09-3e4ee6233026-bound-sa-token\") pod \"cert-manager-55b4d954d5-hf2pd\" (UID: \"1c971a8d-d9e2-4816-8f09-3e4ee6233026\") " pod="cert-manager/cert-manager-55b4d954d5-hf2pd"
Jul 27 17:39:40.375879 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:40.375849    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1c971a8d-d9e2-4816-8f09-3e4ee6233026-bound-sa-token\") pod \"cert-manager-55b4d954d5-hf2pd\" (UID: \"1c971a8d-d9e2-4816-8f09-3e4ee6233026\") " pod="cert-manager/cert-manager-55b4d954d5-hf2pd"
Jul 27 17:39:40.376018 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:40.375925    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9rbq\" (UniqueName: \"kubernetes.io/projected/1c971a8d-d9e2-4816-8f09-3e4ee6233026-kube-api-access-p9rbq\") pod \"cert-manager-55b4d954d5-hf2pd\" (UID: \"1c971a8d-d9e2-4816-8f09-3e4ee6233026\") " pod="cert-manager/cert-manager-55b4d954d5-hf2pd"
Jul 27 17:39:40.385223 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:40.385192    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1c971a8d-d9e2-4816-8f09-3e4ee6233026-bound-sa-token\") pod \"cert-manager-55b4d954d5-hf2pd\" (UID: \"1c971a8d-d9e2-4816-8f09-3e4ee6233026\") " pod="cert-manager/cert-manager-55b4d954d5-hf2pd"
Jul 27 17:39:40.385223 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:40.385210    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9rbq\" (UniqueName: \"kubernetes.io/projected/1c971a8d-d9e2-4816-8f09-3e4ee6233026-kube-api-access-p9rbq\") pod \"cert-manager-55b4d954d5-hf2pd\" (UID: \"1c971a8d-d9e2-4816-8f09-3e4ee6233026\") " pod="cert-manager/cert-manager-55b4d954d5-hf2pd"
Jul 27 17:39:40.451397 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:40.451376    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-hf2pd"
Jul 27 17:39:40.574442 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:40.574409    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-hf2pd"]
Jul 27 17:39:40.578081 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:39:40.578051    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c971a8d_d9e2_4816_8f09_3e4ee6233026.slice/crio-cfdcd6bdcff338ff72ced325c4623e191f29e5c5765da15dddf72e388f509229 WatchSource:0}: Error finding container cfdcd6bdcff338ff72ced325c4623e191f29e5c5765da15dddf72e388f509229: Status 404 returned error can't find the container with id cfdcd6bdcff338ff72ced325c4623e191f29e5c5765da15dddf72e388f509229
Jul 27 17:39:41.520689 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:41.520652    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-hf2pd" event={"ID":"1c971a8d-d9e2-4816-8f09-3e4ee6233026","Type":"ContainerStarted","Data":"9109d590caa8c1adc102b6ef23c276793bf5e17ac360da288fc4163c84e86d8e"}
Jul 27 17:39:41.521097 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:41.520694    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-hf2pd" event={"ID":"1c971a8d-d9e2-4816-8f09-3e4ee6233026","Type":"ContainerStarted","Data":"cfdcd6bdcff338ff72ced325c4623e191f29e5c5765da15dddf72e388f509229"}
Jul 27 17:39:41.542830 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:39:41.542787    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-hf2pd" podStartSLOduration=1.5427702970000001 podStartE2EDuration="1.542770297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:39:41.541150526 +0000 UTC m=+674.027961814" watchObservedRunningTime="2026-07-27 17:39:41.542770297 +0000 UTC m=+674.029581584"
Jul 27 17:40:17.081005 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:17.080966    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-67d9b7fdbd-h8v7t"]
Jul 27 17:40:17.087332 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:17.087316    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-h8v7t"
Jul 27 17:40:17.088515 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:17.088490    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlsdr\" (UniqueName: \"kubernetes.io/projected/564cbc55-94c2-475f-9577-6006b84d1177-kube-api-access-dlsdr\") pod \"lws-controller-manager-67d9b7fdbd-h8v7t\" (UID: \"564cbc55-94c2-475f-9577-6006b84d1177\") " pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-h8v7t"
Jul 27 17:40:17.088611 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:17.088526    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/564cbc55-94c2-475f-9577-6006b84d1177-manager-config\") pod \"lws-controller-manager-67d9b7fdbd-h8v7t\" (UID: \"564cbc55-94c2-475f-9577-6006b84d1177\") " pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-h8v7t"
Jul 27 17:40:17.088611 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:17.088543    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/564cbc55-94c2-475f-9577-6006b84d1177-cert\") pod \"lws-controller-manager-67d9b7fdbd-h8v7t\" (UID: \"564cbc55-94c2-475f-9577-6006b84d1177\") " pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-h8v7t"
Jul 27 17:40:17.088611 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:17.088601    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/564cbc55-94c2-475f-9577-6006b84d1177-metrics-cert\") pod \"lws-controller-manager-67d9b7fdbd-h8v7t\" (UID: \"564cbc55-94c2-475f-9577-6006b84d1177\") " pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-h8v7t"
Jul 27 17:40:17.092396 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:17.092379    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 27 17:40:17.092472 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:17.092447    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 27 17:40:17.092529 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:17.092512    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 27 17:40:17.092574 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:17.092452    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 27 17:40:17.092620 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:17.092447    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 27 17:40:17.092661 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:17.092524    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-hvdmn\""
Jul 27 17:40:17.110454 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:17.110428    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-67d9b7fdbd-h8v7t"]
Jul 27 17:40:17.189972 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:17.189942    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlsdr\" (UniqueName: \"kubernetes.io/projected/564cbc55-94c2-475f-9577-6006b84d1177-kube-api-access-dlsdr\") pod \"lws-controller-manager-67d9b7fdbd-h8v7t\" (UID: \"564cbc55-94c2-475f-9577-6006b84d1177\") " pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-h8v7t"
Jul 27 17:40:17.190136 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:17.189983    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/564cbc55-94c2-475f-9577-6006b84d1177-manager-config\") pod \"lws-controller-manager-67d9b7fdbd-h8v7t\" (UID: \"564cbc55-94c2-475f-9577-6006b84d1177\") " pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-h8v7t"
Jul 27 17:40:17.190136 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:17.190010    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/564cbc55-94c2-475f-9577-6006b84d1177-cert\") pod \"lws-controller-manager-67d9b7fdbd-h8v7t\" (UID: \"564cbc55-94c2-475f-9577-6006b84d1177\") " pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-h8v7t"
Jul 27 17:40:17.190136 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:17.190075    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/564cbc55-94c2-475f-9577-6006b84d1177-metrics-cert\") pod \"lws-controller-manager-67d9b7fdbd-h8v7t\" (UID: \"564cbc55-94c2-475f-9577-6006b84d1177\") " pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-h8v7t"
Jul 27 17:40:17.190863 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:17.190835    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/564cbc55-94c2-475f-9577-6006b84d1177-manager-config\") pod \"lws-controller-manager-67d9b7fdbd-h8v7t\" (UID: \"564cbc55-94c2-475f-9577-6006b84d1177\") " pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-h8v7t"
Jul 27 17:40:17.192549 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:17.192526    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/564cbc55-94c2-475f-9577-6006b84d1177-metrics-cert\") pod \"lws-controller-manager-67d9b7fdbd-h8v7t\" (UID: \"564cbc55-94c2-475f-9577-6006b84d1177\") " pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-h8v7t"
Jul 27 17:40:17.192656 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:17.192594    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/564cbc55-94c2-475f-9577-6006b84d1177-cert\") pod \"lws-controller-manager-67d9b7fdbd-h8v7t\" (UID: \"564cbc55-94c2-475f-9577-6006b84d1177\") " pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-h8v7t"
Jul 27 17:40:17.203524 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:17.203501    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlsdr\" (UniqueName: \"kubernetes.io/projected/564cbc55-94c2-475f-9577-6006b84d1177-kube-api-access-dlsdr\") pod \"lws-controller-manager-67d9b7fdbd-h8v7t\" (UID: \"564cbc55-94c2-475f-9577-6006b84d1177\") " pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-h8v7t"
Jul 27 17:40:17.397312 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:17.397232    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-h8v7t"
Jul 27 17:40:17.531809 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:17.531782    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-67d9b7fdbd-h8v7t"]
Jul 27 17:40:17.533922 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:40:17.533892    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod564cbc55_94c2_475f_9577_6006b84d1177.slice/crio-2154ef22694cabe80f9be523c2fc4c1dbdba3508c520ca8c0d0ea5afb3f4a059 WatchSource:0}: Error finding container 2154ef22694cabe80f9be523c2fc4c1dbdba3508c520ca8c0d0ea5afb3f4a059: Status 404 returned error can't find the container with id 2154ef22694cabe80f9be523c2fc4c1dbdba3508c520ca8c0d0ea5afb3f4a059
Jul 27 17:40:17.657425 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:17.657402    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-h8v7t" event={"ID":"564cbc55-94c2-475f-9577-6006b84d1177","Type":"ContainerStarted","Data":"2154ef22694cabe80f9be523c2fc4c1dbdba3508c520ca8c0d0ea5afb3f4a059"}
Jul 27 17:40:20.672255 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:20.672219    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-h8v7t" event={"ID":"564cbc55-94c2-475f-9577-6006b84d1177","Type":"ContainerStarted","Data":"167551e2fe1ef51e07266a5d5396850850e8b6b3aeacfb3978a716dff7cf4d80"}
Jul 27 17:40:20.672630 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:20.672340    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-h8v7t"
Jul 27 17:40:20.690330 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:20.690226    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-h8v7t" podStartSLOduration=0.787643978 podStartE2EDuration="3.690209831s" totalImagesPullingTime="2.902565853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:40:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:40:20.689378782 +0000 UTC m=+713.176190099" watchObservedRunningTime="2026-07-27 17:40:20.690209831 +0000 UTC m=+713.177021109"
Jul 27 17:40:29.289114 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:29.289083    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-5z76z"]
Jul 27 17:40:29.296526 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:29.296498    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-5z76z"
Jul 27 17:40:29.298826 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:29.298804    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 27 17:40:29.299044 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:29.299025    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jdd5j\""
Jul 27 17:40:29.299125 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:29.299080    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 27 17:40:29.307397 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:29.307375    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-5z76z"]
Jul 27 17:40:29.391936 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:29.391901    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7hr6\" (UniqueName: \"kubernetes.io/projected/3882525b-8726-46b2-b717-9736e21fd723-kube-api-access-q7hr6\") pod \"kuadrant-operator-controller-manager-66866f5fcc-5z76z\" (UID: \"3882525b-8726-46b2-b717-9736e21fd723\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-5z76z"
Jul 27 17:40:29.392107 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:29.391971    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3882525b-8726-46b2-b717-9736e21fd723-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-5z76z\" (UID: \"3882525b-8726-46b2-b717-9736e21fd723\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-5z76z"
Jul 27 17:40:29.492605 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:29.492562    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3882525b-8726-46b2-b717-9736e21fd723-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-5z76z\" (UID: \"3882525b-8726-46b2-b717-9736e21fd723\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-5z76z"
Jul 27 17:40:29.492797 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:29.492641    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7hr6\" (UniqueName: \"kubernetes.io/projected/3882525b-8726-46b2-b717-9736e21fd723-kube-api-access-q7hr6\") pod \"kuadrant-operator-controller-manager-66866f5fcc-5z76z\" (UID: \"3882525b-8726-46b2-b717-9736e21fd723\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-5z76z"
Jul 27 17:40:29.492965 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:29.492943    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3882525b-8726-46b2-b717-9736e21fd723-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-5z76z\" (UID: \"3882525b-8726-46b2-b717-9736e21fd723\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-5z76z"
Jul 27 17:40:29.503449 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:29.503417    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7hr6\" (UniqueName: \"kubernetes.io/projected/3882525b-8726-46b2-b717-9736e21fd723-kube-api-access-q7hr6\") pod \"kuadrant-operator-controller-manager-66866f5fcc-5z76z\" (UID: \"3882525b-8726-46b2-b717-9736e21fd723\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-5z76z"
Jul 27 17:40:29.607016 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:29.606928    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-5z76z"
Jul 27 17:40:29.754921 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:29.754889    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-5z76z"]
Jul 27 17:40:29.756824 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:40:29.756786    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3882525b_8726_46b2_b717_9736e21fd723.slice/crio-c8f76afc602576e7b81c0a1459490560bfe6c46a78565049dabf9db0a53ee314 WatchSource:0}: Error finding container c8f76afc602576e7b81c0a1459490560bfe6c46a78565049dabf9db0a53ee314: Status 404 returned error can't find the container with id c8f76afc602576e7b81c0a1459490560bfe6c46a78565049dabf9db0a53ee314
Jul 27 17:40:30.713868 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:30.713834    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-5z76z" event={"ID":"3882525b-8726-46b2-b717-9736e21fd723","Type":"ContainerStarted","Data":"c8f76afc602576e7b81c0a1459490560bfe6c46a78565049dabf9db0a53ee314"}
Jul 27 17:40:31.679203 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:31.679171    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-67d9b7fdbd-h8v7t"
Jul 27 17:40:34.736983 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:34.736942    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-5z76z" event={"ID":"3882525b-8726-46b2-b717-9736e21fd723","Type":"ContainerStarted","Data":"2cf04cbff5ae8f223137509bee405ef7baf8777ac33707517b7121761fa9fe3d"}
Jul 27 17:40:34.737440 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:34.737068    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-5z76z"
Jul 27 17:40:34.768657 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:34.768601    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-5z76z" podStartSLOduration=1.389371418 podStartE2EDuration="5.768583375s" totalImagesPullingTime="4.379211957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:40:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:40:34.766639929 +0000 UTC m=+727.253451252" watchObservedRunningTime="2026-07-27 17:40:34.768583375 +0000 UTC m=+727.255394661"
Jul 27 17:40:36.469525 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:36.469493    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-447cz"]
Jul 27 17:40:36.472871 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:36.472854    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-447cz"
Jul 27 17:40:36.475246 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:36.475226    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-xxfhq\""
Jul 27 17:40:36.475459 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:36.475445    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 27 17:40:36.483688 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:36.483666    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-447cz"]
Jul 27 17:40:36.552224 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:36.552194    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f497\" (UniqueName: \"kubernetes.io/projected/a1235f6f-c04c-46d7-aa23-0c2c78f13c21-kube-api-access-2f497\") pod \"dns-operator-controller-manager-ccbdf686-447cz\" (UID: \"a1235f6f-c04c-46d7-aa23-0c2c78f13c21\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-447cz"
Jul 27 17:40:36.576421 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:36.576392    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-ncggk"]
Jul 27 17:40:36.579730 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:36.579711    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ncggk"
Jul 27 17:40:36.581832 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:36.581811    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 27 17:40:36.582111 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:36.582098    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6qg96\""
Jul 27 17:40:36.582174 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:36.582119    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 27 17:40:36.589110 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:36.589089    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-ncggk"]
Jul 27 17:40:36.653080 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:36.653050    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1244ce20-3490-453e-8bf3-716230099686-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-ncggk\" (UID: \"1244ce20-3490-453e-8bf3-716230099686\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ncggk"
Jul 27 17:40:36.653080 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:36.653076    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q4wx\" (UniqueName: \"kubernetes.io/projected/1244ce20-3490-453e-8bf3-716230099686-kube-api-access-9q4wx\") pod \"kuadrant-console-plugin-598f86468b-ncggk\" (UID: \"1244ce20-3490-453e-8bf3-716230099686\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ncggk"
Jul 27 17:40:36.653275 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:36.653100    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1244ce20-3490-453e-8bf3-716230099686-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-ncggk\" (UID: \"1244ce20-3490-453e-8bf3-716230099686\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ncggk"
Jul 27 17:40:36.653275 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:36.653127    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f497\" (UniqueName: \"kubernetes.io/projected/a1235f6f-c04c-46d7-aa23-0c2c78f13c21-kube-api-access-2f497\") pod \"dns-operator-controller-manager-ccbdf686-447cz\" (UID: \"a1235f6f-c04c-46d7-aa23-0c2c78f13c21\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-447cz"
Jul 27 17:40:36.666155 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:36.666133    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f497\" (UniqueName: \"kubernetes.io/projected/a1235f6f-c04c-46d7-aa23-0c2c78f13c21-kube-api-access-2f497\") pod \"dns-operator-controller-manager-ccbdf686-447cz\" (UID: \"a1235f6f-c04c-46d7-aa23-0c2c78f13c21\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-447cz"
Jul 27 17:40:36.754612 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:36.754541    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1244ce20-3490-453e-8bf3-716230099686-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-ncggk\" (UID: \"1244ce20-3490-453e-8bf3-716230099686\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ncggk"
Jul 27 17:40:36.754612 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:36.754578    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q4wx\" (UniqueName: \"kubernetes.io/projected/1244ce20-3490-453e-8bf3-716230099686-kube-api-access-9q4wx\") pod \"kuadrant-console-plugin-598f86468b-ncggk\" (UID: \"1244ce20-3490-453e-8bf3-716230099686\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ncggk"
Jul 27 17:40:36.754612 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:36.754610    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1244ce20-3490-453e-8bf3-716230099686-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-ncggk\" (UID: \"1244ce20-3490-453e-8bf3-716230099686\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ncggk"
Jul 27 17:40:36.755245 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:36.755228    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1244ce20-3490-453e-8bf3-716230099686-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-ncggk\" (UID: \"1244ce20-3490-453e-8bf3-716230099686\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ncggk"
Jul 27 17:40:36.756825 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:36.756811    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1244ce20-3490-453e-8bf3-716230099686-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-ncggk\" (UID: \"1244ce20-3490-453e-8bf3-716230099686\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ncggk"
Jul 27 17:40:36.762531 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:36.762500    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q4wx\" (UniqueName: \"kubernetes.io/projected/1244ce20-3490-453e-8bf3-716230099686-kube-api-access-9q4wx\") pod \"kuadrant-console-plugin-598f86468b-ncggk\" (UID: \"1244ce20-3490-453e-8bf3-716230099686\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ncggk"
Jul 27 17:40:36.784517 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:36.784493    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-447cz"
Jul 27 17:40:36.889227 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:36.889199    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ncggk"
Jul 27 17:40:36.922590 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:36.922558    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-447cz"]
Jul 27 17:40:36.924272 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:40:36.924242    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1235f6f_c04c_46d7_aa23_0c2c78f13c21.slice/crio-39b68902f536af16c485a77c46105a26fa39588e0c97eec27a99da675c150a69 WatchSource:0}: Error finding container 39b68902f536af16c485a77c46105a26fa39588e0c97eec27a99da675c150a69: Status 404 returned error can't find the container with id 39b68902f536af16c485a77c46105a26fa39588e0c97eec27a99da675c150a69
Jul 27 17:40:37.017630 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:37.017581    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-ncggk"]
Jul 27 17:40:37.020145 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:40:37.020113    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1244ce20_3490_453e_8bf3_716230099686.slice/crio-e7bff00869fdeb6666532fd1b8c0feaec7f04a3ca989c9ae35c9699fc6a9ea5b WatchSource:0}: Error finding container e7bff00869fdeb6666532fd1b8c0feaec7f04a3ca989c9ae35c9699fc6a9ea5b: Status 404 returned error can't find the container with id e7bff00869fdeb6666532fd1b8c0feaec7f04a3ca989c9ae35c9699fc6a9ea5b
Jul 27 17:40:37.751145 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:37.751110    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-447cz" event={"ID":"a1235f6f-c04c-46d7-aa23-0c2c78f13c21","Type":"ContainerStarted","Data":"39b68902f536af16c485a77c46105a26fa39588e0c97eec27a99da675c150a69"}
Jul 27 17:40:37.752310 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:37.752288    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ncggk" event={"ID":"1244ce20-3490-453e-8bf3-716230099686","Type":"ContainerStarted","Data":"e7bff00869fdeb6666532fd1b8c0feaec7f04a3ca989c9ae35c9699fc6a9ea5b"}
Jul 27 17:40:39.765326 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:39.765285    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-447cz" event={"ID":"a1235f6f-c04c-46d7-aa23-0c2c78f13c21","Type":"ContainerStarted","Data":"b377cfa923c84e4ba1f617b38d233dc8b76fa4a59c3939edf0b4f0d6308c6cdd"}
Jul 27 17:40:39.765920 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:39.765439    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-447cz"
Jul 27 17:40:39.793956 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:39.793903    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-447cz" podStartSLOduration=1.652779231 podStartE2EDuration="3.793886544s" totalImagesPullingTime="2.141107313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:40:39.791328365 +0000 UTC m=+732.278139678" watchObservedRunningTime="2026-07-27 17:40:39.793886544 +0000 UTC m=+732.280697830"
Jul 27 17:40:42.782498 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:42.782464    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ncggk" event={"ID":"1244ce20-3490-453e-8bf3-716230099686","Type":"ContainerStarted","Data":"dd091f2e09ef442ab0e66c2116ca912f498c9e0adc85d667a75a135d4827bc03"}
Jul 27 17:40:42.804577 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:42.804524    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-ncggk" podStartSLOduration=1.879022639 podStartE2EDuration="6.8045068s" totalImagesPullingTime="4.925484161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:40:42.802849158 +0000 UTC m=+735.289660444" watchObservedRunningTime="2026-07-27 17:40:42.8045068 +0000 UTC m=+735.291318086"
Jul 27 17:40:45.743905 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:45.743874    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-5z76z"
Jul 27 17:40:50.773221 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:40:50.773190    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-447cz"
Jul 27 17:41:18.927816 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:18.927777    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-q5kfx"]
Jul 27 17:41:18.953994 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:18.953963    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-q5kfx"]
Jul 27 17:41:18.954182 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:18.954053    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-q5kfx"
Jul 27 17:41:18.956196 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:18.956175    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 27 17:41:19.008659 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:19.008627    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx4lf\" (UniqueName: \"kubernetes.io/projected/53a4dc14-0af7-47d4-b5d0-4a66bb723611-kube-api-access-mx4lf\") pod \"limitador-limitador-859c4d96d6-q5kfx\" (UID: \"53a4dc14-0af7-47d4-b5d0-4a66bb723611\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-q5kfx"
Jul 27 17:41:19.008838 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:19.008684    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/53a4dc14-0af7-47d4-b5d0-4a66bb723611-config-file\") pod \"limitador-limitador-859c4d96d6-q5kfx\" (UID: \"53a4dc14-0af7-47d4-b5d0-4a66bb723611\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-q5kfx"
Jul 27 17:41:19.027342 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:19.027312    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-q5kfx"]
Jul 27 17:41:19.109142 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:19.109110    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx4lf\" (UniqueName: \"kubernetes.io/projected/53a4dc14-0af7-47d4-b5d0-4a66bb723611-kube-api-access-mx4lf\") pod \"limitador-limitador-859c4d96d6-q5kfx\" (UID: \"53a4dc14-0af7-47d4-b5d0-4a66bb723611\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-q5kfx"
Jul 27 17:41:19.109290 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:19.109165    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/53a4dc14-0af7-47d4-b5d0-4a66bb723611-config-file\") pod \"limitador-limitador-859c4d96d6-q5kfx\" (UID: \"53a4dc14-0af7-47d4-b5d0-4a66bb723611\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-q5kfx"
Jul 27 17:41:19.109841 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:19.109823    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/53a4dc14-0af7-47d4-b5d0-4a66bb723611-config-file\") pod \"limitador-limitador-859c4d96d6-q5kfx\" (UID: \"53a4dc14-0af7-47d4-b5d0-4a66bb723611\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-q5kfx"
Jul 27 17:41:19.121020 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:19.120994    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx4lf\" (UniqueName: \"kubernetes.io/projected/53a4dc14-0af7-47d4-b5d0-4a66bb723611-kube-api-access-mx4lf\") pod \"limitador-limitador-859c4d96d6-q5kfx\" (UID: \"53a4dc14-0af7-47d4-b5d0-4a66bb723611\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-q5kfx"
Jul 27 17:41:19.264576 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:19.264498    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-q5kfx"
Jul 27 17:41:19.398394 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:19.398369    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-q5kfx"]
Jul 27 17:41:19.400222 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:41:19.400195    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a4dc14_0af7_47d4_b5d0_4a66bb723611.slice/crio-d366dc111952ea0527922ab8eefe1f2864357f8cb50f23cdd65ec1e0a718ee47 WatchSource:0}: Error finding container d366dc111952ea0527922ab8eefe1f2864357f8cb50f23cdd65ec1e0a718ee47: Status 404 returned error can't find the container with id d366dc111952ea0527922ab8eefe1f2864357f8cb50f23cdd65ec1e0a718ee47
Jul 27 17:41:19.953827 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:19.953788    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-q5kfx" event={"ID":"53a4dc14-0af7-47d4-b5d0-4a66bb723611","Type":"ContainerStarted","Data":"d366dc111952ea0527922ab8eefe1f2864357f8cb50f23cdd65ec1e0a718ee47"}
Jul 27 17:41:21.966537 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:21.966498    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-q5kfx" event={"ID":"53a4dc14-0af7-47d4-b5d0-4a66bb723611","Type":"ContainerStarted","Data":"f9d059cb06c55047ce8985d3061a7a891c071b122ef911b88c91e08cf68a2d64"}
Jul 27 17:41:21.966951 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:21.966578    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-q5kfx"
Jul 27 17:41:21.985722 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:21.985673    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-q5kfx" podStartSLOduration=2.319424321 podStartE2EDuration="3.985656168s" totalImagesPullingTime="1.666231847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:41:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:41:21.983338714 +0000 UTC m=+774.470150013" watchObservedRunningTime="2026-07-27 17:41:21.985656168 +0000 UTC m=+774.472467454"
Jul 27 17:41:32.458822 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:32.458170    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-q5kfx"]
Jul 27 17:41:32.458822 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:32.458471    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-q5kfx" podUID="53a4dc14-0af7-47d4-b5d0-4a66bb723611" containerName="limitador" containerID="cri-o://f9d059cb06c55047ce8985d3061a7a891c071b122ef911b88c91e08cf68a2d64" gracePeriod=30
Jul 27 17:41:32.459629 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:32.459480    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-q5kfx"
Jul 27 17:41:33.011382 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:33.011359    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-q5kfx"
Jul 27 17:41:33.016831 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:33.016802    2557 generic.go:356] "Generic (PLEG): container finished" podID="53a4dc14-0af7-47d4-b5d0-4a66bb723611" containerID="f9d059cb06c55047ce8985d3061a7a891c071b122ef911b88c91e08cf68a2d64" exitCode=0
Jul 27 17:41:33.016972 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:33.016840    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-q5kfx"
Jul 27 17:41:33.016972 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:33.016864    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-q5kfx" event={"ID":"53a4dc14-0af7-47d4-b5d0-4a66bb723611","Type":"ContainerDied","Data":"f9d059cb06c55047ce8985d3061a7a891c071b122ef911b88c91e08cf68a2d64"}
Jul 27 17:41:33.016972 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:33.016909    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-q5kfx" event={"ID":"53a4dc14-0af7-47d4-b5d0-4a66bb723611","Type":"ContainerDied","Data":"d366dc111952ea0527922ab8eefe1f2864357f8cb50f23cdd65ec1e0a718ee47"}
Jul 27 17:41:33.016972 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:33.016934    2557 scope.go:117] "RemoveContainer" containerID="f9d059cb06c55047ce8985d3061a7a891c071b122ef911b88c91e08cf68a2d64"
Jul 27 17:41:33.024817 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:33.024713    2557 scope.go:117] "RemoveContainer" containerID="f9d059cb06c55047ce8985d3061a7a891c071b122ef911b88c91e08cf68a2d64"
Jul 27 17:41:33.025006 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:41:33.024986    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9d059cb06c55047ce8985d3061a7a891c071b122ef911b88c91e08cf68a2d64\": container with ID starting with f9d059cb06c55047ce8985d3061a7a891c071b122ef911b88c91e08cf68a2d64 not found: ID does not exist" containerID="f9d059cb06c55047ce8985d3061a7a891c071b122ef911b88c91e08cf68a2d64"
Jul 27 17:41:33.025050 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:33.025017    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9d059cb06c55047ce8985d3061a7a891c071b122ef911b88c91e08cf68a2d64"} err="failed to get container status \"f9d059cb06c55047ce8985d3061a7a891c071b122ef911b88c91e08cf68a2d64\": rpc error: code = NotFound desc = could not find container \"f9d059cb06c55047ce8985d3061a7a891c071b122ef911b88c91e08cf68a2d64\": container with ID starting with f9d059cb06c55047ce8985d3061a7a891c071b122ef911b88c91e08cf68a2d64 not found: ID does not exist"
Jul 27 17:41:33.126595 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:33.126562    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/53a4dc14-0af7-47d4-b5d0-4a66bb723611-config-file\") pod \"53a4dc14-0af7-47d4-b5d0-4a66bb723611\" (UID: \"53a4dc14-0af7-47d4-b5d0-4a66bb723611\") "
Jul 27 17:41:33.126776 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:33.126667    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mx4lf\" (UniqueName: \"kubernetes.io/projected/53a4dc14-0af7-47d4-b5d0-4a66bb723611-kube-api-access-mx4lf\") pod \"53a4dc14-0af7-47d4-b5d0-4a66bb723611\" (UID: \"53a4dc14-0af7-47d4-b5d0-4a66bb723611\") "
Jul 27 17:41:33.127088 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:33.127051    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53a4dc14-0af7-47d4-b5d0-4a66bb723611-config-file" (OuterVolumeSpecName: "config-file") pod "53a4dc14-0af7-47d4-b5d0-4a66bb723611" (UID: "53a4dc14-0af7-47d4-b5d0-4a66bb723611"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:41:33.128838 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:33.128817    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53a4dc14-0af7-47d4-b5d0-4a66bb723611-kube-api-access-mx4lf" (OuterVolumeSpecName: "kube-api-access-mx4lf") pod "53a4dc14-0af7-47d4-b5d0-4a66bb723611" (UID: "53a4dc14-0af7-47d4-b5d0-4a66bb723611"). InnerVolumeSpecName "kube-api-access-mx4lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:41:33.228088 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:33.228055    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx4lf\" (UniqueName: \"kubernetes.io/projected/53a4dc14-0af7-47d4-b5d0-4a66bb723611-kube-api-access-mx4lf\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:41:33.228088 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:33.228083    2557 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/53a4dc14-0af7-47d4-b5d0-4a66bb723611-config-file\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:41:33.339236 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:33.339206    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-q5kfx"]
Jul 27 17:41:33.350920 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:33.350894    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-q5kfx"]
Jul 27 17:41:34.117889 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:34.117858    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53a4dc14-0af7-47d4-b5d0-4a66bb723611" path="/var/lib/kubelet/pods/53a4dc14-0af7-47d4-b5d0-4a66bb723611/volumes"
Jul 27 17:41:43.394336 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:43.394297    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cf4765b85-w4xqt"]
Jul 27 17:41:43.394860 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:43.394634    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53a4dc14-0af7-47d4-b5d0-4a66bb723611" containerName="limitador"
Jul 27 17:41:43.394860 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:43.394645    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="53a4dc14-0af7-47d4-b5d0-4a66bb723611" containerName="limitador"
Jul 27 17:41:43.394860 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:43.394705    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="53a4dc14-0af7-47d4-b5d0-4a66bb723611" containerName="limitador"
Jul 27 17:41:43.398851 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:43.398831    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf4765b85-w4xqt"
Jul 27 17:41:43.402030 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:43.402008    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 27 17:41:43.402162 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:43.402127    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sszrd\""
Jul 27 17:41:43.411384 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:43.411363    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cf4765b85-w4xqt"]
Jul 27 17:41:43.512894 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:43.512859    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/37c62431-f7c3-409a-8d32-15cfa386cec4-tls-cert\") pod \"authorino-7cf4765b85-w4xqt\" (UID: \"37c62431-f7c3-409a-8d32-15cfa386cec4\") " pod="kuadrant-system/authorino-7cf4765b85-w4xqt"
Jul 27 17:41:43.513091 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:43.512917    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4pfd\" (UniqueName: \"kubernetes.io/projected/37c62431-f7c3-409a-8d32-15cfa386cec4-kube-api-access-f4pfd\") pod \"authorino-7cf4765b85-w4xqt\" (UID: \"37c62431-f7c3-409a-8d32-15cfa386cec4\") " pod="kuadrant-system/authorino-7cf4765b85-w4xqt"
Jul 27 17:41:43.614146 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:43.614111    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/37c62431-f7c3-409a-8d32-15cfa386cec4-tls-cert\") pod \"authorino-7cf4765b85-w4xqt\" (UID: \"37c62431-f7c3-409a-8d32-15cfa386cec4\") " pod="kuadrant-system/authorino-7cf4765b85-w4xqt"
Jul 27 17:41:43.614308 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:43.614171    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4pfd\" (UniqueName: \"kubernetes.io/projected/37c62431-f7c3-409a-8d32-15cfa386cec4-kube-api-access-f4pfd\") pod \"authorino-7cf4765b85-w4xqt\" (UID: \"37c62431-f7c3-409a-8d32-15cfa386cec4\") " pod="kuadrant-system/authorino-7cf4765b85-w4xqt"
Jul 27 17:41:43.616668 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:43.616639    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/37c62431-f7c3-409a-8d32-15cfa386cec4-tls-cert\") pod \"authorino-7cf4765b85-w4xqt\" (UID: \"37c62431-f7c3-409a-8d32-15cfa386cec4\") " pod="kuadrant-system/authorino-7cf4765b85-w4xqt"
Jul 27 17:41:43.624331 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:43.624310    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4pfd\" (UniqueName: \"kubernetes.io/projected/37c62431-f7c3-409a-8d32-15cfa386cec4-kube-api-access-f4pfd\") pod \"authorino-7cf4765b85-w4xqt\" (UID: \"37c62431-f7c3-409a-8d32-15cfa386cec4\") " pod="kuadrant-system/authorino-7cf4765b85-w4xqt"
Jul 27 17:41:43.707295 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:43.707273    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf4765b85-w4xqt"
Jul 27 17:41:43.848429 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:43.848393    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cf4765b85-w4xqt"]
Jul 27 17:41:43.851802 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:41:43.851770    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37c62431_f7c3_409a_8d32_15cfa386cec4.slice/crio-cf7c93f0477ad5b216a98887a9cd83709d828ef95fd6ffd8dbe600b310eea18f WatchSource:0}: Error finding container cf7c93f0477ad5b216a98887a9cd83709d828ef95fd6ffd8dbe600b310eea18f: Status 404 returned error can't find the container with id cf7c93f0477ad5b216a98887a9cd83709d828ef95fd6ffd8dbe600b310eea18f
Jul 27 17:41:44.065961 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:44.065870    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cf4765b85-w4xqt" event={"ID":"37c62431-f7c3-409a-8d32-15cfa386cec4","Type":"ContainerStarted","Data":"cf7c93f0477ad5b216a98887a9cd83709d828ef95fd6ffd8dbe600b310eea18f"}
Jul 27 17:41:46.077831 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:46.077718    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cf4765b85-w4xqt" event={"ID":"37c62431-f7c3-409a-8d32-15cfa386cec4","Type":"ContainerStarted","Data":"2e166df12dc0ad2d771bf4ddae16e8f8c85f0e22fb39532736d13b812b4e7eca"}
Jul 27 17:41:46.095350 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:41:46.095294    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7cf4765b85-w4xqt" podStartSLOduration=1.137428206 podStartE2EDuration="3.095277796s" totalImagesPullingTime="1.95784959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:41:46.09357402 +0000 UTC m=+798.580385308" watchObservedRunningTime="2026-07-27 17:41:46.095277796 +0000 UTC m=+798.582089083"
Jul 27 17:42:31.572448 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:42:31.572408    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-ddz92"]
Jul 27 17:42:31.577122 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:42:31.577098    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-ddz92"
Jul 27 17:42:31.579087 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:42:31.579062    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 27 17:42:31.579691 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:42:31.579670    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-9z6v7\""
Jul 27 17:42:31.579798 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:42:31.579692    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 27 17:42:31.579798 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:42:31.579696    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 27 17:42:31.586172 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:42:31.586150    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-ddz92"]
Jul 27 17:42:31.631178 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:42:31.631143    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5d18178-0bcc-449b-ac6a-0e7f41c0ea41-cert\") pod \"odh-model-controller-75d9cdbddf-ddz92\" (UID: \"b5d18178-0bcc-449b-ac6a-0e7f41c0ea41\") " pod="kserve/odh-model-controller-75d9cdbddf-ddz92"
Jul 27 17:42:31.631178 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:42:31.631177    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr2k9\" (UniqueName: \"kubernetes.io/projected/b5d18178-0bcc-449b-ac6a-0e7f41c0ea41-kube-api-access-dr2k9\") pod \"odh-model-controller-75d9cdbddf-ddz92\" (UID: \"b5d18178-0bcc-449b-ac6a-0e7f41c0ea41\") " pod="kserve/odh-model-controller-75d9cdbddf-ddz92"
Jul 27 17:42:31.732161 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:42:31.732127    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5d18178-0bcc-449b-ac6a-0e7f41c0ea41-cert\") pod \"odh-model-controller-75d9cdbddf-ddz92\" (UID: \"b5d18178-0bcc-449b-ac6a-0e7f41c0ea41\") " pod="kserve/odh-model-controller-75d9cdbddf-ddz92"
Jul 27 17:42:31.732386 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:42:31.732171    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr2k9\" (UniqueName: \"kubernetes.io/projected/b5d18178-0bcc-449b-ac6a-0e7f41c0ea41-kube-api-access-dr2k9\") pod \"odh-model-controller-75d9cdbddf-ddz92\" (UID: \"b5d18178-0bcc-449b-ac6a-0e7f41c0ea41\") " pod="kserve/odh-model-controller-75d9cdbddf-ddz92"
Jul 27 17:42:31.734771 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:42:31.734730    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5d18178-0bcc-449b-ac6a-0e7f41c0ea41-cert\") pod \"odh-model-controller-75d9cdbddf-ddz92\" (UID: \"b5d18178-0bcc-449b-ac6a-0e7f41c0ea41\") " pod="kserve/odh-model-controller-75d9cdbddf-ddz92"
Jul 27 17:42:31.739875 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:42:31.739852    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr2k9\" (UniqueName: \"kubernetes.io/projected/b5d18178-0bcc-449b-ac6a-0e7f41c0ea41-kube-api-access-dr2k9\") pod \"odh-model-controller-75d9cdbddf-ddz92\" (UID: \"b5d18178-0bcc-449b-ac6a-0e7f41c0ea41\") " pod="kserve/odh-model-controller-75d9cdbddf-ddz92"
Jul 27 17:42:31.889229 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:42:31.889140    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-ddz92"
Jul 27 17:42:32.016928 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:42:32.016903    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-ddz92"]
Jul 27 17:42:32.019527 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:42:32.019500    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d18178_0bcc_449b_ac6a_0e7f41c0ea41.slice/crio-d33151d6cbd6d84f4563b51887e9bc151012774cbb040e4a9a6e163fcb81df74 WatchSource:0}: Error finding container d33151d6cbd6d84f4563b51887e9bc151012774cbb040e4a9a6e163fcb81df74: Status 404 returned error can't find the container with id d33151d6cbd6d84f4563b51887e9bc151012774cbb040e4a9a6e163fcb81df74
Jul 27 17:42:32.286558 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:42:32.286518    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-ddz92" event={"ID":"b5d18178-0bcc-449b-ac6a-0e7f41c0ea41","Type":"ContainerStarted","Data":"d33151d6cbd6d84f4563b51887e9bc151012774cbb040e4a9a6e163fcb81df74"}
Jul 27 17:42:35.306373 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:42:35.306210    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-ddz92" event={"ID":"b5d18178-0bcc-449b-ac6a-0e7f41c0ea41","Type":"ContainerStarted","Data":"6d7e95ab6a26397f565b47e89f43241983c4405d8c67127b69a57207e8bdc4d0"}
Jul 27 17:42:35.306373 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:42:35.306347    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-ddz92"
Jul 27 17:42:35.322313 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:42:35.322263    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-ddz92" podStartSLOduration=1.388533775 podStartE2EDuration="4.322245813s" totalImagesPullingTime="2.933712038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:42:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:42:35.320687791 +0000 UTC m=+847.807499112" watchObservedRunningTime="2026-07-27 17:42:35.322245813 +0000 UTC m=+847.809057100"
Jul 27 17:42:46.313251 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:42:46.313214    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-ddz92"
Jul 27 17:44:44.272135 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:44.272104    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj"]
Jul 27 17:44:44.275575 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:44.275557    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj"
Jul 27 17:44:44.277734 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:44.277711    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Jul 27 17:44:44.277861 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:44.277825    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Jul 27 17:44:44.278368 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:44.278347    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-pvj4p\""
Jul 27 17:44:44.278431 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:44.278391    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Jul 27 17:44:44.284984 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:44.284959    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj"]
Jul 27 17:44:44.418387 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:44.418355    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a6178f0-b45b-4d29-8858-9db71a2c5be7-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-x8qmj\" (UID: \"8a6178f0-b45b-4d29-8858-9db71a2c5be7\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj"
Jul 27 17:44:44.418595 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:44.418393    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a6178f0-b45b-4d29-8858-9db71a2c5be7-dshm\") pod \"fc-smoke-test-kserve-7f4687897-x8qmj\" (UID: \"8a6178f0-b45b-4d29-8858-9db71a2c5be7\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj"
Jul 27 17:44:44.418595 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:44.418425    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmv47\" (UniqueName: \"kubernetes.io/projected/8a6178f0-b45b-4d29-8858-9db71a2c5be7-kube-api-access-nmv47\") pod \"fc-smoke-test-kserve-7f4687897-x8qmj\" (UID: \"8a6178f0-b45b-4d29-8858-9db71a2c5be7\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj"
Jul 27 17:44:44.418595 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:44.418496    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a6178f0-b45b-4d29-8858-9db71a2c5be7-home\") pod \"fc-smoke-test-kserve-7f4687897-x8qmj\" (UID: \"8a6178f0-b45b-4d29-8858-9db71a2c5be7\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj"
Jul 27 17:44:44.418595 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:44.418572    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a6178f0-b45b-4d29-8858-9db71a2c5be7-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-x8qmj\" (UID: \"8a6178f0-b45b-4d29-8858-9db71a2c5be7\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj"
Jul 27 17:44:44.418746 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:44.418612    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a6178f0-b45b-4d29-8858-9db71a2c5be7-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-x8qmj\" (UID: \"8a6178f0-b45b-4d29-8858-9db71a2c5be7\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj"
Jul 27 17:44:44.519275 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:44.519246    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a6178f0-b45b-4d29-8858-9db71a2c5be7-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-x8qmj\" (UID: \"8a6178f0-b45b-4d29-8858-9db71a2c5be7\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj"
Jul 27 17:44:44.519463 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:44.519285    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a6178f0-b45b-4d29-8858-9db71a2c5be7-dshm\") pod \"fc-smoke-test-kserve-7f4687897-x8qmj\" (UID: \"8a6178f0-b45b-4d29-8858-9db71a2c5be7\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj"
Jul 27 17:44:44.519463 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:44.519302    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmv47\" (UniqueName: \"kubernetes.io/projected/8a6178f0-b45b-4d29-8858-9db71a2c5be7-kube-api-access-nmv47\") pod \"fc-smoke-test-kserve-7f4687897-x8qmj\" (UID: \"8a6178f0-b45b-4d29-8858-9db71a2c5be7\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj"
Jul 27 17:44:44.519463 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:44.519323    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a6178f0-b45b-4d29-8858-9db71a2c5be7-home\") pod \"fc-smoke-test-kserve-7f4687897-x8qmj\" (UID: \"8a6178f0-b45b-4d29-8858-9db71a2c5be7\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj"
Jul 27 17:44:44.519463 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:44.519352    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a6178f0-b45b-4d29-8858-9db71a2c5be7-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-x8qmj\" (UID: \"8a6178f0-b45b-4d29-8858-9db71a2c5be7\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj"
Jul 27 17:44:44.519463 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:44.519460    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a6178f0-b45b-4d29-8858-9db71a2c5be7-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-x8qmj\" (UID: \"8a6178f0-b45b-4d29-8858-9db71a2c5be7\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj"
Jul 27 17:44:44.519708 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:44.519685    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a6178f0-b45b-4d29-8858-9db71a2c5be7-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-x8qmj\" (UID: \"8a6178f0-b45b-4d29-8858-9db71a2c5be7\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj"
Jul 27 17:44:44.519796 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:44.519745    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a6178f0-b45b-4d29-8858-9db71a2c5be7-home\") pod \"fc-smoke-test-kserve-7f4687897-x8qmj\" (UID: \"8a6178f0-b45b-4d29-8858-9db71a2c5be7\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj"
Jul 27 17:44:44.519864 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:44.519848    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a6178f0-b45b-4d29-8858-9db71a2c5be7-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-x8qmj\" (UID: \"8a6178f0-b45b-4d29-8858-9db71a2c5be7\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj"
Jul 27 17:44:44.521573 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:44.521554    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a6178f0-b45b-4d29-8858-9db71a2c5be7-dshm\") pod \"fc-smoke-test-kserve-7f4687897-x8qmj\" (UID: \"8a6178f0-b45b-4d29-8858-9db71a2c5be7\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj"
Jul 27 17:44:44.521858 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:44.521839    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a6178f0-b45b-4d29-8858-9db71a2c5be7-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-x8qmj\" (UID: \"8a6178f0-b45b-4d29-8858-9db71a2c5be7\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj"
Jul 27 17:44:44.527404 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:44.527348    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmv47\" (UniqueName: \"kubernetes.io/projected/8a6178f0-b45b-4d29-8858-9db71a2c5be7-kube-api-access-nmv47\") pod \"fc-smoke-test-kserve-7f4687897-x8qmj\" (UID: \"8a6178f0-b45b-4d29-8858-9db71a2c5be7\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj"
Jul 27 17:44:44.586706 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:44.586683    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj"
Jul 27 17:44:44.723383 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:44.723360    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj"]
Jul 27 17:44:44.725643 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:44:44.725612    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6178f0_b45b_4d29_8858_9db71a2c5be7.slice/crio-6c611e19ec64fb57685cc9c744d78beddff7617c0b5ccc29c5c833043861ecc8 WatchSource:0}: Error finding container 6c611e19ec64fb57685cc9c744d78beddff7617c0b5ccc29c5c833043861ecc8: Status 404 returned error can't find the container with id 6c611e19ec64fb57685cc9c744d78beddff7617c0b5ccc29c5c833043861ecc8
Jul 27 17:44:44.727587 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:44.727556    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:44:44.903393 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:44.903308    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj" event={"ID":"8a6178f0-b45b-4d29-8858-9db71a2c5be7","Type":"ContainerStarted","Data":"6c611e19ec64fb57685cc9c744d78beddff7617c0b5ccc29c5c833043861ecc8"}
Jul 27 17:44:46.914534 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:46.914493    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj" event={"ID":"8a6178f0-b45b-4d29-8858-9db71a2c5be7","Type":"ContainerStarted","Data":"78003c8148f6cf93b1fe8961e8c57c56e1b1e3621e31f6dd649c3b171f275630"}
Jul 27 17:44:46.931456 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:46.931407    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj" podStartSLOduration=1.364183498 podStartE2EDuration="2.93139243s" totalImagesPullingTime="1.567208932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:44:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:44:46.929778716 +0000 UTC m=+979.416590229" watchObservedRunningTime="2026-07-27 17:44:46.93139243 +0000 UTC m=+979.418203716"
Jul 27 17:44:47.587288 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:47.587255    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj"
Jul 27 17:44:54.587318 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:54.587233    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj"
Jul 27 17:44:54.599923 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:54.599896    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj"
Jul 27 17:44:54.961812 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:44:54.961780    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj"
Jul 27 17:45:25.710708 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:25.710669    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj"]
Jul 27 17:45:25.721089 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:45:25.721052    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:45:25.721089 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:45:25.721089 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 17:45:25.721089 ip-10-0-131-114 kubenswrapper[2557]:  > containerID="78003c8148f6cf93b1fe8961e8c57c56e1b1e3621e31f6dd649c3b171f275630" cmd=["/bin/sleep","15"]
Jul 27 17:45:25.721407 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:25.721109    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj" message="" err=<
Jul 27 17:45:25.721407 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:45:25.721407 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 17:45:25.721407 ip-10-0-131-114 kubenswrapper[2557]:  >
Jul 27 17:45:25.721407 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:45:25.721121    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:45:25.721407 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:45:25.721407 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 17:45:25.721407 ip-10-0-131-114 kubenswrapper[2557]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj" podUID="8a6178f0-b45b-4d29-8858-9db71a2c5be7" containerName="main" containerID="cri-o://78003c8148f6cf93b1fe8961e8c57c56e1b1e3621e31f6dd649c3b171f275630"
Jul 27 17:45:25.721407 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:25.721163    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj" podUID="8a6178f0-b45b-4d29-8858-9db71a2c5be7" containerName="main" containerID="cri-o://78003c8148f6cf93b1fe8961e8c57c56e1b1e3621e31f6dd649c3b171f275630" gracePeriod=60
Jul 27 17:45:25.952069 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:25.951987    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj" podUID="8a6178f0-b45b-4d29-8858-9db71a2c5be7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.30:8000/health\": dial tcp 10.132.0.30:8000: connect: connection refused"
Jul 27 17:45:26.087720 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:26.087693    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj"
Jul 27 17:45:26.094155 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:26.093998    2557 generic.go:356] "Generic (PLEG): container finished" podID="8a6178f0-b45b-4d29-8858-9db71a2c5be7" containerID="78003c8148f6cf93b1fe8961e8c57c56e1b1e3621e31f6dd649c3b171f275630" exitCode=0
Jul 27 17:45:26.094155 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:26.094048    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj" event={"ID":"8a6178f0-b45b-4d29-8858-9db71a2c5be7","Type":"ContainerDied","Data":"78003c8148f6cf93b1fe8961e8c57c56e1b1e3621e31f6dd649c3b171f275630"}
Jul 27 17:45:26.094155 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:26.094113    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj" event={"ID":"8a6178f0-b45b-4d29-8858-9db71a2c5be7","Type":"ContainerDied","Data":"6c611e19ec64fb57685cc9c744d78beddff7617c0b5ccc29c5c833043861ecc8"}
Jul 27 17:45:26.094155 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:26.094054    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj"
Jul 27 17:45:26.094155 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:26.094156    2557 scope.go:117] "RemoveContainer" containerID="78003c8148f6cf93b1fe8961e8c57c56e1b1e3621e31f6dd649c3b171f275630"
Jul 27 17:45:26.103601 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:26.103582    2557 scope.go:117] "RemoveContainer" containerID="78003c8148f6cf93b1fe8961e8c57c56e1b1e3621e31f6dd649c3b171f275630"
Jul 27 17:45:26.103913 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:45:26.103891    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78003c8148f6cf93b1fe8961e8c57c56e1b1e3621e31f6dd649c3b171f275630\": container with ID starting with 78003c8148f6cf93b1fe8961e8c57c56e1b1e3621e31f6dd649c3b171f275630 not found: ID does not exist" containerID="78003c8148f6cf93b1fe8961e8c57c56e1b1e3621e31f6dd649c3b171f275630"
Jul 27 17:45:26.104025 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:26.103923    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78003c8148f6cf93b1fe8961e8c57c56e1b1e3621e31f6dd649c3b171f275630"} err="failed to get container status \"78003c8148f6cf93b1fe8961e8c57c56e1b1e3621e31f6dd649c3b171f275630\": rpc error: code = NotFound desc = could not find container \"78003c8148f6cf93b1fe8961e8c57c56e1b1e3621e31f6dd649c3b171f275630\": container with ID starting with 78003c8148f6cf93b1fe8961e8c57c56e1b1e3621e31f6dd649c3b171f275630 not found: ID does not exist"
Jul 27 17:45:26.181966 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:26.181935    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a6178f0-b45b-4d29-8858-9db71a2c5be7-tmp-dir\") pod \"8a6178f0-b45b-4d29-8858-9db71a2c5be7\" (UID: \"8a6178f0-b45b-4d29-8858-9db71a2c5be7\") "
Jul 27 17:45:26.182141 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:26.181981    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a6178f0-b45b-4d29-8858-9db71a2c5be7-home\") pod \"8a6178f0-b45b-4d29-8858-9db71a2c5be7\" (UID: \"8a6178f0-b45b-4d29-8858-9db71a2c5be7\") "
Jul 27 17:45:26.182141 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:26.182010    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmv47\" (UniqueName: \"kubernetes.io/projected/8a6178f0-b45b-4d29-8858-9db71a2c5be7-kube-api-access-nmv47\") pod \"8a6178f0-b45b-4d29-8858-9db71a2c5be7\" (UID: \"8a6178f0-b45b-4d29-8858-9db71a2c5be7\") "
Jul 27 17:45:26.182141 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:26.182044    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a6178f0-b45b-4d29-8858-9db71a2c5be7-dshm\") pod \"8a6178f0-b45b-4d29-8858-9db71a2c5be7\" (UID: \"8a6178f0-b45b-4d29-8858-9db71a2c5be7\") "
Jul 27 17:45:26.182141 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:26.182113    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a6178f0-b45b-4d29-8858-9db71a2c5be7-tls-certs\") pod \"8a6178f0-b45b-4d29-8858-9db71a2c5be7\" (UID: \"8a6178f0-b45b-4d29-8858-9db71a2c5be7\") "
Jul 27 17:45:26.182141 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:26.182139    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a6178f0-b45b-4d29-8858-9db71a2c5be7-model-cache\") pod \"8a6178f0-b45b-4d29-8858-9db71a2c5be7\" (UID: \"8a6178f0-b45b-4d29-8858-9db71a2c5be7\") "
Jul 27 17:45:26.182411 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:26.182202    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a6178f0-b45b-4d29-8858-9db71a2c5be7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8a6178f0-b45b-4d29-8858-9db71a2c5be7" (UID: "8a6178f0-b45b-4d29-8858-9db71a2c5be7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:45:26.182411 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:26.182282    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a6178f0-b45b-4d29-8858-9db71a2c5be7-home" (OuterVolumeSpecName: "home") pod "8a6178f0-b45b-4d29-8858-9db71a2c5be7" (UID: "8a6178f0-b45b-4d29-8858-9db71a2c5be7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:45:26.182534 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:26.182432    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a6178f0-b45b-4d29-8858-9db71a2c5be7-model-cache" (OuterVolumeSpecName: "model-cache") pod "8a6178f0-b45b-4d29-8858-9db71a2c5be7" (UID: "8a6178f0-b45b-4d29-8858-9db71a2c5be7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:45:26.182534 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:26.182499    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a6178f0-b45b-4d29-8858-9db71a2c5be7-tmp-dir\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:45:26.182534 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:26.182515    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a6178f0-b45b-4d29-8858-9db71a2c5be7-home\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:45:26.184375 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:26.184343    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a6178f0-b45b-4d29-8858-9db71a2c5be7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a6178f0-b45b-4d29-8858-9db71a2c5be7" (UID: "8a6178f0-b45b-4d29-8858-9db71a2c5be7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:45:26.184647 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:26.184622    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a6178f0-b45b-4d29-8858-9db71a2c5be7-dshm" (OuterVolumeSpecName: "dshm") pod "8a6178f0-b45b-4d29-8858-9db71a2c5be7" (UID: "8a6178f0-b45b-4d29-8858-9db71a2c5be7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:45:26.184647 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:26.184637    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a6178f0-b45b-4d29-8858-9db71a2c5be7-kube-api-access-nmv47" (OuterVolumeSpecName: "kube-api-access-nmv47") pod "8a6178f0-b45b-4d29-8858-9db71a2c5be7" (UID: "8a6178f0-b45b-4d29-8858-9db71a2c5be7"). InnerVolumeSpecName "kube-api-access-nmv47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:45:26.283207 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:26.283128    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmv47\" (UniqueName: \"kubernetes.io/projected/8a6178f0-b45b-4d29-8858-9db71a2c5be7-kube-api-access-nmv47\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:45:26.283207 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:26.283153    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a6178f0-b45b-4d29-8858-9db71a2c5be7-dshm\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:45:26.283207 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:26.283162    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a6178f0-b45b-4d29-8858-9db71a2c5be7-tls-certs\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:45:26.283207 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:26.283173    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a6178f0-b45b-4d29-8858-9db71a2c5be7-model-cache\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:45:26.421269 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:26.421240    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj"]
Jul 27 17:45:26.424840 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:26.424807    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x8qmj"]
Jul 27 17:45:28.119024 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:28.118992    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a6178f0-b45b-4d29-8858-9db71a2c5be7" path="/var/lib/kubelet/pods/8a6178f0-b45b-4d29-8858-9db71a2c5be7/volumes"
Jul 27 17:45:57.791704 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:57.791674    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp"]
Jul 27 17:45:57.792291 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:57.792078    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a6178f0-b45b-4d29-8858-9db71a2c5be7" containerName="main"
Jul 27 17:45:57.792291 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:57.792091    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a6178f0-b45b-4d29-8858-9db71a2c5be7" containerName="main"
Jul 27 17:45:57.792291 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:57.792160    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a6178f0-b45b-4d29-8858-9db71a2c5be7" containerName="main"
Jul 27 17:45:57.795475 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:57.795452    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp"
Jul 27 17:45:57.797711 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:57.797690    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-7jz5v\""
Jul 27 17:45:57.798336 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:57.798317    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Jul 27 17:45:57.798424 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:57.798347    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Jul 27 17:45:57.798424 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:57.798391    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Jul 27 17:45:57.804279 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:57.804258    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp"]
Jul 27 17:45:57.840845 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:57.840794    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpd75\" (UniqueName: \"kubernetes.io/projected/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-kube-api-access-tpd75\") pod \"fc-concurrency-test-kserve-b5d44467c-vpfzp\" (UID: \"46b72f4b-15ef-4e67-8949-e64cf32ba5d8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp"
Jul 27 17:45:57.840979 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:57.840873    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-vpfzp\" (UID: \"46b72f4b-15ef-4e67-8949-e64cf32ba5d8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp"
Jul 27 17:45:57.840979 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:57.840905    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-vpfzp\" (UID: \"46b72f4b-15ef-4e67-8949-e64cf32ba5d8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp"
Jul 27 17:45:57.841049 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:57.841006    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-vpfzp\" (UID: \"46b72f4b-15ef-4e67-8949-e64cf32ba5d8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp"
Jul 27 17:45:57.841049 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:57.841031    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-home\") pod \"fc-concurrency-test-kserve-b5d44467c-vpfzp\" (UID: \"46b72f4b-15ef-4e67-8949-e64cf32ba5d8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp"
Jul 27 17:45:57.841116 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:57.841074    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-vpfzp\" (UID: \"46b72f4b-15ef-4e67-8949-e64cf32ba5d8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp"
Jul 27 17:45:57.941615 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:57.941579    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-vpfzp\" (UID: \"46b72f4b-15ef-4e67-8949-e64cf32ba5d8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp"
Jul 27 17:45:57.941815 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:57.941647    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-vpfzp\" (UID: \"46b72f4b-15ef-4e67-8949-e64cf32ba5d8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp"
Jul 27 17:45:57.941815 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:57.941667    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-home\") pod \"fc-concurrency-test-kserve-b5d44467c-vpfzp\" (UID: \"46b72f4b-15ef-4e67-8949-e64cf32ba5d8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp"
Jul 27 17:45:57.941815 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:57.941701    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-vpfzp\" (UID: \"46b72f4b-15ef-4e67-8949-e64cf32ba5d8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp"
Jul 27 17:45:57.941815 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:57.941719    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpd75\" (UniqueName: \"kubernetes.io/projected/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-kube-api-access-tpd75\") pod \"fc-concurrency-test-kserve-b5d44467c-vpfzp\" (UID: \"46b72f4b-15ef-4e67-8949-e64cf32ba5d8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp"
Jul 27 17:45:57.941965 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:57.941861    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-vpfzp\" (UID: \"46b72f4b-15ef-4e67-8949-e64cf32ba5d8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp"
Jul 27 17:45:57.942091 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:57.942070    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-home\") pod \"fc-concurrency-test-kserve-b5d44467c-vpfzp\" (UID: \"46b72f4b-15ef-4e67-8949-e64cf32ba5d8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp"
Jul 27 17:45:57.942138 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:57.942079    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-vpfzp\" (UID: \"46b72f4b-15ef-4e67-8949-e64cf32ba5d8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp"
Jul 27 17:45:57.942197 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:57.942137    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-vpfzp\" (UID: \"46b72f4b-15ef-4e67-8949-e64cf32ba5d8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp"
Jul 27 17:45:57.944010 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:57.943990    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-vpfzp\" (UID: \"46b72f4b-15ef-4e67-8949-e64cf32ba5d8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp"
Jul 27 17:45:57.944329 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:57.944309    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-vpfzp\" (UID: \"46b72f4b-15ef-4e67-8949-e64cf32ba5d8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp"
Jul 27 17:45:57.951360 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:57.951335    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpd75\" (UniqueName: \"kubernetes.io/projected/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-kube-api-access-tpd75\") pod \"fc-concurrency-test-kserve-b5d44467c-vpfzp\" (UID: \"46b72f4b-15ef-4e67-8949-e64cf32ba5d8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp"
Jul 27 17:45:58.107587 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:58.107501    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp"
Jul 27 17:45:58.243601 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:58.243572    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp"]
Jul 27 17:45:58.245789 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:45:58.245736    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b72f4b_15ef_4e67_8949_e64cf32ba5d8.slice/crio-b97bf0f1ed75d8daf34e200c4818f5a62adc375deb4f168e65ca9ceac15ef662 WatchSource:0}: Error finding container b97bf0f1ed75d8daf34e200c4818f5a62adc375deb4f168e65ca9ceac15ef662: Status 404 returned error can't find the container with id b97bf0f1ed75d8daf34e200c4818f5a62adc375deb4f168e65ca9ceac15ef662
Jul 27 17:45:59.241268 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:59.241232    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp" event={"ID":"46b72f4b-15ef-4e67-8949-e64cf32ba5d8","Type":"ContainerStarted","Data":"db530596f4d642a2f4b6babf2750c146c525e38e95c36599a352fef89dd4a348"}
Jul 27 17:45:59.241706 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:59.241284    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp" event={"ID":"46b72f4b-15ef-4e67-8949-e64cf32ba5d8","Type":"ContainerStarted","Data":"b97bf0f1ed75d8daf34e200c4818f5a62adc375deb4f168e65ca9ceac15ef662"}
Jul 27 17:45:59.258190 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:45:59.258141    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp" podStartSLOduration=2.258127421 podStartE2EDuration="2.258127421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:45:59.25634698 +0000 UTC m=+1051.743158267" watchObservedRunningTime="2026-07-27 17:45:59.258127421 +0000 UTC m=+1051.744938708"
Jul 27 17:46:00.108036 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:00.108003    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp"
Jul 27 17:46:08.108106 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:08.108059    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp"
Jul 27 17:46:08.121229 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:08.121203    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp"
Jul 27 17:46:08.293573 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:08.293548    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp"
Jul 27 17:46:35.711090 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:35.711010    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp"]
Jul 27 17:46:35.721987 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:46:35.721944    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:46:35.721987 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:46:35.721987 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 17:46:35.721987 ip-10-0-131-114 kubenswrapper[2557]:  > containerID="db530596f4d642a2f4b6babf2750c146c525e38e95c36599a352fef89dd4a348" cmd=["/bin/sleep","15"]
Jul 27 17:46:35.722275 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:35.722010    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp" message="" err=<
Jul 27 17:46:35.722275 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:46:35.722275 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 17:46:35.722275 ip-10-0-131-114 kubenswrapper[2557]:  >
Jul 27 17:46:35.722275 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:46:35.722028    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:46:35.722275 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:46:35.722275 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 17:46:35.722275 ip-10-0-131-114 kubenswrapper[2557]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp" podUID="46b72f4b-15ef-4e67-8949-e64cf32ba5d8" containerName="main" containerID="cri-o://db530596f4d642a2f4b6babf2750c146c525e38e95c36599a352fef89dd4a348"
Jul 27 17:46:35.722275 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:35.722067    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp" podUID="46b72f4b-15ef-4e67-8949-e64cf32ba5d8" containerName="main" containerID="cri-o://db530596f4d642a2f4b6babf2750c146c525e38e95c36599a352fef89dd4a348" gracePeriod=60
Jul 27 17:46:36.086037 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:36.086012    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp"
Jul 27 17:46:36.171801 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:36.171767    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-model-cache\") pod \"46b72f4b-15ef-4e67-8949-e64cf32ba5d8\" (UID: \"46b72f4b-15ef-4e67-8949-e64cf32ba5d8\") "
Jul 27 17:46:36.171982 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:36.171814    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-home\") pod \"46b72f4b-15ef-4e67-8949-e64cf32ba5d8\" (UID: \"46b72f4b-15ef-4e67-8949-e64cf32ba5d8\") "
Jul 27 17:46:36.171982 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:36.171846    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-tls-certs\") pod \"46b72f4b-15ef-4e67-8949-e64cf32ba5d8\" (UID: \"46b72f4b-15ef-4e67-8949-e64cf32ba5d8\") "
Jul 27 17:46:36.171982 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:36.171868    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpd75\" (UniqueName: \"kubernetes.io/projected/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-kube-api-access-tpd75\") pod \"46b72f4b-15ef-4e67-8949-e64cf32ba5d8\" (UID: \"46b72f4b-15ef-4e67-8949-e64cf32ba5d8\") "
Jul 27 17:46:36.171982 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:36.171901    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-dshm\") pod \"46b72f4b-15ef-4e67-8949-e64cf32ba5d8\" (UID: \"46b72f4b-15ef-4e67-8949-e64cf32ba5d8\") "
Jul 27 17:46:36.171982 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:36.171950    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-tmp-dir\") pod \"46b72f4b-15ef-4e67-8949-e64cf32ba5d8\" (UID: \"46b72f4b-15ef-4e67-8949-e64cf32ba5d8\") "
Jul 27 17:46:36.172302 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:36.171989    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "46b72f4b-15ef-4e67-8949-e64cf32ba5d8" (UID: "46b72f4b-15ef-4e67-8949-e64cf32ba5d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:46:36.172302 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:36.172060    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-home" (OuterVolumeSpecName: "home") pod "46b72f4b-15ef-4e67-8949-e64cf32ba5d8" (UID: "46b72f4b-15ef-4e67-8949-e64cf32ba5d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:46:36.172302 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:36.172213    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "46b72f4b-15ef-4e67-8949-e64cf32ba5d8" (UID: "46b72f4b-15ef-4e67-8949-e64cf32ba5d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:46:36.172302 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:36.172265    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-model-cache\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:46:36.172302 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:36.172286    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-home\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:46:36.174135 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:36.174105    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-dshm" (OuterVolumeSpecName: "dshm") pod "46b72f4b-15ef-4e67-8949-e64cf32ba5d8" (UID: "46b72f4b-15ef-4e67-8949-e64cf32ba5d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:46:36.174352 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:36.174326    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "46b72f4b-15ef-4e67-8949-e64cf32ba5d8" (UID: "46b72f4b-15ef-4e67-8949-e64cf32ba5d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:46:36.175567 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:36.175547    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-kube-api-access-tpd75" (OuterVolumeSpecName: "kube-api-access-tpd75") pod "46b72f4b-15ef-4e67-8949-e64cf32ba5d8" (UID: "46b72f4b-15ef-4e67-8949-e64cf32ba5d8"). InnerVolumeSpecName "kube-api-access-tpd75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:46:36.272946 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:36.272860    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-tls-certs\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:46:36.272946 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:36.272889    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpd75\" (UniqueName: \"kubernetes.io/projected/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-kube-api-access-tpd75\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:46:36.272946 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:36.272903    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-dshm\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:46:36.272946 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:36.272915    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46b72f4b-15ef-4e67-8949-e64cf32ba5d8-tmp-dir\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:46:36.409690 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:36.409660    2557 generic.go:356] "Generic (PLEG): container finished" podID="46b72f4b-15ef-4e67-8949-e64cf32ba5d8" containerID="db530596f4d642a2f4b6babf2750c146c525e38e95c36599a352fef89dd4a348" exitCode=0
Jul 27 17:46:36.409881 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:36.409708    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp"
Jul 27 17:46:36.409881 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:36.409733    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp" event={"ID":"46b72f4b-15ef-4e67-8949-e64cf32ba5d8","Type":"ContainerDied","Data":"db530596f4d642a2f4b6babf2750c146c525e38e95c36599a352fef89dd4a348"}
Jul 27 17:46:36.409881 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:36.409804    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp" event={"ID":"46b72f4b-15ef-4e67-8949-e64cf32ba5d8","Type":"ContainerDied","Data":"b97bf0f1ed75d8daf34e200c4818f5a62adc375deb4f168e65ca9ceac15ef662"}
Jul 27 17:46:36.409881 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:36.409832    2557 scope.go:117] "RemoveContainer" containerID="db530596f4d642a2f4b6babf2750c146c525e38e95c36599a352fef89dd4a348"
Jul 27 17:46:36.419645 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:36.419629    2557 scope.go:117] "RemoveContainer" containerID="db530596f4d642a2f4b6babf2750c146c525e38e95c36599a352fef89dd4a348"
Jul 27 17:46:36.419912 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:46:36.419895    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db530596f4d642a2f4b6babf2750c146c525e38e95c36599a352fef89dd4a348\": container with ID starting with db530596f4d642a2f4b6babf2750c146c525e38e95c36599a352fef89dd4a348 not found: ID does not exist" containerID="db530596f4d642a2f4b6babf2750c146c525e38e95c36599a352fef89dd4a348"
Jul 27 17:46:36.419983 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:36.419920    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db530596f4d642a2f4b6babf2750c146c525e38e95c36599a352fef89dd4a348"} err="failed to get container status \"db530596f4d642a2f4b6babf2750c146c525e38e95c36599a352fef89dd4a348\": rpc error: code = NotFound desc = could not find container \"db530596f4d642a2f4b6babf2750c146c525e38e95c36599a352fef89dd4a348\": container with ID starting with db530596f4d642a2f4b6babf2750c146c525e38e95c36599a352fef89dd4a348 not found: ID does not exist"
Jul 27 17:46:36.447715 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:36.447689    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp"]
Jul 27 17:46:36.453013 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:36.452991    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-vpfzp"]
Jul 27 17:46:38.118551 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:38.118518    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46b72f4b-15ef-4e67-8949-e64cf32ba5d8" path="/var/lib/kubelet/pods/46b72f4b-15ef-4e67-8949-e64cf32ba5d8/volumes"
Jul 27 17:46:56.033220 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.033191    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"]
Jul 27 17:46:56.033707 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.033688    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46b72f4b-15ef-4e67-8949-e64cf32ba5d8" containerName="main"
Jul 27 17:46:56.033796 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.033710    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="46b72f4b-15ef-4e67-8949-e64cf32ba5d8" containerName="main"
Jul 27 17:46:56.033855 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.033823    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="46b72f4b-15ef-4e67-8949-e64cf32ba5d8" containerName="main"
Jul 27 17:46:56.036783 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.036765    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:46:56.039269 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.039244    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Jul 27 17:46:56.039376 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.039244    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Jul 27 17:46:56.039376 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.039327    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-h9cl9\""
Jul 27 17:46:56.039600 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.039584    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Jul 27 17:46:56.053323 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.053303    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"]
Jul 27 17:46:56.132236 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.132205    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e2302da-91ff-4ac3-8095-90389a25c8c0-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-7px2c\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:46:56.132236 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.132239    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e2302da-91ff-4ac3-8095-90389a25c8c0-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-7px2c\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:46:56.132484 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.132264    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e2302da-91ff-4ac3-8095-90389a25c8c0-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-7px2c\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:46:56.132484 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.132346    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e2302da-91ff-4ac3-8095-90389a25c8c0-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-7px2c\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:46:56.132484 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.132376    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8j95\" (UniqueName: \"kubernetes.io/projected/9e2302da-91ff-4ac3-8095-90389a25c8c0-kube-api-access-g8j95\") pod \"router-gateway-1-openshift-default-5b4d9458bd-7px2c\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:46:56.132484 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.132407    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2302da-91ff-4ac3-8095-90389a25c8c0-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-7px2c\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:46:56.132691 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.132510    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e2302da-91ff-4ac3-8095-90389a25c8c0-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-7px2c\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:46:56.132691 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.132577    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2302da-91ff-4ac3-8095-90389a25c8c0-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-7px2c\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:46:56.132883 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.132867    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e2302da-91ff-4ac3-8095-90389a25c8c0-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-7px2c\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:46:56.233744 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.233706    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e2302da-91ff-4ac3-8095-90389a25c8c0-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-7px2c\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:46:56.233744 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.233741    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e2302da-91ff-4ac3-8095-90389a25c8c0-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-7px2c\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:46:56.234022 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.233797    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e2302da-91ff-4ac3-8095-90389a25c8c0-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-7px2c\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:46:56.234022 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.233814    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8j95\" (UniqueName: \"kubernetes.io/projected/9e2302da-91ff-4ac3-8095-90389a25c8c0-kube-api-access-g8j95\") pod \"router-gateway-1-openshift-default-5b4d9458bd-7px2c\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:46:56.234022 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.233833    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2302da-91ff-4ac3-8095-90389a25c8c0-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-7px2c\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:46:56.234190 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.234065    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e2302da-91ff-4ac3-8095-90389a25c8c0-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-7px2c\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:46:56.234190 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.234131    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2302da-91ff-4ac3-8095-90389a25c8c0-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-7px2c\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:46:56.234190 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.234186    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e2302da-91ff-4ac3-8095-90389a25c8c0-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-7px2c\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:46:56.234369 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.234214    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e2302da-91ff-4ac3-8095-90389a25c8c0-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-7px2c\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:46:56.234369 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.234233    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2302da-91ff-4ac3-8095-90389a25c8c0-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-7px2c\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:46:56.234469 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.234407    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e2302da-91ff-4ac3-8095-90389a25c8c0-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-7px2c\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:46:56.234553 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.234527    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e2302da-91ff-4ac3-8095-90389a25c8c0-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-7px2c\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:46:56.234615 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.234569    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2302da-91ff-4ac3-8095-90389a25c8c0-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-7px2c\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:46:56.235002 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.234976    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e2302da-91ff-4ac3-8095-90389a25c8c0-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-7px2c\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:46:56.236003 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.235983    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e2302da-91ff-4ac3-8095-90389a25c8c0-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-7px2c\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:46:56.236343 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.236325    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e2302da-91ff-4ac3-8095-90389a25c8c0-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-7px2c\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:46:56.241675 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.241652    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e2302da-91ff-4ac3-8095-90389a25c8c0-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-7px2c\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:46:56.241925 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.241905    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8j95\" (UniqueName: \"kubernetes.io/projected/9e2302da-91ff-4ac3-8095-90389a25c8c0-kube-api-access-g8j95\") pod \"router-gateway-1-openshift-default-5b4d9458bd-7px2c\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:46:56.348473 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.348406    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:46:56.482596 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.482571    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"]
Jul 27 17:46:56.484785 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:46:56.484744    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e2302da_91ff_4ac3_8095_90389a25c8c0.slice/crio-e04b1e81e63aeff36a3e0bb8374e88a08cf3b18b922892d9abc4fe66e24a1abb WatchSource:0}: Error finding container e04b1e81e63aeff36a3e0bb8374e88a08cf3b18b922892d9abc4fe66e24a1abb: Status 404 returned error can't find the container with id e04b1e81e63aeff36a3e0bb8374e88a08cf3b18b922892d9abc4fe66e24a1abb
Jul 27 17:46:56.502935 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:56.502911    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c" event={"ID":"9e2302da-91ff-4ac3-8095-90389a25c8c0","Type":"ContainerStarted","Data":"e04b1e81e63aeff36a3e0bb8374e88a08cf3b18b922892d9abc4fe66e24a1abb"}
Jul 27 17:46:59.125689 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:59.125638    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 27 17:46:59.125979 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:59.125731    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 27 17:46:59.125979 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:59.125793    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 27 17:46:59.522750 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:59.522717    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c" event={"ID":"9e2302da-91ff-4ac3-8095-90389a25c8c0","Type":"ContainerStarted","Data":"c63337f5805f7c3c258ba9879a52852c51fffc472959169fab708b20b4f1f088"}
Jul 27 17:46:59.542805 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:46:59.542740    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c" podStartSLOduration=0.904176879 podStartE2EDuration="3.5427251s" totalImagesPullingTime="2.638548221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:46:59.540504952 +0000 UTC m=+1112.027316253" watchObservedRunningTime="2026-07-27 17:46:59.5427251 +0000 UTC m=+1112.029536387"
Jul 27 17:47:00.349051 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:00.349008    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:47:00.354356 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:00.354333    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:47:00.527306 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:00.527272    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:47:00.528706 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:00.528687    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:47:01.383459 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:01.383430    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z"]
Jul 27 17:47:01.386742 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:01.386722    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z"
Jul 27 17:47:01.389408 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:01.389389    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Jul 27 17:47:01.389515 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:01.389390    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-x89zg\""
Jul 27 17:47:01.399236 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:01.399215    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z"]
Jul 27 17:47:01.480109 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:01.480075    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z\" (UID: \"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z"
Jul 27 17:47:01.480109 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:01.480107    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z\" (UID: \"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z"
Jul 27 17:47:01.480371 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:01.480133    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw7wp\" (UniqueName: \"kubernetes.io/projected/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-kube-api-access-mw7wp\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z\" (UID: \"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z"
Jul 27 17:47:01.480371 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:01.480245    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z\" (UID: \"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z"
Jul 27 17:47:01.480371 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:01.480289    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z\" (UID: \"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z"
Jul 27 17:47:01.480371 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:01.480333    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z\" (UID: \"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z"
Jul 27 17:47:01.580829 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:01.580796    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z\" (UID: \"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z"
Jul 27 17:47:01.580829 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:01.580829    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z\" (UID: \"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z"
Jul 27 17:47:01.581038 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:01.580852    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw7wp\" (UniqueName: \"kubernetes.io/projected/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-kube-api-access-mw7wp\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z\" (UID: \"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z"
Jul 27 17:47:01.581038 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:01.580910    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z\" (UID: \"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z"
Jul 27 17:47:01.581038 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:01.580975    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z\" (UID: \"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z"
Jul 27 17:47:01.581038 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:01.581035    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z\" (UID: \"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z"
Jul 27 17:47:01.581406 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:01.581360    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z\" (UID: \"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z"
Jul 27 17:47:01.581406 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:01.581372    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z\" (UID: \"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z"
Jul 27 17:47:01.581611 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:01.581588    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z\" (UID: \"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z"
Jul 27 17:47:01.583049 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:01.583023    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z\" (UID: \"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z"
Jul 27 17:47:01.583365 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:01.583345    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z\" (UID: \"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z"
Jul 27 17:47:01.589006 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:01.588984    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw7wp\" (UniqueName: \"kubernetes.io/projected/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-kube-api-access-mw7wp\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z\" (UID: \"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z"
Jul 27 17:47:01.698191 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:01.698152    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z"
Jul 27 17:47:01.826935 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:01.826913    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z"]
Jul 27 17:47:01.829629 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:47:01.829607    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda77a90c1_b9f7_4b3b_8b5a_f5d4fd67ca93.slice/crio-2f2aacef51caab88a8544a781c4427cc5673087088db3f3273988f0c781f852f WatchSource:0}: Error finding container 2f2aacef51caab88a8544a781c4427cc5673087088db3f3273988f0c781f852f: Status 404 returned error can't find the container with id 2f2aacef51caab88a8544a781c4427cc5673087088db3f3273988f0c781f852f
Jul 27 17:47:02.538530 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:02.538490    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z" event={"ID":"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93","Type":"ContainerStarted","Data":"9b752e8e28497615b9fd7cab03bc03e4e204537b38cb09feda6a8349131faeab"}
Jul 27 17:47:02.538939 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:02.538546    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z" event={"ID":"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93","Type":"ContainerStarted","Data":"2f2aacef51caab88a8544a781c4427cc5673087088db3f3273988f0c781f852f"}
Jul 27 17:47:02.558875 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:02.558830    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z" podStartSLOduration=1.558816165 podStartE2EDuration="1.558816165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:47:02.557204604 +0000 UTC m=+1115.044015917" watchObservedRunningTime="2026-07-27 17:47:02.558816165 +0000 UTC m=+1115.045627504"
Jul 27 17:47:02.698743 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:02.698706    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z"
Jul 27 17:47:03.773813 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:03.773781    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z"]
Jul 27 17:47:04.626348 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:47:04.626311    2557 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-49b5edd9/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs: secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
Jul 27 17:47:04.626533 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:47:04.626396    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-tls-certs podName:a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93 nodeName:}" failed. No retries permitted until 2026-07-27 17:47:05.126378882 +0000 UTC m=+1117.613190147 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-tls-certs") pod "gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z" (UID: "a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93") : secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
Jul 27 17:47:05.131013 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:47:05.130980    2557 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-49b5edd9/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs: secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
Jul 27 17:47:05.131419 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:47:05.131054    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-tls-certs podName:a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93 nodeName:}" failed. No retries permitted until 2026-07-27 17:47:06.131039338 +0000 UTC m=+1118.617850603 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-tls-certs") pod "gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z" (UID: "a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93") : secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
Jul 27 17:47:05.561271 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:47:05.561233    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:47:05.561271 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:47:05.561271 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 17:47:05.561271 ip-10-0-131-114 kubenswrapper[2557]:  > containerID="9b752e8e28497615b9fd7cab03bc03e4e204537b38cb09feda6a8349131faeab" cmd=["/bin/sleep","15"]
Jul 27 17:47:05.561604 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:05.561282    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z" message="" err=<
Jul 27 17:47:05.561604 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:47:05.561604 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 17:47:05.561604 ip-10-0-131-114 kubenswrapper[2557]:  >
Jul 27 17:47:05.561604 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:47:05.561297    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:47:05.561604 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:47:05.561604 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 17:47:05.561604 ip-10-0-131-114 kubenswrapper[2557]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z" podUID="a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93" containerName="main" containerID="cri-o://9b752e8e28497615b9fd7cab03bc03e4e204537b38cb09feda6a8349131faeab"
Jul 27 17:47:05.561604 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:05.561351    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z" podUID="a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93" containerName="main" containerID="cri-o://9b752e8e28497615b9fd7cab03bc03e4e204537b38cb09feda6a8349131faeab" gracePeriod=60
Jul 27 17:47:05.846335 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:05.846310    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z"
Jul 27 17:47:05.939195 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:05.939161    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-dshm\") pod \"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93\" (UID: \"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93\") "
Jul 27 17:47:05.939195 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:05.939201    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-model-cache\") pod \"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93\" (UID: \"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93\") "
Jul 27 17:47:05.939462 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:05.939255    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-tls-certs\") pod \"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93\" (UID: \"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93\") "
Jul 27 17:47:05.939462 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:05.939324    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mw7wp\" (UniqueName: \"kubernetes.io/projected/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-kube-api-access-mw7wp\") pod \"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93\" (UID: \"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93\") "
Jul 27 17:47:05.939462 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:05.939414    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-tmp-dir\") pod \"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93\" (UID: \"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93\") "
Jul 27 17:47:05.939462 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:05.939453    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-home\") pod \"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93\" (UID: \"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93\") "
Jul 27 17:47:05.939680 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:05.939500    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-model-cache" (OuterVolumeSpecName: "model-cache") pod "a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93" (UID: "a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:47:05.939680 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:05.939644    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93" (UID: "a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:47:05.939815 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:05.939725    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-home" (OuterVolumeSpecName: "home") pod "a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93" (UID: "a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:47:05.939815 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:05.939737    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-model-cache\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:47:05.939815 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:05.939804    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-tmp-dir\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:47:05.941580 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:05.941554    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-dshm" (OuterVolumeSpecName: "dshm") pod "a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93" (UID: "a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:47:05.941685 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:05.941598    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-kube-api-access-mw7wp" (OuterVolumeSpecName: "kube-api-access-mw7wp") pod "a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93" (UID: "a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93"). InnerVolumeSpecName "kube-api-access-mw7wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:47:05.941685 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:05.941599    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93" (UID: "a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:47:06.040442 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:06.040407    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-dshm\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:47:06.040442 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:06.040434    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-tls-certs\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:47:06.040442 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:06.040447    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw7wp\" (UniqueName: \"kubernetes.io/projected/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-kube-api-access-mw7wp\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:47:06.040665 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:06.040456    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93-home\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:47:06.558539 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:06.558504    2557 generic.go:356] "Generic (PLEG): container finished" podID="a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93" containerID="9b752e8e28497615b9fd7cab03bc03e4e204537b38cb09feda6a8349131faeab" exitCode=0
Jul 27 17:47:06.559063 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:06.558546    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z"
Jul 27 17:47:06.559063 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:06.558586    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z" event={"ID":"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93","Type":"ContainerDied","Data":"9b752e8e28497615b9fd7cab03bc03e4e204537b38cb09feda6a8349131faeab"}
Jul 27 17:47:06.559063 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:06.558636    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z" event={"ID":"a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93","Type":"ContainerDied","Data":"2f2aacef51caab88a8544a781c4427cc5673087088db3f3273988f0c781f852f"}
Jul 27 17:47:06.559063 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:06.558659    2557 scope.go:117] "RemoveContainer" containerID="9b752e8e28497615b9fd7cab03bc03e4e204537b38cb09feda6a8349131faeab"
Jul 27 17:47:06.567864 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:06.567663    2557 scope.go:117] "RemoveContainer" containerID="9b752e8e28497615b9fd7cab03bc03e4e204537b38cb09feda6a8349131faeab"
Jul 27 17:47:06.567994 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:47:06.567970    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b752e8e28497615b9fd7cab03bc03e4e204537b38cb09feda6a8349131faeab\": container with ID starting with 9b752e8e28497615b9fd7cab03bc03e4e204537b38cb09feda6a8349131faeab not found: ID does not exist" containerID="9b752e8e28497615b9fd7cab03bc03e4e204537b38cb09feda6a8349131faeab"
Jul 27 17:47:06.568041 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:06.567995    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b752e8e28497615b9fd7cab03bc03e4e204537b38cb09feda6a8349131faeab"} err="failed to get container status \"9b752e8e28497615b9fd7cab03bc03e4e204537b38cb09feda6a8349131faeab\": rpc error: code = NotFound desc = could not find container \"9b752e8e28497615b9fd7cab03bc03e4e204537b38cb09feda6a8349131faeab\": container with ID starting with 9b752e8e28497615b9fd7cab03bc03e4e204537b38cb09feda6a8349131faeab not found: ID does not exist"
Jul 27 17:47:06.573270 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:06.573245    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z"]
Jul 27 17:47:06.577669 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:06.577646    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-6249z"]
Jul 27 17:47:08.125250 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:08.125216    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93" path="/var/lib/kubelet/pods/a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93/volumes"
Jul 27 17:47:08.595158 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:08.595126    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"]
Jul 27 17:47:08.595382 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:08.595358    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c" podUID="9e2302da-91ff-4ac3-8095-90389a25c8c0" containerName="istio-proxy" containerID="cri-o://c63337f5805f7c3c258ba9879a52852c51fffc472959169fab708b20b4f1f088" gracePeriod=30
Jul 27 17:47:11.549051 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.549027    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:47:11.588253 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.588220    2557 generic.go:356] "Generic (PLEG): container finished" podID="9e2302da-91ff-4ac3-8095-90389a25c8c0" containerID="c63337f5805f7c3c258ba9879a52852c51fffc472959169fab708b20b4f1f088" exitCode=0
Jul 27 17:47:11.588403 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.588264    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"
Jul 27 17:47:11.588403 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.588262    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c" event={"ID":"9e2302da-91ff-4ac3-8095-90389a25c8c0","Type":"ContainerDied","Data":"c63337f5805f7c3c258ba9879a52852c51fffc472959169fab708b20b4f1f088"}
Jul 27 17:47:11.588488 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.588407    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c" event={"ID":"9e2302da-91ff-4ac3-8095-90389a25c8c0","Type":"ContainerDied","Data":"e04b1e81e63aeff36a3e0bb8374e88a08cf3b18b922892d9abc4fe66e24a1abb"}
Jul 27 17:47:11.588488 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.588437    2557 scope.go:117] "RemoveContainer" containerID="c63337f5805f7c3c258ba9879a52852c51fffc472959169fab708b20b4f1f088"
Jul 27 17:47:11.597234 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.597215    2557 scope.go:117] "RemoveContainer" containerID="c63337f5805f7c3c258ba9879a52852c51fffc472959169fab708b20b4f1f088"
Jul 27 17:47:11.597520 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:47:11.597501    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c63337f5805f7c3c258ba9879a52852c51fffc472959169fab708b20b4f1f088\": container with ID starting with c63337f5805f7c3c258ba9879a52852c51fffc472959169fab708b20b4f1f088 not found: ID does not exist" containerID="c63337f5805f7c3c258ba9879a52852c51fffc472959169fab708b20b4f1f088"
Jul 27 17:47:11.597587 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.597528    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c63337f5805f7c3c258ba9879a52852c51fffc472959169fab708b20b4f1f088"} err="failed to get container status \"c63337f5805f7c3c258ba9879a52852c51fffc472959169fab708b20b4f1f088\": rpc error: code = NotFound desc = could not find container \"c63337f5805f7c3c258ba9879a52852c51fffc472959169fab708b20b4f1f088\": container with ID starting with c63337f5805f7c3c258ba9879a52852c51fffc472959169fab708b20b4f1f088 not found: ID does not exist"
Jul 27 17:47:11.691326 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.691295    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e2302da-91ff-4ac3-8095-90389a25c8c0-istio-podinfo\") pod \"9e2302da-91ff-4ac3-8095-90389a25c8c0\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") "
Jul 27 17:47:11.691492 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.691362    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e2302da-91ff-4ac3-8095-90389a25c8c0-istiod-ca-cert\") pod \"9e2302da-91ff-4ac3-8095-90389a25c8c0\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") "
Jul 27 17:47:11.691492 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.691389    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e2302da-91ff-4ac3-8095-90389a25c8c0-istio-token\") pod \"9e2302da-91ff-4ac3-8095-90389a25c8c0\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") "
Jul 27 17:47:11.691492 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.691417    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e2302da-91ff-4ac3-8095-90389a25c8c0-istio-data\") pod \"9e2302da-91ff-4ac3-8095-90389a25c8c0\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") "
Jul 27 17:47:11.691492 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.691463    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e2302da-91ff-4ac3-8095-90389a25c8c0-workload-certs\") pod \"9e2302da-91ff-4ac3-8095-90389a25c8c0\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") "
Jul 27 17:47:11.691690 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.691528    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e2302da-91ff-4ac3-8095-90389a25c8c0-istio-envoy\") pod \"9e2302da-91ff-4ac3-8095-90389a25c8c0\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") "
Jul 27 17:47:11.691690 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.691550    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2302da-91ff-4ac3-8095-90389a25c8c0-workload-socket\") pod \"9e2302da-91ff-4ac3-8095-90389a25c8c0\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") "
Jul 27 17:47:11.691690 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.691575    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2302da-91ff-4ac3-8095-90389a25c8c0-credential-socket\") pod \"9e2302da-91ff-4ac3-8095-90389a25c8c0\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") "
Jul 27 17:47:11.691690 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.691608    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8j95\" (UniqueName: \"kubernetes.io/projected/9e2302da-91ff-4ac3-8095-90389a25c8c0-kube-api-access-g8j95\") pod \"9e2302da-91ff-4ac3-8095-90389a25c8c0\" (UID: \"9e2302da-91ff-4ac3-8095-90389a25c8c0\") "
Jul 27 17:47:11.692155 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.691722    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2302da-91ff-4ac3-8095-90389a25c8c0-istio-data" (OuterVolumeSpecName: "istio-data") pod "9e2302da-91ff-4ac3-8095-90389a25c8c0" (UID: "9e2302da-91ff-4ac3-8095-90389a25c8c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:47:11.692155 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.691844    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2302da-91ff-4ac3-8095-90389a25c8c0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9e2302da-91ff-4ac3-8095-90389a25c8c0" (UID: "9e2302da-91ff-4ac3-8095-90389a25c8c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:47:11.692155 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.691859    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2302da-91ff-4ac3-8095-90389a25c8c0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9e2302da-91ff-4ac3-8095-90389a25c8c0" (UID: "9e2302da-91ff-4ac3-8095-90389a25c8c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:47:11.692155 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.691934    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2302da-91ff-4ac3-8095-90389a25c8c0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9e2302da-91ff-4ac3-8095-90389a25c8c0" (UID: "9e2302da-91ff-4ac3-8095-90389a25c8c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:47:11.692155 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.691977    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e2302da-91ff-4ac3-8095-90389a25c8c0-istio-data\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:47:11.692155 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.691997    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e2302da-91ff-4ac3-8095-90389a25c8c0-workload-certs\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:47:11.692155 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.692012    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2302da-91ff-4ac3-8095-90389a25c8c0-workload-socket\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:47:11.692508 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.692233    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e2302da-91ff-4ac3-8095-90389a25c8c0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9e2302da-91ff-4ac3-8095-90389a25c8c0" (UID: "9e2302da-91ff-4ac3-8095-90389a25c8c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:47:11.693744 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.693712    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e2302da-91ff-4ac3-8095-90389a25c8c0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9e2302da-91ff-4ac3-8095-90389a25c8c0" (UID: "9e2302da-91ff-4ac3-8095-90389a25c8c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:47:11.693744 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.693731    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e2302da-91ff-4ac3-8095-90389a25c8c0-istio-token" (OuterVolumeSpecName: "istio-token") pod "9e2302da-91ff-4ac3-8095-90389a25c8c0" (UID: "9e2302da-91ff-4ac3-8095-90389a25c8c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:47:11.693927 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.693793    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e2302da-91ff-4ac3-8095-90389a25c8c0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9e2302da-91ff-4ac3-8095-90389a25c8c0" (UID: "9e2302da-91ff-4ac3-8095-90389a25c8c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 17:47:11.693927 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.693875    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e2302da-91ff-4ac3-8095-90389a25c8c0-kube-api-access-g8j95" (OuterVolumeSpecName: "kube-api-access-g8j95") pod "9e2302da-91ff-4ac3-8095-90389a25c8c0" (UID: "9e2302da-91ff-4ac3-8095-90389a25c8c0"). InnerVolumeSpecName "kube-api-access-g8j95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:47:11.793315 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.793283    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e2302da-91ff-4ac3-8095-90389a25c8c0-istio-podinfo\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:47:11.793315 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.793308    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e2302da-91ff-4ac3-8095-90389a25c8c0-istiod-ca-cert\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:47:11.793315 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.793316    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e2302da-91ff-4ac3-8095-90389a25c8c0-istio-token\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:47:11.793315 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.793324    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e2302da-91ff-4ac3-8095-90389a25c8c0-istio-envoy\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:47:11.793570 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.793332    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e2302da-91ff-4ac3-8095-90389a25c8c0-credential-socket\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:47:11.793570 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.793341    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8j95\" (UniqueName: \"kubernetes.io/projected/9e2302da-91ff-4ac3-8095-90389a25c8c0-kube-api-access-g8j95\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:47:11.914838 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.914811    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"]
Jul 27 17:47:11.920199 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:11.920175    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-7px2c"]
Jul 27 17:47:12.119882 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:12.119855    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e2302da-91ff-4ac3-8095-90389a25c8c0" path="/var/lib/kubelet/pods/9e2302da-91ff-4ac3-8095-90389a25c8c0/volumes"
Jul 27 17:47:23.499881 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.499845    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj"]
Jul 27 17:47:23.500369 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.500354    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e2302da-91ff-4ac3-8095-90389a25c8c0" containerName="istio-proxy"
Jul 27 17:47:23.500411 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.500372    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e2302da-91ff-4ac3-8095-90389a25c8c0" containerName="istio-proxy"
Jul 27 17:47:23.500411 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.500387    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93" containerName="main"
Jul 27 17:47:23.500411 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.500396    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93" containerName="main"
Jul 27 17:47:23.500561 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.500497    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a77a90c1-b9f7-4b3b-8b5a-f5d4fd67ca93" containerName="main"
Jul 27 17:47:23.500561 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.500509    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e2302da-91ff-4ac3-8095-90389a25c8c0" containerName="istio-proxy"
Jul 27 17:47:23.511036 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.511014    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj"
Jul 27 17:47:23.513954 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.513927    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Jul 27 17:47:23.514353 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.514334    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Jul 27 17:47:23.515068 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.514681    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-6qm87\""
Jul 27 17:47:23.515352 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.515330    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Jul 27 17:47:23.517466 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.517433    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj"]
Jul 27 17:47:23.691934 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.691898    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj\" (UID: \"f02c3980-bb51-4c17-b38f-c6e2e1994dbf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj"
Jul 27 17:47:23.691934 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.691935    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdncx\" (UniqueName: \"kubernetes.io/projected/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-kube-api-access-hdncx\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj\" (UID: \"f02c3980-bb51-4c17-b38f-c6e2e1994dbf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj"
Jul 27 17:47:23.692195 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.691962    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj\" (UID: \"f02c3980-bb51-4c17-b38f-c6e2e1994dbf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj"
Jul 27 17:47:23.692195 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.692031    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj\" (UID: \"f02c3980-bb51-4c17-b38f-c6e2e1994dbf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj"
Jul 27 17:47:23.692195 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.692109    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj\" (UID: \"f02c3980-bb51-4c17-b38f-c6e2e1994dbf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj"
Jul 27 17:47:23.692195 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.692145    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj\" (UID: \"f02c3980-bb51-4c17-b38f-c6e2e1994dbf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj"
Jul 27 17:47:23.793730 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.793585    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj\" (UID: \"f02c3980-bb51-4c17-b38f-c6e2e1994dbf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj"
Jul 27 17:47:23.793730 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.793631    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj\" (UID: \"f02c3980-bb51-4c17-b38f-c6e2e1994dbf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj"
Jul 27 17:47:23.794054 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.793786    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj\" (UID: \"f02c3980-bb51-4c17-b38f-c6e2e1994dbf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj"
Jul 27 17:47:23.794054 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.793830    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdncx\" (UniqueName: \"kubernetes.io/projected/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-kube-api-access-hdncx\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj\" (UID: \"f02c3980-bb51-4c17-b38f-c6e2e1994dbf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj"
Jul 27 17:47:23.794054 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.793884    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj\" (UID: \"f02c3980-bb51-4c17-b38f-c6e2e1994dbf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj"
Jul 27 17:47:23.794213 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.794149    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj\" (UID: \"f02c3980-bb51-4c17-b38f-c6e2e1994dbf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj"
Jul 27 17:47:23.794372 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.794351    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj\" (UID: \"f02c3980-bb51-4c17-b38f-c6e2e1994dbf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj"
Jul 27 17:47:23.794438 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.794412    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj\" (UID: \"f02c3980-bb51-4c17-b38f-c6e2e1994dbf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj"
Jul 27 17:47:23.794508 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.794481    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj\" (UID: \"f02c3980-bb51-4c17-b38f-c6e2e1994dbf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj"
Jul 27 17:47:23.796151 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.796123    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj\" (UID: \"f02c3980-bb51-4c17-b38f-c6e2e1994dbf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj"
Jul 27 17:47:23.796386 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.796369    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj\" (UID: \"f02c3980-bb51-4c17-b38f-c6e2e1994dbf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj"
Jul 27 17:47:23.801655 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.801631    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdncx\" (UniqueName: \"kubernetes.io/projected/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-kube-api-access-hdncx\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj\" (UID: \"f02c3980-bb51-4c17-b38f-c6e2e1994dbf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj"
Jul 27 17:47:23.824474 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.824448    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj"
Jul 27 17:47:23.953951 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:23.953919    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj"]
Jul 27 17:47:23.956487 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:47:23.956456    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf02c3980_bb51_4c17_b38f_c6e2e1994dbf.slice/crio-551d2c05ee6404b0f9b3f49248c4777950a74dcd3da59775c52301e63dc0d9a8 WatchSource:0}: Error finding container 551d2c05ee6404b0f9b3f49248c4777950a74dcd3da59775c52301e63dc0d9a8: Status 404 returned error can't find the container with id 551d2c05ee6404b0f9b3f49248c4777950a74dcd3da59775c52301e63dc0d9a8
Jul 27 17:47:24.650442 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:24.650410    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj" event={"ID":"f02c3980-bb51-4c17-b38f-c6e2e1994dbf","Type":"ContainerStarted","Data":"10fcca123403cb7e2f340f09f88a108e6439f076bdccc56f1b79d527b974d178"}
Jul 27 17:47:24.650867 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:24.650458    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj" event={"ID":"f02c3980-bb51-4c17-b38f-c6e2e1994dbf","Type":"ContainerStarted","Data":"551d2c05ee6404b0f9b3f49248c4777950a74dcd3da59775c52301e63dc0d9a8"}
Jul 27 17:47:24.669635 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:24.669588    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj" podStartSLOduration=1.669574965 podStartE2EDuration="1.669574965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:47:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:47:24.668622289 +0000 UTC m=+1137.155433577" watchObservedRunningTime="2026-07-27 17:47:24.669574965 +0000 UTC m=+1137.156386252"
Jul 27 17:47:24.825091 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:24.825064    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj"
Jul 27 17:47:25.893784 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:25.893481    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj"]
Jul 27 17:47:26.720816 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:47:26.720781    2557 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-94799d44/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs: secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Jul 27 17:47:26.720998 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:47:26.720855    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-tls-certs podName:f02c3980-bb51-4c17-b38f-c6e2e1994dbf nodeName:}" failed. No retries permitted until 2026-07-27 17:47:27.22083836 +0000 UTC m=+1139.707649625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-tls-certs") pod "gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj" (UID: "f02c3980-bb51-4c17-b38f-c6e2e1994dbf") : secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Jul 27 17:47:27.224576 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:47:27.224545    2557 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-94799d44/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs: secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Jul 27 17:47:27.224977 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:47:27.224605    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-tls-certs podName:f02c3980-bb51-4c17-b38f-c6e2e1994dbf nodeName:}" failed. No retries permitted until 2026-07-27 17:47:28.224592034 +0000 UTC m=+1140.711403299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-tls-certs") pod "gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj" (UID: "f02c3980-bb51-4c17-b38f-c6e2e1994dbf") : secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Jul 27 17:47:27.672669 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:47:27.672624    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:47:27.672669 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:47:27.672669 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 17:47:27.672669 ip-10-0-131-114 kubenswrapper[2557]:  > containerID="10fcca123403cb7e2f340f09f88a108e6439f076bdccc56f1b79d527b974d178" cmd=["/bin/sleep","15"]
Jul 27 17:47:27.672952 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:27.672682    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj" message="" err=<
Jul 27 17:47:27.672952 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:47:27.672952 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 17:47:27.672952 ip-10-0-131-114 kubenswrapper[2557]:  >
Jul 27 17:47:27.672952 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:47:27.672694    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:47:27.672952 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:47:27.672952 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 17:47:27.672952 ip-10-0-131-114 kubenswrapper[2557]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj" podUID="f02c3980-bb51-4c17-b38f-c6e2e1994dbf" containerName="main" containerID="cri-o://10fcca123403cb7e2f340f09f88a108e6439f076bdccc56f1b79d527b974d178"
Jul 27 17:47:27.672952 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:27.672727    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj" podUID="f02c3980-bb51-4c17-b38f-c6e2e1994dbf" containerName="main" containerID="cri-o://10fcca123403cb7e2f340f09f88a108e6439f076bdccc56f1b79d527b974d178" gracePeriod=60
Jul 27 17:47:27.920177 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:27.920152    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj"
Jul 27 17:47:27.930958 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:27.930919    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-tmp-dir\") pod \"f02c3980-bb51-4c17-b38f-c6e2e1994dbf\" (UID: \"f02c3980-bb51-4c17-b38f-c6e2e1994dbf\") "
Jul 27 17:47:27.931020 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:27.930968    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdncx\" (UniqueName: \"kubernetes.io/projected/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-kube-api-access-hdncx\") pod \"f02c3980-bb51-4c17-b38f-c6e2e1994dbf\" (UID: \"f02c3980-bb51-4c17-b38f-c6e2e1994dbf\") "
Jul 27 17:47:27.931020 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:27.930991    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-home\") pod \"f02c3980-bb51-4c17-b38f-c6e2e1994dbf\" (UID: \"f02c3980-bb51-4c17-b38f-c6e2e1994dbf\") "
Jul 27 17:47:27.931119 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:27.931023    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-model-cache\") pod \"f02c3980-bb51-4c17-b38f-c6e2e1994dbf\" (UID: \"f02c3980-bb51-4c17-b38f-c6e2e1994dbf\") "
Jul 27 17:47:27.931119 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:27.931054    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-dshm\") pod \"f02c3980-bb51-4c17-b38f-c6e2e1994dbf\" (UID: \"f02c3980-bb51-4c17-b38f-c6e2e1994dbf\") "
Jul 27 17:47:27.931119 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:27.931082    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-tls-certs\") pod \"f02c3980-bb51-4c17-b38f-c6e2e1994dbf\" (UID: \"f02c3980-bb51-4c17-b38f-c6e2e1994dbf\") "
Jul 27 17:47:27.931265 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:27.931176    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f02c3980-bb51-4c17-b38f-c6e2e1994dbf" (UID: "f02c3980-bb51-4c17-b38f-c6e2e1994dbf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:47:27.931325 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:27.931296    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-model-cache" (OuterVolumeSpecName: "model-cache") pod "f02c3980-bb51-4c17-b38f-c6e2e1994dbf" (UID: "f02c3980-bb51-4c17-b38f-c6e2e1994dbf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:47:27.931325 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:27.931312    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-home" (OuterVolumeSpecName: "home") pod "f02c3980-bb51-4c17-b38f-c6e2e1994dbf" (UID: "f02c3980-bb51-4c17-b38f-c6e2e1994dbf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:47:27.931427 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:27.931394    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-tmp-dir\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:47:27.931427 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:27.931411    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-home\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:47:27.931496 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:27.931427    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-model-cache\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:47:27.933384 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:27.933358    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-dshm" (OuterVolumeSpecName: "dshm") pod "f02c3980-bb51-4c17-b38f-c6e2e1994dbf" (UID: "f02c3980-bb51-4c17-b38f-c6e2e1994dbf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:47:27.933499 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:27.933452    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-kube-api-access-hdncx" (OuterVolumeSpecName: "kube-api-access-hdncx") pod "f02c3980-bb51-4c17-b38f-c6e2e1994dbf" (UID: "f02c3980-bb51-4c17-b38f-c6e2e1994dbf"). InnerVolumeSpecName "kube-api-access-hdncx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:47:27.933604 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:27.933584    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f02c3980-bb51-4c17-b38f-c6e2e1994dbf" (UID: "f02c3980-bb51-4c17-b38f-c6e2e1994dbf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:47:28.032566 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:28.032533    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-dshm\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:47:28.032566 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:28.032561    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-tls-certs\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:47:28.032566 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:28.032573    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdncx\" (UniqueName: \"kubernetes.io/projected/f02c3980-bb51-4c17-b38f-c6e2e1994dbf-kube-api-access-hdncx\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:47:28.669646 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:28.669613    2557 generic.go:356] "Generic (PLEG): container finished" podID="f02c3980-bb51-4c17-b38f-c6e2e1994dbf" containerID="10fcca123403cb7e2f340f09f88a108e6439f076bdccc56f1b79d527b974d178" exitCode=0
Jul 27 17:47:28.670054 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:28.669665    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj"
Jul 27 17:47:28.670054 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:28.669700    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj" event={"ID":"f02c3980-bb51-4c17-b38f-c6e2e1994dbf","Type":"ContainerDied","Data":"10fcca123403cb7e2f340f09f88a108e6439f076bdccc56f1b79d527b974d178"}
Jul 27 17:47:28.670054 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:28.669770    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj" event={"ID":"f02c3980-bb51-4c17-b38f-c6e2e1994dbf","Type":"ContainerDied","Data":"551d2c05ee6404b0f9b3f49248c4777950a74dcd3da59775c52301e63dc0d9a8"}
Jul 27 17:47:28.670054 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:28.669798    2557 scope.go:117] "RemoveContainer" containerID="10fcca123403cb7e2f340f09f88a108e6439f076bdccc56f1b79d527b974d178"
Jul 27 17:47:28.678098 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:28.677961    2557 scope.go:117] "RemoveContainer" containerID="10fcca123403cb7e2f340f09f88a108e6439f076bdccc56f1b79d527b974d178"
Jul 27 17:47:28.678214 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:47:28.678197    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10fcca123403cb7e2f340f09f88a108e6439f076bdccc56f1b79d527b974d178\": container with ID starting with 10fcca123403cb7e2f340f09f88a108e6439f076bdccc56f1b79d527b974d178 not found: ID does not exist" containerID="10fcca123403cb7e2f340f09f88a108e6439f076bdccc56f1b79d527b974d178"
Jul 27 17:47:28.678269 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:28.678225    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10fcca123403cb7e2f340f09f88a108e6439f076bdccc56f1b79d527b974d178"} err="failed to get container status \"10fcca123403cb7e2f340f09f88a108e6439f076bdccc56f1b79d527b974d178\": rpc error: code = NotFound desc = could not find container \"10fcca123403cb7e2f340f09f88a108e6439f076bdccc56f1b79d527b974d178\": container with ID starting with 10fcca123403cb7e2f340f09f88a108e6439f076bdccc56f1b79d527b974d178 not found: ID does not exist"
Jul 27 17:47:28.684703 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:28.684683    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj"]
Jul 27 17:47:28.688588 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:28.688568    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvmswj"]
Jul 27 17:47:30.118769 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:30.118726    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f02c3980-bb51-4c17-b38f-c6e2e1994dbf" path="/var/lib/kubelet/pods/f02c3980-bb51-4c17-b38f-c6e2e1994dbf/volumes"
Jul 27 17:47:33.408012 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.407982    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"]
Jul 27 17:47:33.408389 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.408333    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f02c3980-bb51-4c17-b38f-c6e2e1994dbf" containerName="main"
Jul 27 17:47:33.408389 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.408344    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f02c3980-bb51-4c17-b38f-c6e2e1994dbf" containerName="main"
Jul 27 17:47:33.408469 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.408399    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f02c3980-bb51-4c17-b38f-c6e2e1994dbf" containerName="main"
Jul 27 17:47:33.413413 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.413393    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"
Jul 27 17:47:33.416157 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.416137    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-srq75\""
Jul 27 17:47:33.416786 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.416743    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Jul 27 17:47:33.416893 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.416871    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Jul 27 17:47:33.416957 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.416920    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Jul 27 17:47:33.417009 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.416983    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 27 17:47:33.427288 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.427265    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"]
Jul 27 17:47:33.479448 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.479373    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6865ee1-94c1-4fca-b216-cae682b7f4d8-model-cache\") pod \"auth-enabled-test-kserve-85d957ff8-n49nr\" (UID: \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"
Jul 27 17:47:33.479448 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.479408    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6865ee1-94c1-4fca-b216-cae682b7f4d8-dshm\") pod \"auth-enabled-test-kserve-85d957ff8-n49nr\" (UID: \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"
Jul 27 17:47:33.479650 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.479505    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6865ee1-94c1-4fca-b216-cae682b7f4d8-kserve-provision-location\") pod \"auth-enabled-test-kserve-85d957ff8-n49nr\" (UID: \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"
Jul 27 17:47:33.479650 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.479556    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6865ee1-94c1-4fca-b216-cae682b7f4d8-tls-certs\") pod \"auth-enabled-test-kserve-85d957ff8-n49nr\" (UID: \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"
Jul 27 17:47:33.479650 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.479586    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6865ee1-94c1-4fca-b216-cae682b7f4d8-home\") pod \"auth-enabled-test-kserve-85d957ff8-n49nr\" (UID: \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"
Jul 27 17:47:33.479650 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.479609    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6865ee1-94c1-4fca-b216-cae682b7f4d8-tmp-dir\") pod \"auth-enabled-test-kserve-85d957ff8-n49nr\" (UID: \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"
Jul 27 17:47:33.479650 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.479633    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdskb\" (UniqueName: \"kubernetes.io/projected/f6865ee1-94c1-4fca-b216-cae682b7f4d8-kube-api-access-qdskb\") pod \"auth-enabled-test-kserve-85d957ff8-n49nr\" (UID: \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"
Jul 27 17:47:33.580827 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.580790    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6865ee1-94c1-4fca-b216-cae682b7f4d8-model-cache\") pod \"auth-enabled-test-kserve-85d957ff8-n49nr\" (UID: \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"
Jul 27 17:47:33.580827 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.580831    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6865ee1-94c1-4fca-b216-cae682b7f4d8-dshm\") pod \"auth-enabled-test-kserve-85d957ff8-n49nr\" (UID: \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"
Jul 27 17:47:33.581257 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.580881    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6865ee1-94c1-4fca-b216-cae682b7f4d8-kserve-provision-location\") pod \"auth-enabled-test-kserve-85d957ff8-n49nr\" (UID: \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"
Jul 27 17:47:33.581257 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.580906    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6865ee1-94c1-4fca-b216-cae682b7f4d8-tls-certs\") pod \"auth-enabled-test-kserve-85d957ff8-n49nr\" (UID: \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"
Jul 27 17:47:33.581257 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.580925    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6865ee1-94c1-4fca-b216-cae682b7f4d8-home\") pod \"auth-enabled-test-kserve-85d957ff8-n49nr\" (UID: \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"
Jul 27 17:47:33.581257 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.581117    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6865ee1-94c1-4fca-b216-cae682b7f4d8-tmp-dir\") pod \"auth-enabled-test-kserve-85d957ff8-n49nr\" (UID: \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"
Jul 27 17:47:33.581257 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.581156    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdskb\" (UniqueName: \"kubernetes.io/projected/f6865ee1-94c1-4fca-b216-cae682b7f4d8-kube-api-access-qdskb\") pod \"auth-enabled-test-kserve-85d957ff8-n49nr\" (UID: \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"
Jul 27 17:47:33.581257 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.581204    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6865ee1-94c1-4fca-b216-cae682b7f4d8-model-cache\") pod \"auth-enabled-test-kserve-85d957ff8-n49nr\" (UID: \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"
Jul 27 17:47:33.581257 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.581243    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6865ee1-94c1-4fca-b216-cae682b7f4d8-home\") pod \"auth-enabled-test-kserve-85d957ff8-n49nr\" (UID: \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"
Jul 27 17:47:33.581525 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.581308    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6865ee1-94c1-4fca-b216-cae682b7f4d8-kserve-provision-location\") pod \"auth-enabled-test-kserve-85d957ff8-n49nr\" (UID: \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"
Jul 27 17:47:33.581579 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.581557    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6865ee1-94c1-4fca-b216-cae682b7f4d8-tmp-dir\") pod \"auth-enabled-test-kserve-85d957ff8-n49nr\" (UID: \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"
Jul 27 17:47:33.583319 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.583295    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6865ee1-94c1-4fca-b216-cae682b7f4d8-dshm\") pod \"auth-enabled-test-kserve-85d957ff8-n49nr\" (UID: \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"
Jul 27 17:47:33.583475 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.583457    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6865ee1-94c1-4fca-b216-cae682b7f4d8-tls-certs\") pod \"auth-enabled-test-kserve-85d957ff8-n49nr\" (UID: \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"
Jul 27 17:47:33.588673 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.588652    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdskb\" (UniqueName: \"kubernetes.io/projected/f6865ee1-94c1-4fca-b216-cae682b7f4d8-kube-api-access-qdskb\") pod \"auth-enabled-test-kserve-85d957ff8-n49nr\" (UID: \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"
Jul 27 17:47:33.723976 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.723947    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"
Jul 27 17:47:33.855557 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:33.855531    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"]
Jul 27 17:47:33.857820 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:47:33.857793    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6865ee1_94c1_4fca_b216_cae682b7f4d8.slice/crio-e1ad9ae010dc8165e08629e43d0623116c862b01432a71c906faa743486331d6 WatchSource:0}: Error finding container e1ad9ae010dc8165e08629e43d0623116c862b01432a71c906faa743486331d6: Status 404 returned error can't find the container with id e1ad9ae010dc8165e08629e43d0623116c862b01432a71c906faa743486331d6
Jul 27 17:47:34.705850 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:34.705698    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr" event={"ID":"f6865ee1-94c1-4fca-b216-cae682b7f4d8","Type":"ContainerStarted","Data":"e1ad9ae010dc8165e08629e43d0623116c862b01432a71c906faa743486331d6"}
Jul 27 17:47:37.725188 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:37.725146    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr" event={"ID":"f6865ee1-94c1-4fca-b216-cae682b7f4d8","Type":"ContainerStarted","Data":"7657b2b86cf4265be303c809cf08b8da628c25c0501d37314a64f246c68d46ab"}
Jul 27 17:47:43.755872 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:43.755792    2557 generic.go:356] "Generic (PLEG): container finished" podID="f6865ee1-94c1-4fca-b216-cae682b7f4d8" containerID="7657b2b86cf4265be303c809cf08b8da628c25c0501d37314a64f246c68d46ab" exitCode=0
Jul 27 17:47:43.755872 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:47:43.755836    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr" event={"ID":"f6865ee1-94c1-4fca-b216-cae682b7f4d8","Type":"ContainerDied","Data":"7657b2b86cf4265be303c809cf08b8da628c25c0501d37314a64f246c68d46ab"}
Jul 27 17:48:03.838575 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:03.838410    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 27 17:48:03.846659 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:03.846629    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:48:03.849458 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:03.849047    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Jul 27 17:48:03.849458 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:03.849179    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-f8v96\""
Jul 27 17:48:03.849714 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:03.849488    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-bss89\""
Jul 27 17:48:03.850033 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:03.849997    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Jul 27 17:48:03.855060 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:03.851955    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 27 17:48:03.855060 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:03.853169    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Jul 27 17:48:03.919159 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:03.919130    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 27 17:48:03.975975 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:03.975932    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6289a2da-842a-484c-ad2d-c5ef9571790d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6289a2da-842a-484c-ad2d-c5ef9571790d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:48:03.976153 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:03.975996    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6289a2da-842a-484c-ad2d-c5ef9571790d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6289a2da-842a-484c-ad2d-c5ef9571790d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:48:03.976153 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:03.976074    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ddmb\" (UniqueName: \"kubernetes.io/projected/6289a2da-842a-484c-ad2d-c5ef9571790d-kube-api-access-4ddmb\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6289a2da-842a-484c-ad2d-c5ef9571790d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:48:03.976153 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:03.976144    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6289a2da-842a-484c-ad2d-c5ef9571790d-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6289a2da-842a-484c-ad2d-c5ef9571790d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:48:03.976362 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:03.976174    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6289a2da-842a-484c-ad2d-c5ef9571790d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6289a2da-842a-484c-ad2d-c5ef9571790d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:48:03.976362 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:03.976201    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6289a2da-842a-484c-ad2d-c5ef9571790d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6289a2da-842a-484c-ad2d-c5ef9571790d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:48:03.976362 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:03.976327    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6289a2da-842a-484c-ad2d-c5ef9571790d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6289a2da-842a-484c-ad2d-c5ef9571790d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:48:04.076941 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:04.076908    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6289a2da-842a-484c-ad2d-c5ef9571790d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6289a2da-842a-484c-ad2d-c5ef9571790d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:48:04.077142 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:04.076955    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6289a2da-842a-484c-ad2d-c5ef9571790d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6289a2da-842a-484c-ad2d-c5ef9571790d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:48:04.077142 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:04.076982    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6289a2da-842a-484c-ad2d-c5ef9571790d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6289a2da-842a-484c-ad2d-c5ef9571790d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:48:04.077142 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:04.077015    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ddmb\" (UniqueName: \"kubernetes.io/projected/6289a2da-842a-484c-ad2d-c5ef9571790d-kube-api-access-4ddmb\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6289a2da-842a-484c-ad2d-c5ef9571790d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:48:04.077142 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:04.077065    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6289a2da-842a-484c-ad2d-c5ef9571790d-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6289a2da-842a-484c-ad2d-c5ef9571790d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:48:04.077142 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:04.077086    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6289a2da-842a-484c-ad2d-c5ef9571790d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6289a2da-842a-484c-ad2d-c5ef9571790d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:48:04.077142 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:04.077108    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6289a2da-842a-484c-ad2d-c5ef9571790d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6289a2da-842a-484c-ad2d-c5ef9571790d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:48:04.077450 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:04.077375    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6289a2da-842a-484c-ad2d-c5ef9571790d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6289a2da-842a-484c-ad2d-c5ef9571790d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:48:04.077692 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:04.077513    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6289a2da-842a-484c-ad2d-c5ef9571790d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6289a2da-842a-484c-ad2d-c5ef9571790d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:48:04.077692 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:04.077674    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6289a2da-842a-484c-ad2d-c5ef9571790d-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6289a2da-842a-484c-ad2d-c5ef9571790d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:48:04.077884 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:04.077730    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6289a2da-842a-484c-ad2d-c5ef9571790d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6289a2da-842a-484c-ad2d-c5ef9571790d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:48:04.079696 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:04.079667    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6289a2da-842a-484c-ad2d-c5ef9571790d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6289a2da-842a-484c-ad2d-c5ef9571790d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:48:04.079832 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:04.079788    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6289a2da-842a-484c-ad2d-c5ef9571790d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6289a2da-842a-484c-ad2d-c5ef9571790d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:48:04.086446 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:04.086421    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ddmb\" (UniqueName: \"kubernetes.io/projected/6289a2da-842a-484c-ad2d-c5ef9571790d-kube-api-access-4ddmb\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6289a2da-842a-484c-ad2d-c5ef9571790d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:48:04.167375 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:04.167337    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:48:16.107162 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:16.107136    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 27 17:48:16.109369 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:48:16.109340    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6289a2da_842a_484c_ad2d_c5ef9571790d.slice/crio-b36465e1650c8aaa630627a54e6a104c80fe9720f2ae0aa240988946e04c30bf WatchSource:0}: Error finding container b36465e1650c8aaa630627a54e6a104c80fe9720f2ae0aa240988946e04c30bf: Status 404 returned error can't find the container with id b36465e1650c8aaa630627a54e6a104c80fe9720f2ae0aa240988946e04c30bf
Jul 27 17:48:16.940999 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:16.940959    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"6289a2da-842a-484c-ad2d-c5ef9571790d","Type":"ContainerStarted","Data":"bf0de8ef88fe4641d96e019072e224f67e5b71451ed50517b5c7df3830eead83"}
Jul 27 17:48:16.941288 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:16.941015    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"6289a2da-842a-484c-ad2d-c5ef9571790d","Type":"ContainerStarted","Data":"b36465e1650c8aaa630627a54e6a104c80fe9720f2ae0aa240988946e04c30bf"}
Jul 27 17:48:16.944671 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:16.943772    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr" event={"ID":"f6865ee1-94c1-4fca-b216-cae682b7f4d8","Type":"ContainerStarted","Data":"7fe4077c440c08a80b3158c3867aeaf3d74294a5fa2d5054950b5c7cfda18208"}
Jul 27 17:48:16.982772 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:16.982693    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr" podStartSLOduration=1.908365215 podStartE2EDuration="43.982673226s" totalImagesPullingTime="36.232145407s" totalInitContainerRuntime="5.842162604s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:47:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:48:16.981202483 +0000 UTC m=+1189.468013770" watchObservedRunningTime="2026-07-27 17:48:16.982673226 +0000 UTC m=+1189.469484514"
Jul 27 17:48:17.724968 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:17.724907    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"
Jul 27 17:48:22.975593 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:22.975563    2557 generic.go:356] "Generic (PLEG): container finished" podID="6289a2da-842a-484c-ad2d-c5ef9571790d" containerID="bf0de8ef88fe4641d96e019072e224f67e5b71451ed50517b5c7df3830eead83" exitCode=0
Jul 27 17:48:22.975997 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:22.975616    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"6289a2da-842a-484c-ad2d-c5ef9571790d","Type":"ContainerDied","Data":"bf0de8ef88fe4641d96e019072e224f67e5b71451ed50517b5c7df3830eead83"}
Jul 27 17:48:23.726202 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:23.725353    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"
Jul 27 17:48:23.726823 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:23.726793    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr" podUID="f6865ee1-94c1-4fca-b216-cae682b7f4d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 27 17:48:23.984775 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:23.984681    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"6289a2da-842a-484c-ad2d-c5ef9571790d","Type":"ContainerStarted","Data":"d85caf4376e9c668209f43fa974458c1e5caaa7b535222599f1d2ecc3c0107e9"}
Jul 27 17:48:24.007356 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:24.007302    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=14.423185126 podStartE2EDuration="21.007285365s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.584100239s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:48:24.005498231 +0000 UTC m=+1196.492309520" watchObservedRunningTime="2026-07-27 17:48:24.007285365 +0000 UTC m=+1196.494096652"
Jul 27 17:48:33.725264 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:33.725215    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr" podUID="f6865ee1-94c1-4fca-b216-cae682b7f4d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 27 17:48:43.725290 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:43.725241    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr" podUID="f6865ee1-94c1-4fca-b216-cae682b7f4d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 27 17:48:53.724951 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:48:53.724899    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr" podUID="f6865ee1-94c1-4fca-b216-cae682b7f4d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 27 17:49:03.724947 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:49:03.724897    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr" podUID="f6865ee1-94c1-4fca-b216-cae682b7f4d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 27 17:49:13.724636 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:49:13.724584    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr" podUID="f6865ee1-94c1-4fca-b216-cae682b7f4d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 27 17:49:23.724816 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:49:23.724703    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr" podUID="f6865ee1-94c1-4fca-b216-cae682b7f4d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 27 17:49:33.724849 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:49:33.724799    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr" podUID="f6865ee1-94c1-4fca-b216-cae682b7f4d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 27 17:49:43.725159 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:49:43.725114    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr" podUID="f6865ee1-94c1-4fca-b216-cae682b7f4d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 27 17:49:53.724966 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:49:53.724914    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr" podUID="f6865ee1-94c1-4fca-b216-cae682b7f4d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 27 17:50:03.725270 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:50:03.725227    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr" podUID="f6865ee1-94c1-4fca-b216-cae682b7f4d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 27 17:50:13.724785 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:50:13.724728    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr" podUID="f6865ee1-94c1-4fca-b216-cae682b7f4d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 27 17:50:23.725370 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:50:23.725326    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr" podUID="f6865ee1-94c1-4fca-b216-cae682b7f4d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 27 17:50:33.724853 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:50:33.724817    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr" podUID="f6865ee1-94c1-4fca-b216-cae682b7f4d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 27 17:50:43.735214 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:50:43.735180    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"
Jul 27 17:50:43.743121 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:50:43.743099    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"
Jul 27 17:50:48.001935 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:50:48.001892    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"]
Jul 27 17:51:03.026288 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:03.026182    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr" podUID="f6865ee1-94c1-4fca-b216-cae682b7f4d8" containerName="main" containerID="cri-o://7fe4077c440c08a80b3158c3867aeaf3d74294a5fa2d5054950b5c7cfda18208" gracePeriod=45
Jul 27 17:51:03.782126 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:03.782100    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"
Jul 27 17:51:03.810325 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:03.810243    2557 generic.go:356] "Generic (PLEG): container finished" podID="f6865ee1-94c1-4fca-b216-cae682b7f4d8" containerID="7fe4077c440c08a80b3158c3867aeaf3d74294a5fa2d5054950b5c7cfda18208" exitCode=0
Jul 27 17:51:03.810325 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:03.810294    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"
Jul 27 17:51:03.810557 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:03.810312    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr" event={"ID":"f6865ee1-94c1-4fca-b216-cae682b7f4d8","Type":"ContainerDied","Data":"7fe4077c440c08a80b3158c3867aeaf3d74294a5fa2d5054950b5c7cfda18208"}
Jul 27 17:51:03.810557 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:03.810379    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr" event={"ID":"f6865ee1-94c1-4fca-b216-cae682b7f4d8","Type":"ContainerDied","Data":"e1ad9ae010dc8165e08629e43d0623116c862b01432a71c906faa743486331d6"}
Jul 27 17:51:03.810557 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:03.810410    2557 scope.go:117] "RemoveContainer" containerID="7fe4077c440c08a80b3158c3867aeaf3d74294a5fa2d5054950b5c7cfda18208"
Jul 27 17:51:03.819183 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:03.819163    2557 scope.go:117] "RemoveContainer" containerID="7657b2b86cf4265be303c809cf08b8da628c25c0501d37314a64f246c68d46ab"
Jul 27 17:51:03.827144 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:03.827128    2557 scope.go:117] "RemoveContainer" containerID="7fe4077c440c08a80b3158c3867aeaf3d74294a5fa2d5054950b5c7cfda18208"
Jul 27 17:51:03.827377 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:51:03.827360    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fe4077c440c08a80b3158c3867aeaf3d74294a5fa2d5054950b5c7cfda18208\": container with ID starting with 7fe4077c440c08a80b3158c3867aeaf3d74294a5fa2d5054950b5c7cfda18208 not found: ID does not exist" containerID="7fe4077c440c08a80b3158c3867aeaf3d74294a5fa2d5054950b5c7cfda18208"
Jul 27 17:51:03.827431 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:03.827387    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fe4077c440c08a80b3158c3867aeaf3d74294a5fa2d5054950b5c7cfda18208"} err="failed to get container status \"7fe4077c440c08a80b3158c3867aeaf3d74294a5fa2d5054950b5c7cfda18208\": rpc error: code = NotFound desc = could not find container \"7fe4077c440c08a80b3158c3867aeaf3d74294a5fa2d5054950b5c7cfda18208\": container with ID starting with 7fe4077c440c08a80b3158c3867aeaf3d74294a5fa2d5054950b5c7cfda18208 not found: ID does not exist"
Jul 27 17:51:03.827431 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:03.827405    2557 scope.go:117] "RemoveContainer" containerID="7657b2b86cf4265be303c809cf08b8da628c25c0501d37314a64f246c68d46ab"
Jul 27 17:51:03.827637 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:51:03.827619    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7657b2b86cf4265be303c809cf08b8da628c25c0501d37314a64f246c68d46ab\": container with ID starting with 7657b2b86cf4265be303c809cf08b8da628c25c0501d37314a64f246c68d46ab not found: ID does not exist" containerID="7657b2b86cf4265be303c809cf08b8da628c25c0501d37314a64f246c68d46ab"
Jul 27 17:51:03.827694 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:03.827645    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7657b2b86cf4265be303c809cf08b8da628c25c0501d37314a64f246c68d46ab"} err="failed to get container status \"7657b2b86cf4265be303c809cf08b8da628c25c0501d37314a64f246c68d46ab\": rpc error: code = NotFound desc = could not find container \"7657b2b86cf4265be303c809cf08b8da628c25c0501d37314a64f246c68d46ab\": container with ID starting with 7657b2b86cf4265be303c809cf08b8da628c25c0501d37314a64f246c68d46ab not found: ID does not exist"
Jul 27 17:51:03.919175 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:03.919142    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6865ee1-94c1-4fca-b216-cae682b7f4d8-kserve-provision-location\") pod \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\" (UID: \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\") "
Jul 27 17:51:03.919354 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:03.919196    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6865ee1-94c1-4fca-b216-cae682b7f4d8-dshm\") pod \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\" (UID: \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\") "
Jul 27 17:51:03.919354 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:03.919256    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6865ee1-94c1-4fca-b216-cae682b7f4d8-model-cache\") pod \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\" (UID: \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\") "
Jul 27 17:51:03.919354 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:03.919291    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6865ee1-94c1-4fca-b216-cae682b7f4d8-tls-certs\") pod \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\" (UID: \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\") "
Jul 27 17:51:03.919354 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:03.919325    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6865ee1-94c1-4fca-b216-cae682b7f4d8-tmp-dir\") pod \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\" (UID: \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\") "
Jul 27 17:51:03.919564 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:03.919360    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdskb\" (UniqueName: \"kubernetes.io/projected/f6865ee1-94c1-4fca-b216-cae682b7f4d8-kube-api-access-qdskb\") pod \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\" (UID: \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\") "
Jul 27 17:51:03.919564 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:03.919506    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6865ee1-94c1-4fca-b216-cae682b7f4d8-home\") pod \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\" (UID: \"f6865ee1-94c1-4fca-b216-cae682b7f4d8\") "
Jul 27 17:51:03.919673 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:03.919592    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6865ee1-94c1-4fca-b216-cae682b7f4d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "f6865ee1-94c1-4fca-b216-cae682b7f4d8" (UID: "f6865ee1-94c1-4fca-b216-cae682b7f4d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:51:03.919856 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:03.919835    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6865ee1-94c1-4fca-b216-cae682b7f4d8-model-cache\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:51:03.920235 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:03.920214    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6865ee1-94c1-4fca-b216-cae682b7f4d8-home" (OuterVolumeSpecName: "home") pod "f6865ee1-94c1-4fca-b216-cae682b7f4d8" (UID: "f6865ee1-94c1-4fca-b216-cae682b7f4d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:51:03.921606 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:03.921578    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6865ee1-94c1-4fca-b216-cae682b7f4d8-kube-api-access-qdskb" (OuterVolumeSpecName: "kube-api-access-qdskb") pod "f6865ee1-94c1-4fca-b216-cae682b7f4d8" (UID: "f6865ee1-94c1-4fca-b216-cae682b7f4d8"). InnerVolumeSpecName "kube-api-access-qdskb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:51:03.921834 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:03.921815    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6865ee1-94c1-4fca-b216-cae682b7f4d8-dshm" (OuterVolumeSpecName: "dshm") pod "f6865ee1-94c1-4fca-b216-cae682b7f4d8" (UID: "f6865ee1-94c1-4fca-b216-cae682b7f4d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:51:03.921920 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:03.921902    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6865ee1-94c1-4fca-b216-cae682b7f4d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f6865ee1-94c1-4fca-b216-cae682b7f4d8" (UID: "f6865ee1-94c1-4fca-b216-cae682b7f4d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:51:03.933091 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:03.933067    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6865ee1-94c1-4fca-b216-cae682b7f4d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f6865ee1-94c1-4fca-b216-cae682b7f4d8" (UID: "f6865ee1-94c1-4fca-b216-cae682b7f4d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:51:03.975293 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:03.975266    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6865ee1-94c1-4fca-b216-cae682b7f4d8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f6865ee1-94c1-4fca-b216-cae682b7f4d8" (UID: "f6865ee1-94c1-4fca-b216-cae682b7f4d8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:51:04.020797 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:04.020745    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6865ee1-94c1-4fca-b216-cae682b7f4d8-tls-certs\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:51:04.020797 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:04.020794    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6865ee1-94c1-4fca-b216-cae682b7f4d8-tmp-dir\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:51:04.020941 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:04.020804    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdskb\" (UniqueName: \"kubernetes.io/projected/f6865ee1-94c1-4fca-b216-cae682b7f4d8-kube-api-access-qdskb\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:51:04.020941 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:04.020814    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6865ee1-94c1-4fca-b216-cae682b7f4d8-home\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:51:04.020941 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:04.020823    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6865ee1-94c1-4fca-b216-cae682b7f4d8-kserve-provision-location\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:51:04.020941 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:04.020832    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6865ee1-94c1-4fca-b216-cae682b7f4d8-dshm\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:51:04.136354 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:04.136319    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"]
Jul 27 17:51:04.142921 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:04.142899    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr"]
Jul 27 17:51:04.736633 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:04.736588    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-85d957ff8-n49nr" podUID="f6865ee1-94c1-4fca-b216-cae682b7f4d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": context deadline exceeded"
Jul 27 17:51:06.119152 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:06.119117    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6865ee1-94c1-4fca-b216-cae682b7f4d8" path="/var/lib/kubelet/pods/f6865ee1-94c1-4fca-b216-cae682b7f4d8/volumes"
Jul 27 17:51:09.340004 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:09.339965    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 27 17:51:24.362250 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:24.362207    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="6289a2da-842a-484c-ad2d-c5ef9571790d" containerName="main" containerID="cri-o://d85caf4376e9c668209f43fa974458c1e5caaa7b535222599f1d2ecc3c0107e9" gracePeriod=45
Jul 27 17:51:24.908274 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:24.908243    2557 generic.go:356] "Generic (PLEG): container finished" podID="6289a2da-842a-484c-ad2d-c5ef9571790d" containerID="d85caf4376e9c668209f43fa974458c1e5caaa7b535222599f1d2ecc3c0107e9" exitCode=0
Jul 27 17:51:24.908439 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:24.908316    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"6289a2da-842a-484c-ad2d-c5ef9571790d","Type":"ContainerDied","Data":"d85caf4376e9c668209f43fa974458c1e5caaa7b535222599f1d2ecc3c0107e9"}
Jul 27 17:51:25.024036 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:25.024012    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:51:25.096893 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:25.096823    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6289a2da-842a-484c-ad2d-c5ef9571790d-model-cache\") pod \"6289a2da-842a-484c-ad2d-c5ef9571790d\" (UID: \"6289a2da-842a-484c-ad2d-c5ef9571790d\") "
Jul 27 17:51:25.096893 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:25.096859    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6289a2da-842a-484c-ad2d-c5ef9571790d-tls-certs\") pod \"6289a2da-842a-484c-ad2d-c5ef9571790d\" (UID: \"6289a2da-842a-484c-ad2d-c5ef9571790d\") "
Jul 27 17:51:25.097108 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:25.096916    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6289a2da-842a-484c-ad2d-c5ef9571790d-home\") pod \"6289a2da-842a-484c-ad2d-c5ef9571790d\" (UID: \"6289a2da-842a-484c-ad2d-c5ef9571790d\") "
Jul 27 17:51:25.097108 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:25.096941    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6289a2da-842a-484c-ad2d-c5ef9571790d-dshm\") pod \"6289a2da-842a-484c-ad2d-c5ef9571790d\" (UID: \"6289a2da-842a-484c-ad2d-c5ef9571790d\") "
Jul 27 17:51:25.097108 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:25.096982    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6289a2da-842a-484c-ad2d-c5ef9571790d-tmp-dir\") pod \"6289a2da-842a-484c-ad2d-c5ef9571790d\" (UID: \"6289a2da-842a-484c-ad2d-c5ef9571790d\") "
Jul 27 17:51:25.097108 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:25.097019    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ddmb\" (UniqueName: \"kubernetes.io/projected/6289a2da-842a-484c-ad2d-c5ef9571790d-kube-api-access-4ddmb\") pod \"6289a2da-842a-484c-ad2d-c5ef9571790d\" (UID: \"6289a2da-842a-484c-ad2d-c5ef9571790d\") "
Jul 27 17:51:25.097108 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:25.097047    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6289a2da-842a-484c-ad2d-c5ef9571790d-kserve-provision-location\") pod \"6289a2da-842a-484c-ad2d-c5ef9571790d\" (UID: \"6289a2da-842a-484c-ad2d-c5ef9571790d\") "
Jul 27 17:51:25.097360 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:25.097100    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6289a2da-842a-484c-ad2d-c5ef9571790d-model-cache" (OuterVolumeSpecName: "model-cache") pod "6289a2da-842a-484c-ad2d-c5ef9571790d" (UID: "6289a2da-842a-484c-ad2d-c5ef9571790d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:51:25.097360 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:25.097334    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6289a2da-842a-484c-ad2d-c5ef9571790d-model-cache\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:51:25.097806 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:25.097780    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6289a2da-842a-484c-ad2d-c5ef9571790d-home" (OuterVolumeSpecName: "home") pod "6289a2da-842a-484c-ad2d-c5ef9571790d" (UID: "6289a2da-842a-484c-ad2d-c5ef9571790d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:51:25.099244 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:25.099221    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6289a2da-842a-484c-ad2d-c5ef9571790d-dshm" (OuterVolumeSpecName: "dshm") pod "6289a2da-842a-484c-ad2d-c5ef9571790d" (UID: "6289a2da-842a-484c-ad2d-c5ef9571790d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:51:25.099416 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:25.099393    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6289a2da-842a-484c-ad2d-c5ef9571790d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6289a2da-842a-484c-ad2d-c5ef9571790d" (UID: "6289a2da-842a-484c-ad2d-c5ef9571790d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:51:25.099528 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:25.099508    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6289a2da-842a-484c-ad2d-c5ef9571790d-kube-api-access-4ddmb" (OuterVolumeSpecName: "kube-api-access-4ddmb") pod "6289a2da-842a-484c-ad2d-c5ef9571790d" (UID: "6289a2da-842a-484c-ad2d-c5ef9571790d"). InnerVolumeSpecName "kube-api-access-4ddmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:51:25.109684 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:25.109662    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6289a2da-842a-484c-ad2d-c5ef9571790d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6289a2da-842a-484c-ad2d-c5ef9571790d" (UID: "6289a2da-842a-484c-ad2d-c5ef9571790d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:51:25.154630 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:25.154604    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6289a2da-842a-484c-ad2d-c5ef9571790d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6289a2da-842a-484c-ad2d-c5ef9571790d" (UID: "6289a2da-842a-484c-ad2d-c5ef9571790d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:51:25.197910 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:25.197886    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6289a2da-842a-484c-ad2d-c5ef9571790d-tmp-dir\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:51:25.197910 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:25.197911    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ddmb\" (UniqueName: \"kubernetes.io/projected/6289a2da-842a-484c-ad2d-c5ef9571790d-kube-api-access-4ddmb\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:51:25.198040 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:25.197920    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6289a2da-842a-484c-ad2d-c5ef9571790d-kserve-provision-location\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:51:25.198040 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:25.197929    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6289a2da-842a-484c-ad2d-c5ef9571790d-tls-certs\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:51:25.198040 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:25.197938    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6289a2da-842a-484c-ad2d-c5ef9571790d-home\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:51:25.198040 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:25.197947    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6289a2da-842a-484c-ad2d-c5ef9571790d-dshm\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:51:25.914166 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:25.914137    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:51:25.914166 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:25.914141    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"6289a2da-842a-484c-ad2d-c5ef9571790d","Type":"ContainerDied","Data":"b36465e1650c8aaa630627a54e6a104c80fe9720f2ae0aa240988946e04c30bf"}
Jul 27 17:51:25.914700 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:25.914200    2557 scope.go:117] "RemoveContainer" containerID="d85caf4376e9c668209f43fa974458c1e5caaa7b535222599f1d2ecc3c0107e9"
Jul 27 17:51:25.925374 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:25.925349    2557 scope.go:117] "RemoveContainer" containerID="bf0de8ef88fe4641d96e019072e224f67e5b71451ed50517b5c7df3830eead83"
Jul 27 17:51:25.938041 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:25.938017    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 27 17:51:25.943999 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:25.943979    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 27 17:51:26.119233 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:51:26.119203    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6289a2da-842a-484c-ad2d-c5ef9571790d" path="/var/lib/kubelet/pods/6289a2da-842a-484c-ad2d-c5ef9571790d/volumes"
Jul 27 17:52:20.702811 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.702717    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr"]
Jul 27 17:52:20.705137 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.703102    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6289a2da-842a-484c-ad2d-c5ef9571790d" containerName="main"
Jul 27 17:52:20.705137 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.703115    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6289a2da-842a-484c-ad2d-c5ef9571790d" containerName="main"
Jul 27 17:52:20.705137 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.703131    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6865ee1-94c1-4fca-b216-cae682b7f4d8" containerName="storage-initializer"
Jul 27 17:52:20.705137 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.703136    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6865ee1-94c1-4fca-b216-cae682b7f4d8" containerName="storage-initializer"
Jul 27 17:52:20.705137 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.703143    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6865ee1-94c1-4fca-b216-cae682b7f4d8" containerName="main"
Jul 27 17:52:20.705137 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.703149    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6865ee1-94c1-4fca-b216-cae682b7f4d8" containerName="main"
Jul 27 17:52:20.705137 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.703168    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6289a2da-842a-484c-ad2d-c5ef9571790d" containerName="storage-initializer"
Jul 27 17:52:20.705137 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.703173    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6289a2da-842a-484c-ad2d-c5ef9571790d" containerName="storage-initializer"
Jul 27 17:52:20.705137 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.703231    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6865ee1-94c1-4fca-b216-cae682b7f4d8" containerName="main"
Jul 27 17:52:20.705137 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.703238    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6289a2da-842a-484c-ad2d-c5ef9571790d" containerName="main"
Jul 27 17:52:20.706088 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.706072    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr"
Jul 27 17:52:20.709316 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.709286    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 27 17:52:20.709685 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.709393    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Jul 27 17:52:20.709685 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.709431    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-qxwwh\""
Jul 27 17:52:20.709954 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.709932    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Jul 27 17:52:20.719872 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.719847    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr"]
Jul 27 17:52:20.780688 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.780658    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44pfb\" (UniqueName: \"kubernetes.io/projected/a9a3946d-5922-40aa-9309-75f69e1ce1fe-kube-api-access-44pfb\") pod \"scheduler-inline-config-test-kserve-8655bc757-gn8rr\" (UID: \"a9a3946d-5922-40aa-9309-75f69e1ce1fe\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr"
Jul 27 17:52:20.780903 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.780695    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9a3946d-5922-40aa-9309-75f69e1ce1fe-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-gn8rr\" (UID: \"a9a3946d-5922-40aa-9309-75f69e1ce1fe\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr"
Jul 27 17:52:20.780903 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.780790    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9a3946d-5922-40aa-9309-75f69e1ce1fe-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-gn8rr\" (UID: \"a9a3946d-5922-40aa-9309-75f69e1ce1fe\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr"
Jul 27 17:52:20.780903 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.780847    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9a3946d-5922-40aa-9309-75f69e1ce1fe-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-gn8rr\" (UID: \"a9a3946d-5922-40aa-9309-75f69e1ce1fe\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr"
Jul 27 17:52:20.780903 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.780871    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9a3946d-5922-40aa-9309-75f69e1ce1fe-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-gn8rr\" (UID: \"a9a3946d-5922-40aa-9309-75f69e1ce1fe\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr"
Jul 27 17:52:20.781064 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.780922    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9a3946d-5922-40aa-9309-75f69e1ce1fe-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-gn8rr\" (UID: \"a9a3946d-5922-40aa-9309-75f69e1ce1fe\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr"
Jul 27 17:52:20.881840 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.881809    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44pfb\" (UniqueName: \"kubernetes.io/projected/a9a3946d-5922-40aa-9309-75f69e1ce1fe-kube-api-access-44pfb\") pod \"scheduler-inline-config-test-kserve-8655bc757-gn8rr\" (UID: \"a9a3946d-5922-40aa-9309-75f69e1ce1fe\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr"
Jul 27 17:52:20.882037 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.881847    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9a3946d-5922-40aa-9309-75f69e1ce1fe-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-gn8rr\" (UID: \"a9a3946d-5922-40aa-9309-75f69e1ce1fe\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr"
Jul 27 17:52:20.882037 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.881887    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9a3946d-5922-40aa-9309-75f69e1ce1fe-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-gn8rr\" (UID: \"a9a3946d-5922-40aa-9309-75f69e1ce1fe\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr"
Jul 27 17:52:20.882037 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.881916    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9a3946d-5922-40aa-9309-75f69e1ce1fe-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-gn8rr\" (UID: \"a9a3946d-5922-40aa-9309-75f69e1ce1fe\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr"
Jul 27 17:52:20.882037 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.881935    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9a3946d-5922-40aa-9309-75f69e1ce1fe-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-gn8rr\" (UID: \"a9a3946d-5922-40aa-9309-75f69e1ce1fe\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr"
Jul 27 17:52:20.882037 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.881991    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9a3946d-5922-40aa-9309-75f69e1ce1fe-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-gn8rr\" (UID: \"a9a3946d-5922-40aa-9309-75f69e1ce1fe\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr"
Jul 27 17:52:20.882340 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.882325    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9a3946d-5922-40aa-9309-75f69e1ce1fe-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-gn8rr\" (UID: \"a9a3946d-5922-40aa-9309-75f69e1ce1fe\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr"
Jul 27 17:52:20.882394 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.882347    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9a3946d-5922-40aa-9309-75f69e1ce1fe-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-gn8rr\" (UID: \"a9a3946d-5922-40aa-9309-75f69e1ce1fe\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr"
Jul 27 17:52:20.882432 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.882408    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9a3946d-5922-40aa-9309-75f69e1ce1fe-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-gn8rr\" (UID: \"a9a3946d-5922-40aa-9309-75f69e1ce1fe\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr"
Jul 27 17:52:20.884140 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.884117    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9a3946d-5922-40aa-9309-75f69e1ce1fe-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-gn8rr\" (UID: \"a9a3946d-5922-40aa-9309-75f69e1ce1fe\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr"
Jul 27 17:52:20.884400 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.884382    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9a3946d-5922-40aa-9309-75f69e1ce1fe-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-gn8rr\" (UID: \"a9a3946d-5922-40aa-9309-75f69e1ce1fe\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr"
Jul 27 17:52:20.889346 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:20.889326    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44pfb\" (UniqueName: \"kubernetes.io/projected/a9a3946d-5922-40aa-9309-75f69e1ce1fe-kube-api-access-44pfb\") pod \"scheduler-inline-config-test-kserve-8655bc757-gn8rr\" (UID: \"a9a3946d-5922-40aa-9309-75f69e1ce1fe\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr"
Jul 27 17:52:21.019040 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:21.018952    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr"
Jul 27 17:52:21.155036 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:21.155012    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr"]
Jul 27 17:52:21.157277 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:52:21.157249    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9a3946d_5922_40aa_9309_75f69e1ce1fe.slice/crio-e37ede33cb392b6b3ec9d78b3d2720108825efd0a4b650c2ef5c58be8b4d7883 WatchSource:0}: Error finding container e37ede33cb392b6b3ec9d78b3d2720108825efd0a4b650c2ef5c58be8b4d7883: Status 404 returned error can't find the container with id e37ede33cb392b6b3ec9d78b3d2720108825efd0a4b650c2ef5c58be8b4d7883
Jul 27 17:52:21.168838 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:21.168788    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr" event={"ID":"a9a3946d-5922-40aa-9309-75f69e1ce1fe","Type":"ContainerStarted","Data":"e37ede33cb392b6b3ec9d78b3d2720108825efd0a4b650c2ef5c58be8b4d7883"}
Jul 27 17:52:22.175895 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:22.175859    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr" event={"ID":"a9a3946d-5922-40aa-9309-75f69e1ce1fe","Type":"ContainerStarted","Data":"cd8b1cf24cf29dcc23e3aef7171ec78dbc36e12befed3b1e4b08e079fa9aa71a"}
Jul 27 17:52:22.197407 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:22.197361    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr" podStartSLOduration=2.197340337 podStartE2EDuration="2.197340337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:52:22.195431006 +0000 UTC m=+1434.682242314" watchObservedRunningTime="2026-07-27 17:52:22.197340337 +0000 UTC m=+1434.684151623"
Jul 27 17:52:23.019375 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:23.019341    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr"
Jul 27 17:52:31.019820 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:31.019788    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr"
Jul 27 17:52:31.031984 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:31.031957    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr"
Jul 27 17:52:31.230829 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:31.230801    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr"
Jul 27 17:52:54.104719 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:54.104649    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr"]
Jul 27 17:52:54.147304 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:52:54.147240    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:52:54.147304 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:52:54.147304 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 17:52:54.147304 ip-10-0-131-114 kubenswrapper[2557]:  > containerID="cd8b1cf24cf29dcc23e3aef7171ec78dbc36e12befed3b1e4b08e079fa9aa71a" cmd=["/bin/sleep","15"]
Jul 27 17:52:54.147801 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:54.147741    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr" message="" err=<
Jul 27 17:52:54.147801 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:52:54.147801 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 17:52:54.147801 ip-10-0-131-114 kubenswrapper[2557]:  >
Jul 27 17:52:54.148081 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:52:54.148062    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:52:54.148081 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:52:54.148081 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 17:52:54.148081 ip-10-0-131-114 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr" podUID="a9a3946d-5922-40aa-9309-75f69e1ce1fe" containerName="main" containerID="cri-o://cd8b1cf24cf29dcc23e3aef7171ec78dbc36e12befed3b1e4b08e079fa9aa71a"
Jul 27 17:52:54.148409 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:54.148375    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr" podUID="a9a3946d-5922-40aa-9309-75f69e1ce1fe" containerName="main" containerID="cri-o://cd8b1cf24cf29dcc23e3aef7171ec78dbc36e12befed3b1e4b08e079fa9aa71a" gracePeriod=60
Jul 27 17:52:54.222050 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:54.222004    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr" podUID="a9a3946d-5922-40aa-9309-75f69e1ce1fe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Jul 27 17:52:54.559797 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:54.559769    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr"
Jul 27 17:52:54.678238 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:54.678204    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9a3946d-5922-40aa-9309-75f69e1ce1fe-tls-certs\") pod \"a9a3946d-5922-40aa-9309-75f69e1ce1fe\" (UID: \"a9a3946d-5922-40aa-9309-75f69e1ce1fe\") "
Jul 27 17:52:54.678440 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:54.678284    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44pfb\" (UniqueName: \"kubernetes.io/projected/a9a3946d-5922-40aa-9309-75f69e1ce1fe-kube-api-access-44pfb\") pod \"a9a3946d-5922-40aa-9309-75f69e1ce1fe\" (UID: \"a9a3946d-5922-40aa-9309-75f69e1ce1fe\") "
Jul 27 17:52:54.678440 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:54.678333    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9a3946d-5922-40aa-9309-75f69e1ce1fe-dshm\") pod \"a9a3946d-5922-40aa-9309-75f69e1ce1fe\" (UID: \"a9a3946d-5922-40aa-9309-75f69e1ce1fe\") "
Jul 27 17:52:54.678440 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:54.678363    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9a3946d-5922-40aa-9309-75f69e1ce1fe-home\") pod \"a9a3946d-5922-40aa-9309-75f69e1ce1fe\" (UID: \"a9a3946d-5922-40aa-9309-75f69e1ce1fe\") "
Jul 27 17:52:54.678440 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:54.678406    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9a3946d-5922-40aa-9309-75f69e1ce1fe-model-cache\") pod \"a9a3946d-5922-40aa-9309-75f69e1ce1fe\" (UID: \"a9a3946d-5922-40aa-9309-75f69e1ce1fe\") "
Jul 27 17:52:54.678440 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:54.678434    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9a3946d-5922-40aa-9309-75f69e1ce1fe-tmp-dir\") pod \"a9a3946d-5922-40aa-9309-75f69e1ce1fe\" (UID: \"a9a3946d-5922-40aa-9309-75f69e1ce1fe\") "
Jul 27 17:52:54.678873 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:54.678649    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9a3946d-5922-40aa-9309-75f69e1ce1fe-home" (OuterVolumeSpecName: "home") pod "a9a3946d-5922-40aa-9309-75f69e1ce1fe" (UID: "a9a3946d-5922-40aa-9309-75f69e1ce1fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:52:54.678873 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:54.678662    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9a3946d-5922-40aa-9309-75f69e1ce1fe-model-cache" (OuterVolumeSpecName: "model-cache") pod "a9a3946d-5922-40aa-9309-75f69e1ce1fe" (UID: "a9a3946d-5922-40aa-9309-75f69e1ce1fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:52:54.678873 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:54.678819    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9a3946d-5922-40aa-9309-75f69e1ce1fe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a9a3946d-5922-40aa-9309-75f69e1ce1fe" (UID: "a9a3946d-5922-40aa-9309-75f69e1ce1fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:52:54.680585 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:54.680558    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9a3946d-5922-40aa-9309-75f69e1ce1fe-dshm" (OuterVolumeSpecName: "dshm") pod "a9a3946d-5922-40aa-9309-75f69e1ce1fe" (UID: "a9a3946d-5922-40aa-9309-75f69e1ce1fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:52:54.680703 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:54.680653    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9a3946d-5922-40aa-9309-75f69e1ce1fe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a9a3946d-5922-40aa-9309-75f69e1ce1fe" (UID: "a9a3946d-5922-40aa-9309-75f69e1ce1fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:52:54.680852 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:54.680828    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9a3946d-5922-40aa-9309-75f69e1ce1fe-kube-api-access-44pfb" (OuterVolumeSpecName: "kube-api-access-44pfb") pod "a9a3946d-5922-40aa-9309-75f69e1ce1fe" (UID: "a9a3946d-5922-40aa-9309-75f69e1ce1fe"). InnerVolumeSpecName "kube-api-access-44pfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:52:54.780401 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:54.780266    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9a3946d-5922-40aa-9309-75f69e1ce1fe-tmp-dir\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:52:54.780401 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:54.780305    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9a3946d-5922-40aa-9309-75f69e1ce1fe-tls-certs\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:52:54.780401 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:54.780322    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44pfb\" (UniqueName: \"kubernetes.io/projected/a9a3946d-5922-40aa-9309-75f69e1ce1fe-kube-api-access-44pfb\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:52:54.780401 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:54.780334    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9a3946d-5922-40aa-9309-75f69e1ce1fe-dshm\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:52:54.780401 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:54.780348    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9a3946d-5922-40aa-9309-75f69e1ce1fe-home\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:52:54.780401 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:54.780361    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9a3946d-5922-40aa-9309-75f69e1ce1fe-model-cache\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:52:55.342717 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:55.342677    2557 generic.go:356] "Generic (PLEG): container finished" podID="a9a3946d-5922-40aa-9309-75f69e1ce1fe" containerID="cd8b1cf24cf29dcc23e3aef7171ec78dbc36e12befed3b1e4b08e079fa9aa71a" exitCode=0
Jul 27 17:52:55.343151 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:55.342730    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr"
Jul 27 17:52:55.343151 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:55.342746    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr" event={"ID":"a9a3946d-5922-40aa-9309-75f69e1ce1fe","Type":"ContainerDied","Data":"cd8b1cf24cf29dcc23e3aef7171ec78dbc36e12befed3b1e4b08e079fa9aa71a"}
Jul 27 17:52:55.343151 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:55.342828    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr" event={"ID":"a9a3946d-5922-40aa-9309-75f69e1ce1fe","Type":"ContainerDied","Data":"e37ede33cb392b6b3ec9d78b3d2720108825efd0a4b650c2ef5c58be8b4d7883"}
Jul 27 17:52:55.343151 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:55.342859    2557 scope.go:117] "RemoveContainer" containerID="cd8b1cf24cf29dcc23e3aef7171ec78dbc36e12befed3b1e4b08e079fa9aa71a"
Jul 27 17:52:55.353833 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:55.353732    2557 scope.go:117] "RemoveContainer" containerID="cd8b1cf24cf29dcc23e3aef7171ec78dbc36e12befed3b1e4b08e079fa9aa71a"
Jul 27 17:52:55.354103 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:52:55.354075    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd8b1cf24cf29dcc23e3aef7171ec78dbc36e12befed3b1e4b08e079fa9aa71a\": container with ID starting with cd8b1cf24cf29dcc23e3aef7171ec78dbc36e12befed3b1e4b08e079fa9aa71a not found: ID does not exist" containerID="cd8b1cf24cf29dcc23e3aef7171ec78dbc36e12befed3b1e4b08e079fa9aa71a"
Jul 27 17:52:55.354188 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:55.354114    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd8b1cf24cf29dcc23e3aef7171ec78dbc36e12befed3b1e4b08e079fa9aa71a"} err="failed to get container status \"cd8b1cf24cf29dcc23e3aef7171ec78dbc36e12befed3b1e4b08e079fa9aa71a\": rpc error: code = NotFound desc = could not find container \"cd8b1cf24cf29dcc23e3aef7171ec78dbc36e12befed3b1e4b08e079fa9aa71a\": container with ID starting with cd8b1cf24cf29dcc23e3aef7171ec78dbc36e12befed3b1e4b08e079fa9aa71a not found: ID does not exist"
Jul 27 17:52:55.367643 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:55.367611    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr"]
Jul 27 17:52:55.373145 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:55.373115    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-gn8rr"]
Jul 27 17:52:56.118512 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:52:56.118471    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9a3946d-5922-40aa-9309-75f69e1ce1fe" path="/var/lib/kubelet/pods/a9a3946d-5922-40aa-9309-75f69e1ce1fe/volumes"
Jul 27 17:53:24.901775 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:24.898516    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc"]
Jul 27 17:53:24.901775 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:24.899434    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9a3946d-5922-40aa-9309-75f69e1ce1fe" containerName="main"
Jul 27 17:53:24.901775 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:24.899456    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9a3946d-5922-40aa-9309-75f69e1ce1fe" containerName="main"
Jul 27 17:53:24.901775 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:24.899643    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9a3946d-5922-40aa-9309-75f69e1ce1fe" containerName="main"
Jul 27 17:53:24.903151 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:24.903125    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc"
Jul 27 17:53:24.906543 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:24.906515    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-trnx4\""
Jul 27 17:53:24.906675 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:24.906585    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Jul 27 17:53:24.906675 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:24.906627    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Jul 27 17:53:24.906675 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:24.906585    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Jul 27 17:53:24.910037 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:24.910013    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc"]
Jul 27 17:53:25.057726 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:25.057672    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtd5b\" (UniqueName: \"kubernetes.io/projected/c5c787d5-ddc8-4146-b73e-8ed972df9058-kube-api-access-jtd5b\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc\" (UID: \"c5c787d5-ddc8-4146-b73e-8ed972df9058\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc"
Jul 27 17:53:25.058034 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:25.057740    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5c787d5-ddc8-4146-b73e-8ed972df9058-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc\" (UID: \"c5c787d5-ddc8-4146-b73e-8ed972df9058\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc"
Jul 27 17:53:25.058034 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:25.057826    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5c787d5-ddc8-4146-b73e-8ed972df9058-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc\" (UID: \"c5c787d5-ddc8-4146-b73e-8ed972df9058\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc"
Jul 27 17:53:25.058034 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:25.057862    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5c787d5-ddc8-4146-b73e-8ed972df9058-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc\" (UID: \"c5c787d5-ddc8-4146-b73e-8ed972df9058\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc"
Jul 27 17:53:25.058034 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:25.057887    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5c787d5-ddc8-4146-b73e-8ed972df9058-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc\" (UID: \"c5c787d5-ddc8-4146-b73e-8ed972df9058\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc"
Jul 27 17:53:25.058034 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:25.057944    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5c787d5-ddc8-4146-b73e-8ed972df9058-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc\" (UID: \"c5c787d5-ddc8-4146-b73e-8ed972df9058\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc"
Jul 27 17:53:25.159256 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:25.159226    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5c787d5-ddc8-4146-b73e-8ed972df9058-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc\" (UID: \"c5c787d5-ddc8-4146-b73e-8ed972df9058\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc"
Jul 27 17:53:25.159449 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:25.159263    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5c787d5-ddc8-4146-b73e-8ed972df9058-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc\" (UID: \"c5c787d5-ddc8-4146-b73e-8ed972df9058\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc"
Jul 27 17:53:25.159449 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:25.159291    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5c787d5-ddc8-4146-b73e-8ed972df9058-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc\" (UID: \"c5c787d5-ddc8-4146-b73e-8ed972df9058\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc"
Jul 27 17:53:25.159449 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:25.159346    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtd5b\" (UniqueName: \"kubernetes.io/projected/c5c787d5-ddc8-4146-b73e-8ed972df9058-kube-api-access-jtd5b\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc\" (UID: \"c5c787d5-ddc8-4146-b73e-8ed972df9058\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc"
Jul 27 17:53:25.159449 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:25.159384    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5c787d5-ddc8-4146-b73e-8ed972df9058-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc\" (UID: \"c5c787d5-ddc8-4146-b73e-8ed972df9058\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc"
Jul 27 17:53:25.159449 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:25.159433    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5c787d5-ddc8-4146-b73e-8ed972df9058-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc\" (UID: \"c5c787d5-ddc8-4146-b73e-8ed972df9058\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc"
Jul 27 17:53:25.159740 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:25.159717    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5c787d5-ddc8-4146-b73e-8ed972df9058-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc\" (UID: \"c5c787d5-ddc8-4146-b73e-8ed972df9058\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc"
Jul 27 17:53:25.159824 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:25.159787    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5c787d5-ddc8-4146-b73e-8ed972df9058-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc\" (UID: \"c5c787d5-ddc8-4146-b73e-8ed972df9058\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc"
Jul 27 17:53:25.159824 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:25.159814    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5c787d5-ddc8-4146-b73e-8ed972df9058-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc\" (UID: \"c5c787d5-ddc8-4146-b73e-8ed972df9058\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc"
Jul 27 17:53:25.161677 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:25.161653    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5c787d5-ddc8-4146-b73e-8ed972df9058-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc\" (UID: \"c5c787d5-ddc8-4146-b73e-8ed972df9058\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc"
Jul 27 17:53:25.161855 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:25.161836    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5c787d5-ddc8-4146-b73e-8ed972df9058-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc\" (UID: \"c5c787d5-ddc8-4146-b73e-8ed972df9058\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc"
Jul 27 17:53:25.166459 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:25.166439    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtd5b\" (UniqueName: \"kubernetes.io/projected/c5c787d5-ddc8-4146-b73e-8ed972df9058-kube-api-access-jtd5b\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc\" (UID: \"c5c787d5-ddc8-4146-b73e-8ed972df9058\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc"
Jul 27 17:53:25.216449 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:25.216420    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc"
Jul 27 17:53:25.357777 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:25.357732    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc"]
Jul 27 17:53:25.360481 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:53:25.360454    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c787d5_ddc8_4146_b73e_8ed972df9058.slice/crio-a676d605683a56e1f2dbfbd3402e7c597c194280c6e808b462e5933ab9bfac81 WatchSource:0}: Error finding container a676d605683a56e1f2dbfbd3402e7c597c194280c6e808b462e5933ab9bfac81: Status 404 returned error can't find the container with id a676d605683a56e1f2dbfbd3402e7c597c194280c6e808b462e5933ab9bfac81
Jul 27 17:53:25.480801 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:25.480766    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc" event={"ID":"c5c787d5-ddc8-4146-b73e-8ed972df9058","Type":"ContainerStarted","Data":"c5510f29a6fcd7effadcde2c945f7e4a4b14182c8cfb835946ff06d393ff9c10"}
Jul 27 17:53:25.480996 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:25.480821    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc" event={"ID":"c5c787d5-ddc8-4146-b73e-8ed972df9058","Type":"ContainerStarted","Data":"a676d605683a56e1f2dbfbd3402e7c597c194280c6e808b462e5933ab9bfac81"}
Jul 27 17:53:25.499034 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:25.498849    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc" podStartSLOduration=1.4988292300000001 podStartE2EDuration="1.49882923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:53:25.498339525 +0000 UTC m=+1497.985150815" watchObservedRunningTime="2026-07-27 17:53:25.49882923 +0000 UTC m=+1497.985640517"
Jul 27 17:53:26.217228 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:26.217185    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc"
Jul 27 17:53:35.217041 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:35.217006    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc"
Jul 27 17:53:35.230438 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:35.230407    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc"
Jul 27 17:53:35.545132 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:53:35.545048    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc"
Jul 27 17:54:20.660185 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:20.660149    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc"]
Jul 27 17:54:20.670947 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:54:20.670908    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:54:20.670947 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:54:20.670947 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 17:54:20.670947 ip-10-0-131-114 kubenswrapper[2557]:  > containerID="c5510f29a6fcd7effadcde2c945f7e4a4b14182c8cfb835946ff06d393ff9c10" cmd=["/bin/sleep","15"]
Jul 27 17:54:20.671222 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:20.670963    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc" message="" err=<
Jul 27 17:54:20.671222 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:54:20.671222 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 17:54:20.671222 ip-10-0-131-114 kubenswrapper[2557]:  >
Jul 27 17:54:20.671222 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:54:20.670975    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:54:20.671222 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:54:20.671222 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 17:54:20.671222 ip-10-0-131-114 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc" podUID="c5c787d5-ddc8-4146-b73e-8ed972df9058" containerName="main" containerID="cri-o://c5510f29a6fcd7effadcde2c945f7e4a4b14182c8cfb835946ff06d393ff9c10"
Jul 27 17:54:20.671222 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:20.671014    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc" podUID="c5c787d5-ddc8-4146-b73e-8ed972df9058" containerName="main" containerID="cri-o://c5510f29a6fcd7effadcde2c945f7e4a4b14182c8cfb835946ff06d393ff9c10" gracePeriod=60
Jul 27 17:54:21.422260 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:21.422236    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc"
Jul 27 17:54:21.464976 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:21.464944    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5c787d5-ddc8-4146-b73e-8ed972df9058-home\") pod \"c5c787d5-ddc8-4146-b73e-8ed972df9058\" (UID: \"c5c787d5-ddc8-4146-b73e-8ed972df9058\") "
Jul 27 17:54:21.465161 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:21.465010    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5c787d5-ddc8-4146-b73e-8ed972df9058-dshm\") pod \"c5c787d5-ddc8-4146-b73e-8ed972df9058\" (UID: \"c5c787d5-ddc8-4146-b73e-8ed972df9058\") "
Jul 27 17:54:21.465161 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:21.465035    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5c787d5-ddc8-4146-b73e-8ed972df9058-tmp-dir\") pod \"c5c787d5-ddc8-4146-b73e-8ed972df9058\" (UID: \"c5c787d5-ddc8-4146-b73e-8ed972df9058\") "
Jul 27 17:54:21.465161 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:21.465064    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5c787d5-ddc8-4146-b73e-8ed972df9058-model-cache\") pod \"c5c787d5-ddc8-4146-b73e-8ed972df9058\" (UID: \"c5c787d5-ddc8-4146-b73e-8ed972df9058\") "
Jul 27 17:54:21.465161 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:21.465097    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5c787d5-ddc8-4146-b73e-8ed972df9058-tls-certs\") pod \"c5c787d5-ddc8-4146-b73e-8ed972df9058\" (UID: \"c5c787d5-ddc8-4146-b73e-8ed972df9058\") "
Jul 27 17:54:21.465161 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:21.465145    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtd5b\" (UniqueName: \"kubernetes.io/projected/c5c787d5-ddc8-4146-b73e-8ed972df9058-kube-api-access-jtd5b\") pod \"c5c787d5-ddc8-4146-b73e-8ed972df9058\" (UID: \"c5c787d5-ddc8-4146-b73e-8ed972df9058\") "
Jul 27 17:54:21.465437 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:21.465213    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c787d5-ddc8-4146-b73e-8ed972df9058-home" (OuterVolumeSpecName: "home") pod "c5c787d5-ddc8-4146-b73e-8ed972df9058" (UID: "c5c787d5-ddc8-4146-b73e-8ed972df9058"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:54:21.465437 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:21.465292    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c787d5-ddc8-4146-b73e-8ed972df9058-model-cache" (OuterVolumeSpecName: "model-cache") pod "c5c787d5-ddc8-4146-b73e-8ed972df9058" (UID: "c5c787d5-ddc8-4146-b73e-8ed972df9058"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:54:21.465437 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:21.465333    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c787d5-ddc8-4146-b73e-8ed972df9058-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c5c787d5-ddc8-4146-b73e-8ed972df9058" (UID: "c5c787d5-ddc8-4146-b73e-8ed972df9058"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:54:21.465567 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:21.465449    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5c787d5-ddc8-4146-b73e-8ed972df9058-tmp-dir\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:54:21.465567 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:21.465466    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5c787d5-ddc8-4146-b73e-8ed972df9058-model-cache\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:54:21.465567 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:21.465474    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5c787d5-ddc8-4146-b73e-8ed972df9058-home\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:54:21.467538 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:21.467513    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c787d5-ddc8-4146-b73e-8ed972df9058-dshm" (OuterVolumeSpecName: "dshm") pod "c5c787d5-ddc8-4146-b73e-8ed972df9058" (UID: "c5c787d5-ddc8-4146-b73e-8ed972df9058"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:54:21.467712 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:21.467685    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5c787d5-ddc8-4146-b73e-8ed972df9058-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c5c787d5-ddc8-4146-b73e-8ed972df9058" (UID: "c5c787d5-ddc8-4146-b73e-8ed972df9058"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:54:21.467772 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:21.467693    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c787d5-ddc8-4146-b73e-8ed972df9058-kube-api-access-jtd5b" (OuterVolumeSpecName: "kube-api-access-jtd5b") pod "c5c787d5-ddc8-4146-b73e-8ed972df9058" (UID: "c5c787d5-ddc8-4146-b73e-8ed972df9058"). InnerVolumeSpecName "kube-api-access-jtd5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:54:21.566060 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:21.565965    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5c787d5-ddc8-4146-b73e-8ed972df9058-dshm\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:54:21.566060 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:21.565997    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5c787d5-ddc8-4146-b73e-8ed972df9058-tls-certs\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:54:21.566060 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:21.566010    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtd5b\" (UniqueName: \"kubernetes.io/projected/c5c787d5-ddc8-4146-b73e-8ed972df9058-kube-api-access-jtd5b\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:54:21.749958 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:21.749928    2557 generic.go:356] "Generic (PLEG): container finished" podID="c5c787d5-ddc8-4146-b73e-8ed972df9058" containerID="c5510f29a6fcd7effadcde2c945f7e4a4b14182c8cfb835946ff06d393ff9c10" exitCode=0
Jul 27 17:54:21.750461 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:21.749974    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc"
Jul 27 17:54:21.750461 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:21.750024    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc" event={"ID":"c5c787d5-ddc8-4146-b73e-8ed972df9058","Type":"ContainerDied","Data":"c5510f29a6fcd7effadcde2c945f7e4a4b14182c8cfb835946ff06d393ff9c10"}
Jul 27 17:54:21.750461 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:21.750092    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc" event={"ID":"c5c787d5-ddc8-4146-b73e-8ed972df9058","Type":"ContainerDied","Data":"a676d605683a56e1f2dbfbd3402e7c597c194280c6e808b462e5933ab9bfac81"}
Jul 27 17:54:21.750461 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:21.750127    2557 scope.go:117] "RemoveContainer" containerID="c5510f29a6fcd7effadcde2c945f7e4a4b14182c8cfb835946ff06d393ff9c10"
Jul 27 17:54:21.759674 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:21.759655    2557 scope.go:117] "RemoveContainer" containerID="c5510f29a6fcd7effadcde2c945f7e4a4b14182c8cfb835946ff06d393ff9c10"
Jul 27 17:54:21.759955 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:54:21.759937    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5510f29a6fcd7effadcde2c945f7e4a4b14182c8cfb835946ff06d393ff9c10\": container with ID starting with c5510f29a6fcd7effadcde2c945f7e4a4b14182c8cfb835946ff06d393ff9c10 not found: ID does not exist" containerID="c5510f29a6fcd7effadcde2c945f7e4a4b14182c8cfb835946ff06d393ff9c10"
Jul 27 17:54:21.760012 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:21.759964    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5510f29a6fcd7effadcde2c945f7e4a4b14182c8cfb835946ff06d393ff9c10"} err="failed to get container status \"c5510f29a6fcd7effadcde2c945f7e4a4b14182c8cfb835946ff06d393ff9c10\": rpc error: code = NotFound desc = could not find container \"c5510f29a6fcd7effadcde2c945f7e4a4b14182c8cfb835946ff06d393ff9c10\": container with ID starting with c5510f29a6fcd7effadcde2c945f7e4a4b14182c8cfb835946ff06d393ff9c10 not found: ID does not exist"
Jul 27 17:54:21.771030 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:21.771001    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc"]
Jul 27 17:54:21.773977 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:21.773956    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fqfcsc"]
Jul 27 17:54:22.119153 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:22.119116    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5c787d5-ddc8-4146-b73e-8ed972df9058" path="/var/lib/kubelet/pods/c5c787d5-ddc8-4146-b73e-8ed972df9058/volumes"
Jul 27 17:54:40.886113 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:40.886077    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5"]
Jul 27 17:54:40.886547 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:40.886417    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5c787d5-ddc8-4146-b73e-8ed972df9058" containerName="main"
Jul 27 17:54:40.886547 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:40.886428    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5c787d5-ddc8-4146-b73e-8ed972df9058" containerName="main"
Jul 27 17:54:40.886547 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:40.886506    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5c787d5-ddc8-4146-b73e-8ed972df9058" containerName="main"
Jul 27 17:54:40.889544 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:40.889522    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5"
Jul 27 17:54:40.891624 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:40.891603    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Jul 27 17:54:40.891736 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:40.891605    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Jul 27 17:54:40.892263 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:40.892247    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 27 17:54:40.892263 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:40.892257    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-kfjlv\""
Jul 27 17:54:40.897421 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:40.897394    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5"]
Jul 27 17:54:40.906823 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:40.906663    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6bcf433-f507-4917-9c48-778bd4e2912f-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-252m5\" (UID: \"f6bcf433-f507-4917-9c48-778bd4e2912f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5"
Jul 27 17:54:40.906823 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:40.906725    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6bcf433-f507-4917-9c48-778bd4e2912f-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-252m5\" (UID: \"f6bcf433-f507-4917-9c48-778bd4e2912f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5"
Jul 27 17:54:40.906823 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:40.906787    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6bcf433-f507-4917-9c48-778bd4e2912f-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-252m5\" (UID: \"f6bcf433-f507-4917-9c48-778bd4e2912f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5"
Jul 27 17:54:40.907062 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:40.906846    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6bcf433-f507-4917-9c48-778bd4e2912f-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-252m5\" (UID: \"f6bcf433-f507-4917-9c48-778bd4e2912f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5"
Jul 27 17:54:40.907062 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:40.906889    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcc6b\" (UniqueName: \"kubernetes.io/projected/f6bcf433-f507-4917-9c48-778bd4e2912f-kube-api-access-vcc6b\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-252m5\" (UID: \"f6bcf433-f507-4917-9c48-778bd4e2912f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5"
Jul 27 17:54:40.907062 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:40.906931    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6bcf433-f507-4917-9c48-778bd4e2912f-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-252m5\" (UID: \"f6bcf433-f507-4917-9c48-778bd4e2912f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5"
Jul 27 17:54:41.008311 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:41.008281    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6bcf433-f507-4917-9c48-778bd4e2912f-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-252m5\" (UID: \"f6bcf433-f507-4917-9c48-778bd4e2912f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5"
Jul 27 17:54:41.008311 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:41.008317    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6bcf433-f507-4917-9c48-778bd4e2912f-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-252m5\" (UID: \"f6bcf433-f507-4917-9c48-778bd4e2912f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5"
Jul 27 17:54:41.008575 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:41.008339    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6bcf433-f507-4917-9c48-778bd4e2912f-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-252m5\" (UID: \"f6bcf433-f507-4917-9c48-778bd4e2912f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5"
Jul 27 17:54:41.008575 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:41.008365    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcc6b\" (UniqueName: \"kubernetes.io/projected/f6bcf433-f507-4917-9c48-778bd4e2912f-kube-api-access-vcc6b\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-252m5\" (UID: \"f6bcf433-f507-4917-9c48-778bd4e2912f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5"
Jul 27 17:54:41.008575 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:41.008394    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6bcf433-f507-4917-9c48-778bd4e2912f-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-252m5\" (UID: \"f6bcf433-f507-4917-9c48-778bd4e2912f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5"
Jul 27 17:54:41.008575 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:41.008431    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6bcf433-f507-4917-9c48-778bd4e2912f-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-252m5\" (UID: \"f6bcf433-f507-4917-9c48-778bd4e2912f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5"
Jul 27 17:54:41.008843 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:41.008816    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6bcf433-f507-4917-9c48-778bd4e2912f-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-252m5\" (UID: \"f6bcf433-f507-4917-9c48-778bd4e2912f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5"
Jul 27 17:54:41.008909 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:41.008822    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6bcf433-f507-4917-9c48-778bd4e2912f-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-252m5\" (UID: \"f6bcf433-f507-4917-9c48-778bd4e2912f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5"
Jul 27 17:54:41.008909 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:41.008869    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6bcf433-f507-4917-9c48-778bd4e2912f-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-252m5\" (UID: \"f6bcf433-f507-4917-9c48-778bd4e2912f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5"
Jul 27 17:54:41.010695 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:41.010669    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6bcf433-f507-4917-9c48-778bd4e2912f-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-252m5\" (UID: \"f6bcf433-f507-4917-9c48-778bd4e2912f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5"
Jul 27 17:54:41.010877 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:41.010859    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6bcf433-f507-4917-9c48-778bd4e2912f-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-252m5\" (UID: \"f6bcf433-f507-4917-9c48-778bd4e2912f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5"
Jul 27 17:54:41.016008 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:41.015990    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcc6b\" (UniqueName: \"kubernetes.io/projected/f6bcf433-f507-4917-9c48-778bd4e2912f-kube-api-access-vcc6b\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-252m5\" (UID: \"f6bcf433-f507-4917-9c48-778bd4e2912f\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5"
Jul 27 17:54:41.201069 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:41.201033    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5"
Jul 27 17:54:41.352044 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:41.351875    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5"]
Jul 27 17:54:41.354468 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:54:41.354442    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6bcf433_f507_4917_9c48_778bd4e2912f.slice/crio-f0d048bb60ebd445f9072c401367ad7bc864ad6ef1e7b831e09d612ebf199b83 WatchSource:0}: Error finding container f0d048bb60ebd445f9072c401367ad7bc864ad6ef1e7b831e09d612ebf199b83: Status 404 returned error can't find the container with id f0d048bb60ebd445f9072c401367ad7bc864ad6ef1e7b831e09d612ebf199b83
Jul 27 17:54:41.841776 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:41.841726    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5" event={"ID":"f6bcf433-f507-4917-9c48-778bd4e2912f","Type":"ContainerStarted","Data":"fa6bf3a09359231f36d7cff1c9e56095300e33232c291596ec6ff6fc96863ae3"}
Jul 27 17:54:41.841943 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:41.841814    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5" event={"ID":"f6bcf433-f507-4917-9c48-778bd4e2912f","Type":"ContainerStarted","Data":"f0d048bb60ebd445f9072c401367ad7bc864ad6ef1e7b831e09d612ebf199b83"}
Jul 27 17:54:41.866505 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:41.866435    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5" podStartSLOduration=1.866415812 podStartE2EDuration="1.866415812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:54:41.864929489 +0000 UTC m=+1574.351740796" watchObservedRunningTime="2026-07-27 17:54:41.866415812 +0000 UTC m=+1574.353227100"
Jul 27 17:54:42.201449 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:42.201410    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5"
Jul 27 17:54:51.202029 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:51.201975    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5"
Jul 27 17:54:51.214672 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:51.214647    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5"
Jul 27 17:54:51.900115 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:54:51.900082    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5"
Jul 27 17:55:14.548593 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:14.548558    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5"]
Jul 27 17:55:14.558364 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:55:14.558320    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:55:14.558364 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:55:14.558364 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 17:55:14.558364 ip-10-0-131-114 kubenswrapper[2557]:  > containerID="fa6bf3a09359231f36d7cff1c9e56095300e33232c291596ec6ff6fc96863ae3" cmd=["/bin/sleep","15"]
Jul 27 17:55:14.559809 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:14.559781    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5" message="" err=<
Jul 27 17:55:14.559809 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:55:14.559809 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 17:55:14.559809 ip-10-0-131-114 kubenswrapper[2557]:  >
Jul 27 17:55:14.560146 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:55:14.560123    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:55:14.560146 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:55:14.560146 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 17:55:14.560146 ip-10-0-131-114 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5" podUID="f6bcf433-f507-4917-9c48-778bd4e2912f" containerName="main" containerID="cri-o://fa6bf3a09359231f36d7cff1c9e56095300e33232c291596ec6ff6fc96863ae3"
Jul 27 17:55:14.560437 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:14.560414    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5" podUID="f6bcf433-f507-4917-9c48-778bd4e2912f" containerName="main" containerID="cri-o://fa6bf3a09359231f36d7cff1c9e56095300e33232c291596ec6ff6fc96863ae3" gracePeriod=60
Jul 27 17:55:14.930459 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:14.930434    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5"
Jul 27 17:55:14.997390 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:14.997352    2557 generic.go:356] "Generic (PLEG): container finished" podID="f6bcf433-f507-4917-9c48-778bd4e2912f" containerID="fa6bf3a09359231f36d7cff1c9e56095300e33232c291596ec6ff6fc96863ae3" exitCode=0
Jul 27 17:55:14.997578 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:14.997398    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5"
Jul 27 17:55:14.997578 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:14.997409    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5" event={"ID":"f6bcf433-f507-4917-9c48-778bd4e2912f","Type":"ContainerDied","Data":"fa6bf3a09359231f36d7cff1c9e56095300e33232c291596ec6ff6fc96863ae3"}
Jul 27 17:55:14.997578 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:14.997465    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5" event={"ID":"f6bcf433-f507-4917-9c48-778bd4e2912f","Type":"ContainerDied","Data":"f0d048bb60ebd445f9072c401367ad7bc864ad6ef1e7b831e09d612ebf199b83"}
Jul 27 17:55:14.997578 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:14.997481    2557 scope.go:117] "RemoveContainer" containerID="fa6bf3a09359231f36d7cff1c9e56095300e33232c291596ec6ff6fc96863ae3"
Jul 27 17:55:15.006848 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:15.006807    2557 scope.go:117] "RemoveContainer" containerID="fa6bf3a09359231f36d7cff1c9e56095300e33232c291596ec6ff6fc96863ae3"
Jul 27 17:55:15.007148 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:55:15.007129    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa6bf3a09359231f36d7cff1c9e56095300e33232c291596ec6ff6fc96863ae3\": container with ID starting with fa6bf3a09359231f36d7cff1c9e56095300e33232c291596ec6ff6fc96863ae3 not found: ID does not exist" containerID="fa6bf3a09359231f36d7cff1c9e56095300e33232c291596ec6ff6fc96863ae3"
Jul 27 17:55:15.007212 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:15.007161    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa6bf3a09359231f36d7cff1c9e56095300e33232c291596ec6ff6fc96863ae3"} err="failed to get container status \"fa6bf3a09359231f36d7cff1c9e56095300e33232c291596ec6ff6fc96863ae3\": rpc error: code = NotFound desc = could not find container \"fa6bf3a09359231f36d7cff1c9e56095300e33232c291596ec6ff6fc96863ae3\": container with ID starting with fa6bf3a09359231f36d7cff1c9e56095300e33232c291596ec6ff6fc96863ae3 not found: ID does not exist"
Jul 27 17:55:15.117416 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:15.117319    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6bcf433-f507-4917-9c48-778bd4e2912f-dshm\") pod \"f6bcf433-f507-4917-9c48-778bd4e2912f\" (UID: \"f6bcf433-f507-4917-9c48-778bd4e2912f\") "
Jul 27 17:55:15.117416 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:15.117367    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6bcf433-f507-4917-9c48-778bd4e2912f-tls-certs\") pod \"f6bcf433-f507-4917-9c48-778bd4e2912f\" (UID: \"f6bcf433-f507-4917-9c48-778bd4e2912f\") "
Jul 27 17:55:15.117416 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:15.117404    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6bcf433-f507-4917-9c48-778bd4e2912f-home\") pod \"f6bcf433-f507-4917-9c48-778bd4e2912f\" (UID: \"f6bcf433-f507-4917-9c48-778bd4e2912f\") "
Jul 27 17:55:15.117698 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:15.117431    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcc6b\" (UniqueName: \"kubernetes.io/projected/f6bcf433-f507-4917-9c48-778bd4e2912f-kube-api-access-vcc6b\") pod \"f6bcf433-f507-4917-9c48-778bd4e2912f\" (UID: \"f6bcf433-f507-4917-9c48-778bd4e2912f\") "
Jul 27 17:55:15.117698 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:15.117476    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6bcf433-f507-4917-9c48-778bd4e2912f-tmp-dir\") pod \"f6bcf433-f507-4917-9c48-778bd4e2912f\" (UID: \"f6bcf433-f507-4917-9c48-778bd4e2912f\") "
Jul 27 17:55:15.117698 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:15.117527    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6bcf433-f507-4917-9c48-778bd4e2912f-model-cache\") pod \"f6bcf433-f507-4917-9c48-778bd4e2912f\" (UID: \"f6bcf433-f507-4917-9c48-778bd4e2912f\") "
Jul 27 17:55:15.117883 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:15.117783    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6bcf433-f507-4917-9c48-778bd4e2912f-home" (OuterVolumeSpecName: "home") pod "f6bcf433-f507-4917-9c48-778bd4e2912f" (UID: "f6bcf433-f507-4917-9c48-778bd4e2912f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:55:15.117883 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:15.117797    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6bcf433-f507-4917-9c48-778bd4e2912f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f6bcf433-f507-4917-9c48-778bd4e2912f" (UID: "f6bcf433-f507-4917-9c48-778bd4e2912f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:55:15.117965 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:15.117895    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6bcf433-f507-4917-9c48-778bd4e2912f-model-cache" (OuterVolumeSpecName: "model-cache") pod "f6bcf433-f507-4917-9c48-778bd4e2912f" (UID: "f6bcf433-f507-4917-9c48-778bd4e2912f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:55:15.119697 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:15.119672    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6bcf433-f507-4917-9c48-778bd4e2912f-dshm" (OuterVolumeSpecName: "dshm") pod "f6bcf433-f507-4917-9c48-778bd4e2912f" (UID: "f6bcf433-f507-4917-9c48-778bd4e2912f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:55:15.119840 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:15.119781    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6bcf433-f507-4917-9c48-778bd4e2912f-kube-api-access-vcc6b" (OuterVolumeSpecName: "kube-api-access-vcc6b") pod "f6bcf433-f507-4917-9c48-778bd4e2912f" (UID: "f6bcf433-f507-4917-9c48-778bd4e2912f"). InnerVolumeSpecName "kube-api-access-vcc6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:55:15.119840 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:15.119818    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6bcf433-f507-4917-9c48-778bd4e2912f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f6bcf433-f507-4917-9c48-778bd4e2912f" (UID: "f6bcf433-f507-4917-9c48-778bd4e2912f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:55:15.218579 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:15.218537    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6bcf433-f507-4917-9c48-778bd4e2912f-tmp-dir\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:55:15.218579 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:15.218571    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6bcf433-f507-4917-9c48-778bd4e2912f-model-cache\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:55:15.218579 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:15.218583    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6bcf433-f507-4917-9c48-778bd4e2912f-dshm\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:55:15.218878 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:15.218594    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6bcf433-f507-4917-9c48-778bd4e2912f-tls-certs\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:55:15.218878 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:15.218604    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6bcf433-f507-4917-9c48-778bd4e2912f-home\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:55:15.218878 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:15.218616    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcc6b\" (UniqueName: \"kubernetes.io/projected/f6bcf433-f507-4917-9c48-778bd4e2912f-kube-api-access-vcc6b\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:55:15.322068 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:15.322037    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5"]
Jul 27 17:55:15.325773 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:15.325732    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5"]
Jul 27 17:55:15.896779 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:15.896714    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-252m5" podUID="f6bcf433-f507-4917-9c48-778bd4e2912f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 27 17:55:16.119291 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:16.119257    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6bcf433-f507-4917-9c48-778bd4e2912f" path="/var/lib/kubelet/pods/f6bcf433-f507-4917-9c48-778bd4e2912f/volumes"
Jul 27 17:55:42.788900 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:42.788870    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg"]
Jul 27 17:55:42.789284 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:42.789246    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6bcf433-f507-4917-9c48-778bd4e2912f" containerName="main"
Jul 27 17:55:42.789284 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:42.789258    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6bcf433-f507-4917-9c48-778bd4e2912f" containerName="main"
Jul 27 17:55:42.789371 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:42.789316    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6bcf433-f507-4917-9c48-778bd4e2912f" containerName="main"
Jul 27 17:55:42.792329 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:42.792309    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg"
Jul 27 17:55:42.794508 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:42.794473    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Jul 27 17:55:42.795110 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:42.795089    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Jul 27 17:55:42.795230 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:42.795184    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 27 17:55:42.795230 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:42.795211    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-flx8t\""
Jul 27 17:55:42.800885 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:42.800580    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg"]
Jul 27 17:55:42.827956 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:42.827924    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8b9b799-3159-4851-8e76-c5f735872a29-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg\" (UID: \"a8b9b799-3159-4851-8e76-c5f735872a29\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg"
Jul 27 17:55:42.828121 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:42.827963    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csqvq\" (UniqueName: \"kubernetes.io/projected/a8b9b799-3159-4851-8e76-c5f735872a29-kube-api-access-csqvq\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg\" (UID: \"a8b9b799-3159-4851-8e76-c5f735872a29\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg"
Jul 27 17:55:42.828121 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:42.827992    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8b9b799-3159-4851-8e76-c5f735872a29-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg\" (UID: \"a8b9b799-3159-4851-8e76-c5f735872a29\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg"
Jul 27 17:55:42.828121 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:42.828106    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b9b799-3159-4851-8e76-c5f735872a29-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg\" (UID: \"a8b9b799-3159-4851-8e76-c5f735872a29\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg"
Jul 27 17:55:42.828269 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:42.828146    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8b9b799-3159-4851-8e76-c5f735872a29-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg\" (UID: \"a8b9b799-3159-4851-8e76-c5f735872a29\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg"
Jul 27 17:55:42.828269 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:42.828201    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8b9b799-3159-4851-8e76-c5f735872a29-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg\" (UID: \"a8b9b799-3159-4851-8e76-c5f735872a29\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg"
Jul 27 17:55:42.929401 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:42.929357    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8b9b799-3159-4851-8e76-c5f735872a29-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg\" (UID: \"a8b9b799-3159-4851-8e76-c5f735872a29\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg"
Jul 27 17:55:42.929401 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:42.929410    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b9b799-3159-4851-8e76-c5f735872a29-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg\" (UID: \"a8b9b799-3159-4851-8e76-c5f735872a29\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg"
Jul 27 17:55:42.929750 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:42.929526    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8b9b799-3159-4851-8e76-c5f735872a29-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg\" (UID: \"a8b9b799-3159-4851-8e76-c5f735872a29\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg"
Jul 27 17:55:42.929750 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:42.929591    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8b9b799-3159-4851-8e76-c5f735872a29-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg\" (UID: \"a8b9b799-3159-4851-8e76-c5f735872a29\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg"
Jul 27 17:55:42.929750 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:42.929637    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8b9b799-3159-4851-8e76-c5f735872a29-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg\" (UID: \"a8b9b799-3159-4851-8e76-c5f735872a29\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg"
Jul 27 17:55:42.929750 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:42.929675    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csqvq\" (UniqueName: \"kubernetes.io/projected/a8b9b799-3159-4851-8e76-c5f735872a29-kube-api-access-csqvq\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg\" (UID: \"a8b9b799-3159-4851-8e76-c5f735872a29\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg"
Jul 27 17:55:42.930012 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:42.929962    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8b9b799-3159-4851-8e76-c5f735872a29-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg\" (UID: \"a8b9b799-3159-4851-8e76-c5f735872a29\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg"
Jul 27 17:55:42.930073 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:42.930062    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8b9b799-3159-4851-8e76-c5f735872a29-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg\" (UID: \"a8b9b799-3159-4851-8e76-c5f735872a29\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg"
Jul 27 17:55:42.930128 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:42.930086    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8b9b799-3159-4851-8e76-c5f735872a29-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg\" (UID: \"a8b9b799-3159-4851-8e76-c5f735872a29\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg"
Jul 27 17:55:42.932382 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:42.932349    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8b9b799-3159-4851-8e76-c5f735872a29-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg\" (UID: \"a8b9b799-3159-4851-8e76-c5f735872a29\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg"
Jul 27 17:55:42.932639 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:42.932621    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b9b799-3159-4851-8e76-c5f735872a29-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg\" (UID: \"a8b9b799-3159-4851-8e76-c5f735872a29\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg"
Jul 27 17:55:42.937384 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:42.937363    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csqvq\" (UniqueName: \"kubernetes.io/projected/a8b9b799-3159-4851-8e76-c5f735872a29-kube-api-access-csqvq\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg\" (UID: \"a8b9b799-3159-4851-8e76-c5f735872a29\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg"
Jul 27 17:55:43.009750 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:43.009716    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkr9ds"]
Jul 27 17:55:43.013385 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:43.013363    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkr9ds"
Jul 27 17:55:43.015221 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:43.015201    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-96z8p\""
Jul 27 17:55:43.021903 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:43.021879    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkr9ds"]
Jul 27 17:55:43.105286 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:43.105193    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg"
Jul 27 17:55:43.130843 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:43.130805    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38762d94-db85-4889-a64b-c6e65b559dd1-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkr9ds\" (UID: \"38762d94-db85-4889-a64b-c6e65b559dd1\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkr9ds"
Jul 27 17:55:43.131020 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:43.130994    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l78kx\" (UniqueName: \"kubernetes.io/projected/38762d94-db85-4889-a64b-c6e65b559dd1-kube-api-access-l78kx\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkr9ds\" (UID: \"38762d94-db85-4889-a64b-c6e65b559dd1\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkr9ds"
Jul 27 17:55:43.232473 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:43.232442    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38762d94-db85-4889-a64b-c6e65b559dd1-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkr9ds\" (UID: \"38762d94-db85-4889-a64b-c6e65b559dd1\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkr9ds"
Jul 27 17:55:43.232636 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:43.232530    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l78kx\" (UniqueName: \"kubernetes.io/projected/38762d94-db85-4889-a64b-c6e65b559dd1-kube-api-access-l78kx\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkr9ds\" (UID: \"38762d94-db85-4889-a64b-c6e65b559dd1\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkr9ds"
Jul 27 17:55:43.234825 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:43.234796    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38762d94-db85-4889-a64b-c6e65b559dd1-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkr9ds\" (UID: \"38762d94-db85-4889-a64b-c6e65b559dd1\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkr9ds"
Jul 27 17:55:43.243136 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:43.243109    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg"]
Jul 27 17:55:43.244390 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:43.244370    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l78kx\" (UniqueName: \"kubernetes.io/projected/38762d94-db85-4889-a64b-c6e65b559dd1-kube-api-access-l78kx\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkr9ds\" (UID: \"38762d94-db85-4889-a64b-c6e65b559dd1\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkr9ds"
Jul 27 17:55:43.245250 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:55:43.245225    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b9b799_3159_4851_8e76_c5f735872a29.slice/crio-ac978c07ac6394b473567ed249aee2d04a484cf59962127ea70314692d2dced6 WatchSource:0}: Error finding container ac978c07ac6394b473567ed249aee2d04a484cf59962127ea70314692d2dced6: Status 404 returned error can't find the container with id ac978c07ac6394b473567ed249aee2d04a484cf59962127ea70314692d2dced6
Jul 27 17:55:43.324747 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:43.324713    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkr9ds"
Jul 27 17:55:43.455304 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:43.455276    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkr9ds"]
Jul 27 17:55:43.457969 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:55:43.457940    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38762d94_db85_4889_a64b_c6e65b559dd1.slice/crio-aa729ee5cc9a9aa1c1879aa6f5d0c16d0faeab932052f0815627b0a4530e93f1 WatchSource:0}: Error finding container aa729ee5cc9a9aa1c1879aa6f5d0c16d0faeab932052f0815627b0a4530e93f1: Status 404 returned error can't find the container with id aa729ee5cc9a9aa1c1879aa6f5d0c16d0faeab932052f0815627b0a4530e93f1
Jul 27 17:55:43.460003 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:43.459988    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:55:44.139275 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:44.139237    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg" event={"ID":"a8b9b799-3159-4851-8e76-c5f735872a29","Type":"ContainerStarted","Data":"addba91004470a5a3ebfb4b844f0b028926d7140bcff5d835a8382cdaa26a489"}
Jul 27 17:55:44.139727 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:44.139300    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg" event={"ID":"a8b9b799-3159-4851-8e76-c5f735872a29","Type":"ContainerStarted","Data":"ac978c07ac6394b473567ed249aee2d04a484cf59962127ea70314692d2dced6"}
Jul 27 17:55:44.140605 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:44.140578    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkr9ds" event={"ID":"38762d94-db85-4889-a64b-c6e65b559dd1","Type":"ContainerStarted","Data":"aa729ee5cc9a9aa1c1879aa6f5d0c16d0faeab932052f0815627b0a4530e93f1"}
Jul 27 17:55:44.155287 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:44.154817    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg" podStartSLOduration=2.154796867 podStartE2EDuration="2.154796867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:55:44.154274093 +0000 UTC m=+1636.641085401" watchObservedRunningTime="2026-07-27 17:55:44.154796867 +0000 UTC m=+1636.641608176"
Jul 27 17:55:45.105484 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:45.105448    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg"
Jul 27 17:55:46.152868 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:46.152778    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkr9ds" event={"ID":"38762d94-db85-4889-a64b-c6e65b559dd1","Type":"ContainerStarted","Data":"7fa4bef8aa8b8d1c3569475db46598ced83ddf82b5acb944543a6a61ae4563d5"}
Jul 27 17:55:46.153359 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:46.153242    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkr9ds"
Jul 27 17:55:46.171219 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:46.171168    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkr9ds" podStartSLOduration=1.747026792 podStartE2EDuration="4.171152822s" totalImagesPullingTime="2.42412603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:55:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:55:46.170178906 +0000 UTC m=+1638.656990208" watchObservedRunningTime="2026-07-27 17:55:46.171152822 +0000 UTC m=+1638.657964109"
Jul 27 17:55:53.106104 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:53.106071    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg"
Jul 27 17:55:53.118498 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:53.118462    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg"
Jul 27 17:55:53.199610 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:55:53.199583    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg"
Jul 27 17:56:16.093749 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:16.093714    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkr9ds"]
Jul 27 17:56:16.096020 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:16.095987    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkr9ds" podUID="38762d94-db85-4889-a64b-c6e65b559dd1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 17:56:16.103405 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:16.103378    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg"]
Jul 27 17:56:16.112215 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:56:16.112178    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:56:16.112215 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:56:16.112215 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 17:56:16.112215 ip-10-0-131-114 kubenswrapper[2557]:  > containerID="addba91004470a5a3ebfb4b844f0b028926d7140bcff5d835a8382cdaa26a489" cmd=["/bin/sleep","15"]
Jul 27 17:56:16.112432 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:16.112237    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg" message="" err=<
Jul 27 17:56:16.112432 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:56:16.112432 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 17:56:16.112432 ip-10-0-131-114 kubenswrapper[2557]:  >
Jul 27 17:56:16.112432 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:56:16.112252    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:56:16.112432 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:56:16.112432 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 17:56:16.112432 ip-10-0-131-114 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg" podUID="a8b9b799-3159-4851-8e76-c5f735872a29" containerName="main" containerID="cri-o://addba91004470a5a3ebfb4b844f0b028926d7140bcff5d835a8382cdaa26a489"
Jul 27 17:56:16.112432 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:16.112306    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg" podUID="a8b9b799-3159-4851-8e76-c5f735872a29" containerName="main" containerID="cri-o://addba91004470a5a3ebfb4b844f0b028926d7140bcff5d835a8382cdaa26a489" gracePeriod=60
Jul 27 17:56:16.188837 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:16.188798    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg" podUID="a8b9b799-3159-4851-8e76-c5f735872a29" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Jul 27 17:56:16.457907 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:16.457886    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg"
Jul 27 17:56:16.630650 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:16.630568    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8b9b799-3159-4851-8e76-c5f735872a29-dshm\") pod \"a8b9b799-3159-4851-8e76-c5f735872a29\" (UID: \"a8b9b799-3159-4851-8e76-c5f735872a29\") "
Jul 27 17:56:16.630650 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:16.630616    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8b9b799-3159-4851-8e76-c5f735872a29-tmp-dir\") pod \"a8b9b799-3159-4851-8e76-c5f735872a29\" (UID: \"a8b9b799-3159-4851-8e76-c5f735872a29\") "
Jul 27 17:56:16.630650 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:16.630635    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csqvq\" (UniqueName: \"kubernetes.io/projected/a8b9b799-3159-4851-8e76-c5f735872a29-kube-api-access-csqvq\") pod \"a8b9b799-3159-4851-8e76-c5f735872a29\" (UID: \"a8b9b799-3159-4851-8e76-c5f735872a29\") "
Jul 27 17:56:16.630650 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:16.630650    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b9b799-3159-4851-8e76-c5f735872a29-tls-certs\") pod \"a8b9b799-3159-4851-8e76-c5f735872a29\" (UID: \"a8b9b799-3159-4851-8e76-c5f735872a29\") "
Jul 27 17:56:16.631082 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:16.630671    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8b9b799-3159-4851-8e76-c5f735872a29-home\") pod \"a8b9b799-3159-4851-8e76-c5f735872a29\" (UID: \"a8b9b799-3159-4851-8e76-c5f735872a29\") "
Jul 27 17:56:16.631082 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:16.630788    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8b9b799-3159-4851-8e76-c5f735872a29-model-cache\") pod \"a8b9b799-3159-4851-8e76-c5f735872a29\" (UID: \"a8b9b799-3159-4851-8e76-c5f735872a29\") "
Jul 27 17:56:16.631082 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:16.630887    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b9b799-3159-4851-8e76-c5f735872a29-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a8b9b799-3159-4851-8e76-c5f735872a29" (UID: "a8b9b799-3159-4851-8e76-c5f735872a29"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:56:16.631082 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:16.631009    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b9b799-3159-4851-8e76-c5f735872a29-home" (OuterVolumeSpecName: "home") pod "a8b9b799-3159-4851-8e76-c5f735872a29" (UID: "a8b9b799-3159-4851-8e76-c5f735872a29"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:56:16.631288 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:16.631084    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b9b799-3159-4851-8e76-c5f735872a29-model-cache" (OuterVolumeSpecName: "model-cache") pod "a8b9b799-3159-4851-8e76-c5f735872a29" (UID: "a8b9b799-3159-4851-8e76-c5f735872a29"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:56:16.631288 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:16.631125    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8b9b799-3159-4851-8e76-c5f735872a29-tmp-dir\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:56:16.631288 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:16.631140    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8b9b799-3159-4851-8e76-c5f735872a29-home\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:56:16.632970 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:16.632943    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b9b799-3159-4851-8e76-c5f735872a29-dshm" (OuterVolumeSpecName: "dshm") pod "a8b9b799-3159-4851-8e76-c5f735872a29" (UID: "a8b9b799-3159-4851-8e76-c5f735872a29"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:56:16.632970 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:16.632945    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b9b799-3159-4851-8e76-c5f735872a29-kube-api-access-csqvq" (OuterVolumeSpecName: "kube-api-access-csqvq") pod "a8b9b799-3159-4851-8e76-c5f735872a29" (UID: "a8b9b799-3159-4851-8e76-c5f735872a29"). InnerVolumeSpecName "kube-api-access-csqvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:56:16.633108 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:16.632993    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8b9b799-3159-4851-8e76-c5f735872a29-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a8b9b799-3159-4851-8e76-c5f735872a29" (UID: "a8b9b799-3159-4851-8e76-c5f735872a29"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:56:16.731769 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:16.731735    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8b9b799-3159-4851-8e76-c5f735872a29-dshm\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:56:16.731916 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:16.731780    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csqvq\" (UniqueName: \"kubernetes.io/projected/a8b9b799-3159-4851-8e76-c5f735872a29-kube-api-access-csqvq\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:56:16.731916 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:16.731792    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b9b799-3159-4851-8e76-c5f735872a29-tls-certs\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:56:16.731916 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:16.731800    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8b9b799-3159-4851-8e76-c5f735872a29-model-cache\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:56:17.301227 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:17.301192    2557 generic.go:356] "Generic (PLEG): container finished" podID="a8b9b799-3159-4851-8e76-c5f735872a29" containerID="addba91004470a5a3ebfb4b844f0b028926d7140bcff5d835a8382cdaa26a489" exitCode=0
Jul 27 17:56:17.301740 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:17.301237    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg"
Jul 27 17:56:17.301740 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:17.301261    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg" event={"ID":"a8b9b799-3159-4851-8e76-c5f735872a29","Type":"ContainerDied","Data":"addba91004470a5a3ebfb4b844f0b028926d7140bcff5d835a8382cdaa26a489"}
Jul 27 17:56:17.301740 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:17.301301    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg" event={"ID":"a8b9b799-3159-4851-8e76-c5f735872a29","Type":"ContainerDied","Data":"ac978c07ac6394b473567ed249aee2d04a484cf59962127ea70314692d2dced6"}
Jul 27 17:56:17.301740 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:17.301322    2557 scope.go:117] "RemoveContainer" containerID="addba91004470a5a3ebfb4b844f0b028926d7140bcff5d835a8382cdaa26a489"
Jul 27 17:56:17.310515 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:17.310495    2557 scope.go:117] "RemoveContainer" containerID="addba91004470a5a3ebfb4b844f0b028926d7140bcff5d835a8382cdaa26a489"
Jul 27 17:56:17.310821 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:56:17.310790    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"addba91004470a5a3ebfb4b844f0b028926d7140bcff5d835a8382cdaa26a489\": container with ID starting with addba91004470a5a3ebfb4b844f0b028926d7140bcff5d835a8382cdaa26a489 not found: ID does not exist" containerID="addba91004470a5a3ebfb4b844f0b028926d7140bcff5d835a8382cdaa26a489"
Jul 27 17:56:17.310910 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:17.310827    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"addba91004470a5a3ebfb4b844f0b028926d7140bcff5d835a8382cdaa26a489"} err="failed to get container status \"addba91004470a5a3ebfb4b844f0b028926d7140bcff5d835a8382cdaa26a489\": rpc error: code = NotFound desc = could not find container \"addba91004470a5a3ebfb4b844f0b028926d7140bcff5d835a8382cdaa26a489\": container with ID starting with addba91004470a5a3ebfb4b844f0b028926d7140bcff5d835a8382cdaa26a489 not found: ID does not exist"
Jul 27 17:56:17.322976 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:17.322955    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg"]
Jul 27 17:56:17.326113 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:17.326095    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hsbdg"]
Jul 27 17:56:18.118833 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:18.118803    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8b9b799-3159-4851-8e76-c5f735872a29" path="/var/lib/kubelet/pods/a8b9b799-3159-4851-8e76-c5f735872a29/volumes"
Jul 27 17:56:26.096004 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:26.095969    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkr9ds" podUID="38762d94-db85-4889-a64b-c6e65b559dd1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 17:56:31.114641 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:31.114604    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkr9ds" podUID="38762d94-db85-4889-a64b-c6e65b559dd1" containerName="main" containerID="cri-o://7fa4bef8aa8b8d1c3569475db46598ced83ddf82b5acb944543a6a61ae4563d5" gracePeriod=45
Jul 27 17:56:31.362208 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:31.362184    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkr9ds"
Jul 27 17:56:31.366082 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:31.366012    2557 generic.go:356] "Generic (PLEG): container finished" podID="38762d94-db85-4889-a64b-c6e65b559dd1" containerID="7fa4bef8aa8b8d1c3569475db46598ced83ddf82b5acb944543a6a61ae4563d5" exitCode=0
Jul 27 17:56:31.366082 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:31.366047    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkr9ds"
Jul 27 17:56:31.366232 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:31.366078    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkr9ds" event={"ID":"38762d94-db85-4889-a64b-c6e65b559dd1","Type":"ContainerDied","Data":"7fa4bef8aa8b8d1c3569475db46598ced83ddf82b5acb944543a6a61ae4563d5"}
Jul 27 17:56:31.366232 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:31.366124    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkr9ds" event={"ID":"38762d94-db85-4889-a64b-c6e65b559dd1","Type":"ContainerDied","Data":"aa729ee5cc9a9aa1c1879aa6f5d0c16d0faeab932052f0815627b0a4530e93f1"}
Jul 27 17:56:31.366232 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:31.366147    2557 scope.go:117] "RemoveContainer" containerID="7fa4bef8aa8b8d1c3569475db46598ced83ddf82b5acb944543a6a61ae4563d5"
Jul 27 17:56:31.374701 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:31.374685    2557 scope.go:117] "RemoveContainer" containerID="7fa4bef8aa8b8d1c3569475db46598ced83ddf82b5acb944543a6a61ae4563d5"
Jul 27 17:56:31.374966 ip-10-0-131-114 kubenswrapper[2557]: E0727 17:56:31.374949    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fa4bef8aa8b8d1c3569475db46598ced83ddf82b5acb944543a6a61ae4563d5\": container with ID starting with 7fa4bef8aa8b8d1c3569475db46598ced83ddf82b5acb944543a6a61ae4563d5 not found: ID does not exist" containerID="7fa4bef8aa8b8d1c3569475db46598ced83ddf82b5acb944543a6a61ae4563d5"
Jul 27 17:56:31.375052 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:31.374975    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fa4bef8aa8b8d1c3569475db46598ced83ddf82b5acb944543a6a61ae4563d5"} err="failed to get container status \"7fa4bef8aa8b8d1c3569475db46598ced83ddf82b5acb944543a6a61ae4563d5\": rpc error: code = NotFound desc = could not find container \"7fa4bef8aa8b8d1c3569475db46598ced83ddf82b5acb944543a6a61ae4563d5\": container with ID starting with 7fa4bef8aa8b8d1c3569475db46598ced83ddf82b5acb944543a6a61ae4563d5 not found: ID does not exist"
Jul 27 17:56:31.457541 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:31.457510    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38762d94-db85-4889-a64b-c6e65b559dd1-tls-certs\") pod \"38762d94-db85-4889-a64b-c6e65b559dd1\" (UID: \"38762d94-db85-4889-a64b-c6e65b559dd1\") "
Jul 27 17:56:31.457959 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:31.457550    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l78kx\" (UniqueName: \"kubernetes.io/projected/38762d94-db85-4889-a64b-c6e65b559dd1-kube-api-access-l78kx\") pod \"38762d94-db85-4889-a64b-c6e65b559dd1\" (UID: \"38762d94-db85-4889-a64b-c6e65b559dd1\") "
Jul 27 17:56:31.459577 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:31.459556    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38762d94-db85-4889-a64b-c6e65b559dd1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "38762d94-db85-4889-a64b-c6e65b559dd1" (UID: "38762d94-db85-4889-a64b-c6e65b559dd1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:56:31.459669 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:31.459653    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38762d94-db85-4889-a64b-c6e65b559dd1-kube-api-access-l78kx" (OuterVolumeSpecName: "kube-api-access-l78kx") pod "38762d94-db85-4889-a64b-c6e65b559dd1" (UID: "38762d94-db85-4889-a64b-c6e65b559dd1"). InnerVolumeSpecName "kube-api-access-l78kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:56:31.558376 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:31.558352    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38762d94-db85-4889-a64b-c6e65b559dd1-tls-certs\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:56:31.558376 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:31.558374    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l78kx\" (UniqueName: \"kubernetes.io/projected/38762d94-db85-4889-a64b-c6e65b559dd1-kube-api-access-l78kx\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 17:56:31.687834 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:31.687806    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkr9ds"]
Jul 27 17:56:31.693701 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:31.693679    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfkr9ds"]
Jul 27 17:56:32.118959 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:32.118869    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38762d94-db85-4889-a64b-c6e65b559dd1" path="/var/lib/kubelet/pods/38762d94-db85-4889-a64b-c6e65b559dd1/volumes"
Jul 27 17:56:48.195446 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.195414    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj"]
Jul 27 17:56:48.195861 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.195785    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38762d94-db85-4889-a64b-c6e65b559dd1" containerName="main"
Jul 27 17:56:48.195861 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.195797    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="38762d94-db85-4889-a64b-c6e65b559dd1" containerName="main"
Jul 27 17:56:48.195861 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.195816    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8b9b799-3159-4851-8e76-c5f735872a29" containerName="main"
Jul 27 17:56:48.195861 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.195822    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8b9b799-3159-4851-8e76-c5f735872a29" containerName="main"
Jul 27 17:56:48.196000 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.195895    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8b9b799-3159-4851-8e76-c5f735872a29" containerName="main"
Jul 27 17:56:48.196000 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.195905    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="38762d94-db85-4889-a64b-c6e65b559dd1" containerName="main"
Jul 27 17:56:48.198984 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.198969    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj"
Jul 27 17:56:48.201307 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.201286    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"kube-root-ca.crt\""
Jul 27 17:56:48.201885 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.201860    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"openshift-service-ca.crt\""
Jul 27 17:56:48.201885 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.201881    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"autoscale-hpa-deploy-kserve-self-signed-certs\""
Jul 27 17:56:48.202071 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.201901    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"default-dockercfg-xm4b8\""
Jul 27 17:56:48.208188 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.208167    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj"]
Jul 27 17:56:48.300891 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.300857    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb913b33-5cd4-490e-aca7-35d3e56b6a03-tmp-dir\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj\" (UID: \"fb913b33-5cd4-490e-aca7-35d3e56b6a03\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj"
Jul 27 17:56:48.301132 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.300951    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb913b33-5cd4-490e-aca7-35d3e56b6a03-home\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj\" (UID: \"fb913b33-5cd4-490e-aca7-35d3e56b6a03\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj"
Jul 27 17:56:48.301132 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.300972    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb913b33-5cd4-490e-aca7-35d3e56b6a03-model-cache\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj\" (UID: \"fb913b33-5cd4-490e-aca7-35d3e56b6a03\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj"
Jul 27 17:56:48.301132 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.300991    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb913b33-5cd4-490e-aca7-35d3e56b6a03-dshm\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj\" (UID: \"fb913b33-5cd4-490e-aca7-35d3e56b6a03\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj"
Jul 27 17:56:48.301132 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.301021    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb913b33-5cd4-490e-aca7-35d3e56b6a03-tls-certs\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj\" (UID: \"fb913b33-5cd4-490e-aca7-35d3e56b6a03\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj"
Jul 27 17:56:48.301132 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.301098    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4md5\" (UniqueName: \"kubernetes.io/projected/fb913b33-5cd4-490e-aca7-35d3e56b6a03-kube-api-access-s4md5\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj\" (UID: \"fb913b33-5cd4-490e-aca7-35d3e56b6a03\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj"
Jul 27 17:56:48.401659 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.401622    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb913b33-5cd4-490e-aca7-35d3e56b6a03-home\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj\" (UID: \"fb913b33-5cd4-490e-aca7-35d3e56b6a03\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj"
Jul 27 17:56:48.401659 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.401659    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb913b33-5cd4-490e-aca7-35d3e56b6a03-model-cache\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj\" (UID: \"fb913b33-5cd4-490e-aca7-35d3e56b6a03\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj"
Jul 27 17:56:48.401905 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.401677    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb913b33-5cd4-490e-aca7-35d3e56b6a03-dshm\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj\" (UID: \"fb913b33-5cd4-490e-aca7-35d3e56b6a03\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj"
Jul 27 17:56:48.401905 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.401699    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb913b33-5cd4-490e-aca7-35d3e56b6a03-tls-certs\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj\" (UID: \"fb913b33-5cd4-490e-aca7-35d3e56b6a03\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj"
Jul 27 17:56:48.401905 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.401719    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4md5\" (UniqueName: \"kubernetes.io/projected/fb913b33-5cd4-490e-aca7-35d3e56b6a03-kube-api-access-s4md5\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj\" (UID: \"fb913b33-5cd4-490e-aca7-35d3e56b6a03\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj"
Jul 27 17:56:48.401905 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.401784    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb913b33-5cd4-490e-aca7-35d3e56b6a03-tmp-dir\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj\" (UID: \"fb913b33-5cd4-490e-aca7-35d3e56b6a03\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj"
Jul 27 17:56:48.402214 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.402189    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb913b33-5cd4-490e-aca7-35d3e56b6a03-home\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj\" (UID: \"fb913b33-5cd4-490e-aca7-35d3e56b6a03\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj"
Jul 27 17:56:48.402374 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.402229    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb913b33-5cd4-490e-aca7-35d3e56b6a03-model-cache\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj\" (UID: \"fb913b33-5cd4-490e-aca7-35d3e56b6a03\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj"
Jul 27 17:56:48.402374 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.402262    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb913b33-5cd4-490e-aca7-35d3e56b6a03-tmp-dir\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj\" (UID: \"fb913b33-5cd4-490e-aca7-35d3e56b6a03\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj"
Jul 27 17:56:48.404193 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.404167    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb913b33-5cd4-490e-aca7-35d3e56b6a03-dshm\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj\" (UID: \"fb913b33-5cd4-490e-aca7-35d3e56b6a03\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj"
Jul 27 17:56:48.404314 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.404298    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb913b33-5cd4-490e-aca7-35d3e56b6a03-tls-certs\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj\" (UID: \"fb913b33-5cd4-490e-aca7-35d3e56b6a03\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj"
Jul 27 17:56:48.410247 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.410230    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4md5\" (UniqueName: \"kubernetes.io/projected/fb913b33-5cd4-490e-aca7-35d3e56b6a03-kube-api-access-s4md5\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj\" (UID: \"fb913b33-5cd4-490e-aca7-35d3e56b6a03\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj"
Jul 27 17:56:48.510305 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.510223    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj"
Jul 27 17:56:48.644344 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:48.644309    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj"]
Jul 27 17:56:48.647802 ip-10-0-131-114 kubenswrapper[2557]: W0727 17:56:48.647772    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb913b33_5cd4_490e_aca7_35d3e56b6a03.slice/crio-b67f2fdf309072b4c667c1b1a589adfe47c9598a7296caab3424f2f6f375ab16 WatchSource:0}: Error finding container b67f2fdf309072b4c667c1b1a589adfe47c9598a7296caab3424f2f6f375ab16: Status 404 returned error can't find the container with id b67f2fdf309072b4c667c1b1a589adfe47c9598a7296caab3424f2f6f375ab16
Jul 27 17:56:49.449182 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:49.449138    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj" event={"ID":"fb913b33-5cd4-490e-aca7-35d3e56b6a03","Type":"ContainerStarted","Data":"759ed79e41c385c6dcbd43f06088ae245ad0eeff25d9642331f86b9f1fe1abe4"}
Jul 27 17:56:49.449576 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:49.449205    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj" event={"ID":"fb913b33-5cd4-490e-aca7-35d3e56b6a03","Type":"ContainerStarted","Data":"b67f2fdf309072b4c667c1b1a589adfe47c9598a7296caab3424f2f6f375ab16"}
Jul 27 17:56:49.469364 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:49.469315    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj" podStartSLOduration=1.469298102 podStartE2EDuration="1.469298102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:56:49.468300858 +0000 UTC m=+1701.955112158" watchObservedRunningTime="2026-07-27 17:56:49.469298102 +0000 UTC m=+1701.956109388"
Jul 27 17:56:49.511281 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:49.511237    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj"
Jul 27 17:56:58.510989 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:58.510896    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj"
Jul 27 17:56:58.523336 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:58.523311    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj"
Jul 27 17:56:59.506861 ip-10-0-131-114 kubenswrapper[2557]: I0727 17:56:59.506835    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj"
Jul 27 18:00:25.001656 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.001621    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"]
Jul 27 18:00:25.005208 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.005191    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"
Jul 27 18:00:25.011898 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.011716    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-rvs2g\""
Jul 27 18:00:25.012806 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.012780    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Jul 27 18:00:25.013246 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.013083    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Jul 27 18:00:25.013246 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.013126    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Jul 27 18:00:25.013589 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.013565    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Jul 27 18:00:25.027082 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.027055    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"]
Jul 27 18:00:25.036073 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.036047    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd\" (UID: \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"
Jul 27 18:00:25.036217 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.036077    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-model-cache\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd\" (UID: \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"
Jul 27 18:00:25.036217 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.036120    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-home\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd\" (UID: \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"
Jul 27 18:00:25.036217 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.036138    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd\" (UID: \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"
Jul 27 18:00:25.036217 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.036157    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-dshm\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd\" (UID: \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"
Jul 27 18:00:25.036217 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.036185    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-tls-certs\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd\" (UID: \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"
Jul 27 18:00:25.036430 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.036297    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzdtw\" (UniqueName: \"kubernetes.io/projected/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-kube-api-access-rzdtw\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd\" (UID: \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"
Jul 27 18:00:25.137210 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.137173    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzdtw\" (UniqueName: \"kubernetes.io/projected/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-kube-api-access-rzdtw\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd\" (UID: \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"
Jul 27 18:00:25.137410 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.137221    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd\" (UID: \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"
Jul 27 18:00:25.137410 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.137239    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-model-cache\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd\" (UID: \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"
Jul 27 18:00:25.137410 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.137314    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-home\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd\" (UID: \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"
Jul 27 18:00:25.137410 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.137340    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd\" (UID: \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"
Jul 27 18:00:25.137410 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.137366    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-dshm\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd\" (UID: \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"
Jul 27 18:00:25.137669 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.137432    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-tls-certs\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd\" (UID: \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"
Jul 27 18:00:25.137669 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.137637    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd\" (UID: \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"
Jul 27 18:00:25.137814 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.137701    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-model-cache\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd\" (UID: \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"
Jul 27 18:00:25.137885 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.137817    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd\" (UID: \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"
Jul 27 18:00:25.137885 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.137825    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-home\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd\" (UID: \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"
Jul 27 18:00:25.139695 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.139664    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-dshm\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd\" (UID: \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"
Jul 27 18:00:25.140087 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.140066    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-tls-certs\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd\" (UID: \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"
Jul 27 18:00:25.143485 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.143468    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzdtw\" (UniqueName: \"kubernetes.io/projected/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-kube-api-access-rzdtw\") pod \"tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd\" (UID: \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"
Jul 27 18:00:25.317706 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.317617    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"
Jul 27 18:00:25.370828 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.370788    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm"]
Jul 27 18:00:25.376914 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.376720    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm"
Jul 27 18:00:25.384072 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.384019    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm"]
Jul 27 18:00:25.439234 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.439202    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41be4739-3dec-4533-b6f4-197042aab719-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm\" (UID: \"41be4739-3dec-4533-b6f4-197042aab719\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm"
Jul 27 18:00:25.439439 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.439254    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41be4739-3dec-4533-b6f4-197042aab719-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm\" (UID: \"41be4739-3dec-4533-b6f4-197042aab719\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm"
Jul 27 18:00:25.439439 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.439307    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41be4739-3dec-4533-b6f4-197042aab719-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm\" (UID: \"41be4739-3dec-4533-b6f4-197042aab719\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm"
Jul 27 18:00:25.475244 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.475220    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"]
Jul 27 18:00:25.477208 ip-10-0-131-114 kubenswrapper[2557]: W0727 18:00:25.477177    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd34a9fa_a8d2_4e06_8adf_55ef85e210bd.slice/crio-764b725e1fdb7d666f2049e8a7eae2e9dbe0a65209bee00c72b3284b6082c487 WatchSource:0}: Error finding container 764b725e1fdb7d666f2049e8a7eae2e9dbe0a65209bee00c72b3284b6082c487: Status 404 returned error can't find the container with id 764b725e1fdb7d666f2049e8a7eae2e9dbe0a65209bee00c72b3284b6082c487
Jul 27 18:00:25.540062 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.540040    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41be4739-3dec-4533-b6f4-197042aab719-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm\" (UID: \"41be4739-3dec-4533-b6f4-197042aab719\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm"
Jul 27 18:00:25.540172 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.540088    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41be4739-3dec-4533-b6f4-197042aab719-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm\" (UID: \"41be4739-3dec-4533-b6f4-197042aab719\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm"
Jul 27 18:00:25.540172 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.540151    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41be4739-3dec-4533-b6f4-197042aab719-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm\" (UID: \"41be4739-3dec-4533-b6f4-197042aab719\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm"
Jul 27 18:00:25.540454 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.540436    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41be4739-3dec-4533-b6f4-197042aab719-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm\" (UID: \"41be4739-3dec-4533-b6f4-197042aab719\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm"
Jul 27 18:00:25.540511 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.540492    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41be4739-3dec-4533-b6f4-197042aab719-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm\" (UID: \"41be4739-3dec-4533-b6f4-197042aab719\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm"
Jul 27 18:00:25.542296 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.542275    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41be4739-3dec-4533-b6f4-197042aab719-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm\" (UID: \"41be4739-3dec-4533-b6f4-197042aab719\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm"
Jul 27 18:00:25.694663 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.694621    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm"
Jul 27 18:00:25.839115 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:25.839081    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm"]
Jul 27 18:00:25.839237 ip-10-0-131-114 kubenswrapper[2557]: W0727 18:00:25.839186    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41be4739_3dec_4533_b6f4_197042aab719.slice/crio-e349262c502360cdfdf849d0ecfa5657893b9b7c389e969ec618a1b250be259f WatchSource:0}: Error finding container e349262c502360cdfdf849d0ecfa5657893b9b7c389e969ec618a1b250be259f: Status 404 returned error can't find the container with id e349262c502360cdfdf849d0ecfa5657893b9b7c389e969ec618a1b250be259f
Jul 27 18:00:26.446716 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:26.446685    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd" event={"ID":"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd","Type":"ContainerStarted","Data":"6e14baf8399b7eb98bba474c19a280cb450487bff2b7afe38c3e5ed80c389909"}
Jul 27 18:00:26.447188 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:26.446746    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd" event={"ID":"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd","Type":"ContainerStarted","Data":"764b725e1fdb7d666f2049e8a7eae2e9dbe0a65209bee00c72b3284b6082c487"}
Jul 27 18:00:26.448393 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:26.448369    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm" event={"ID":"41be4739-3dec-4533-b6f4-197042aab719","Type":"ContainerStarted","Data":"5816f80c352e2b04b5fe8df56a1ace5f668c37cc6326cc9d7ed420e976dc0214"}
Jul 27 18:00:26.448538 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:26.448407    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm" event={"ID":"41be4739-3dec-4533-b6f4-197042aab719","Type":"ContainerStarted","Data":"e349262c502360cdfdf849d0ecfa5657893b9b7c389e969ec618a1b250be259f"}
Jul 27 18:00:33.490773 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:33.490717    2557 generic.go:356] "Generic (PLEG): container finished" podID="bd34a9fa-a8d2-4e06-8adf-55ef85e210bd" containerID="6e14baf8399b7eb98bba474c19a280cb450487bff2b7afe38c3e5ed80c389909" exitCode=0
Jul 27 18:00:33.491160 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:33.490793    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd" event={"ID":"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd","Type":"ContainerDied","Data":"6e14baf8399b7eb98bba474c19a280cb450487bff2b7afe38c3e5ed80c389909"}
Jul 27 18:00:34.498183 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:34.498144    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd" event={"ID":"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd","Type":"ContainerStarted","Data":"6d38ac86074f7bc7977fe7c49b0f8fbc0cd83f54326665f3e96da170811cd46c"}
Jul 27 18:00:34.517115 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:34.517069    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd" podStartSLOduration=3.380155258 podStartE2EDuration="10.517054351s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.136899093s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:00:34.514627919 +0000 UTC m=+1927.001439206" watchObservedRunningTime="2026-07-27 18:00:34.517054351 +0000 UTC m=+1927.003865654"
Jul 27 18:00:35.318440 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:35.318405    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"
Jul 27 18:00:35.318612 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:35.318487    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"
Jul 27 18:00:35.331130 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:35.331103    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"
Jul 27 18:00:35.514323 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:00:35.514297    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"
Jul 27 18:01:13.681490 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:01:13.681458    2557 generic.go:356] "Generic (PLEG): container finished" podID="41be4739-3dec-4533-b6f4-197042aab719" containerID="5816f80c352e2b04b5fe8df56a1ace5f668c37cc6326cc9d7ed420e976dc0214" exitCode=0
Jul 27 18:01:13.681940 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:01:13.681537    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm" event={"ID":"41be4739-3dec-4533-b6f4-197042aab719","Type":"ContainerDied","Data":"5816f80c352e2b04b5fe8df56a1ace5f668c37cc6326cc9d7ed420e976dc0214"}
Jul 27 18:01:13.682740 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:01:13.682724    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 18:01:52.884671 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:01:52.884585    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm" event={"ID":"41be4739-3dec-4533-b6f4-197042aab719","Type":"ContainerStarted","Data":"ded85c683e5a89b5b6bece932ad81af2b7694db86550491abfb2ea452a5dd241"}
Jul 27 18:01:52.885093 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:01:52.884708    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm"
Jul 27 18:01:52.903498 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:01:52.903451    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm" podStartSLOduration=2.181026149 podStartE2EDuration="1m27.903436619s" totalImagesPullingTime="38.884139868s" totalInitContainerRuntime="46.838270602s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:00:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:01:52.901550507 +0000 UTC m=+2005.388361793" watchObservedRunningTime="2026-07-27 18:01:52.903436619 +0000 UTC m=+2005.390247905"
Jul 27 18:02:23.902823 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:23.902779    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm"
Jul 27 18:02:24.880511 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:24.880480    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm_41be4739-3dec-4533-b6f4-197042aab719/main/0.log"
Jul 27 18:02:25.157345 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.157314    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm"]
Jul 27 18:02:25.157750 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.157561    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm" podUID="41be4739-3dec-4533-b6f4-197042aab719" containerName="main" containerID="cri-o://ded85c683e5a89b5b6bece932ad81af2b7694db86550491abfb2ea452a5dd241" gracePeriod=30
Jul 27 18:02:25.159406 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.159369    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"]
Jul 27 18:02:25.169544 ip-10-0-131-114 kubenswrapper[2557]: E0727 18:02:25.169513    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 18:02:25.169544 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:02:25.169544 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 18:02:25.169544 ip-10-0-131-114 kubenswrapper[2557]:  > containerID="6d38ac86074f7bc7977fe7c49b0f8fbc0cd83f54326665f3e96da170811cd46c" cmd=["/bin/sleep","15"]
Jul 27 18:02:25.169771 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.169566    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd" message="" err=<
Jul 27 18:02:25.169771 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:02:25.169771 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 18:02:25.169771 ip-10-0-131-114 kubenswrapper[2557]:  >
Jul 27 18:02:25.169771 ip-10-0-131-114 kubenswrapper[2557]: E0727 18:02:25.169577    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 18:02:25.169771 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:02:25.169771 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 18:02:25.169771 ip-10-0-131-114 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd" podUID="bd34a9fa-a8d2-4e06-8adf-55ef85e210bd" containerName="main" containerID="cri-o://6d38ac86074f7bc7977fe7c49b0f8fbc0cd83f54326665f3e96da170811cd46c"
Jul 27 18:02:25.169771 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.169607    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd" podUID="bd34a9fa-a8d2-4e06-8adf-55ef85e210bd" containerName="main" containerID="cri-o://6d38ac86074f7bc7977fe7c49b0f8fbc0cd83f54326665f3e96da170811cd46c" gracePeriod=60
Jul 27 18:02:25.581710 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.581687    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"
Jul 27 18:02:25.718805 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.718786    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm"
Jul 27 18:02:25.755348 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.755317    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-kserve-provision-location\") pod \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\" (UID: \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\") "
Jul 27 18:02:25.755513 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.755360    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-dshm\") pod \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\" (UID: \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\") "
Jul 27 18:02:25.755513 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.755395    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-home\") pod \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\" (UID: \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\") "
Jul 27 18:02:25.755513 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.755438    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzdtw\" (UniqueName: \"kubernetes.io/projected/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-kube-api-access-rzdtw\") pod \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\" (UID: \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\") "
Jul 27 18:02:25.755513 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.755478    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-tmp-dir\") pod \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\" (UID: \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\") "
Jul 27 18:02:25.755513 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.755502    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-tls-certs\") pod \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\" (UID: \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\") "
Jul 27 18:02:25.755872 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.755549    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-model-cache\") pod \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\" (UID: \"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd\") "
Jul 27 18:02:25.755872 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.755696    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-home" (OuterVolumeSpecName: "home") pod "bd34a9fa-a8d2-4e06-8adf-55ef85e210bd" (UID: "bd34a9fa-a8d2-4e06-8adf-55ef85e210bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:02:25.755993 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.755972    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "bd34a9fa-a8d2-4e06-8adf-55ef85e210bd" (UID: "bd34a9fa-a8d2-4e06-8adf-55ef85e210bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:02:25.756044 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.755990    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-home\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:02:25.756184 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.756123    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bd34a9fa-a8d2-4e06-8adf-55ef85e210bd" (UID: "bd34a9fa-a8d2-4e06-8adf-55ef85e210bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:02:25.757818 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.757789    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bd34a9fa-a8d2-4e06-8adf-55ef85e210bd" (UID: "bd34a9fa-a8d2-4e06-8adf-55ef85e210bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:02:25.757915 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.757852    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-kube-api-access-rzdtw" (OuterVolumeSpecName: "kube-api-access-rzdtw") pod "bd34a9fa-a8d2-4e06-8adf-55ef85e210bd" (UID: "bd34a9fa-a8d2-4e06-8adf-55ef85e210bd"). InnerVolumeSpecName "kube-api-access-rzdtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:02:25.757915 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.757891    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-dshm" (OuterVolumeSpecName: "dshm") pod "bd34a9fa-a8d2-4e06-8adf-55ef85e210bd" (UID: "bd34a9fa-a8d2-4e06-8adf-55ef85e210bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:02:25.815399 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.815356    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bd34a9fa-a8d2-4e06-8adf-55ef85e210bd" (UID: "bd34a9fa-a8d2-4e06-8adf-55ef85e210bd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:02:25.857250 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.857222    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41be4739-3dec-4533-b6f4-197042aab719-tmp-dir\") pod \"41be4739-3dec-4533-b6f4-197042aab719\" (UID: \"41be4739-3dec-4533-b6f4-197042aab719\") "
Jul 27 18:02:25.857407 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.857317    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41be4739-3dec-4533-b6f4-197042aab719-tls-certs\") pod \"41be4739-3dec-4533-b6f4-197042aab719\" (UID: \"41be4739-3dec-4533-b6f4-197042aab719\") "
Jul 27 18:02:25.857407 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.857356    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41be4739-3dec-4533-b6f4-197042aab719-kserve-provision-location\") pod \"41be4739-3dec-4533-b6f4-197042aab719\" (UID: \"41be4739-3dec-4533-b6f4-197042aab719\") "
Jul 27 18:02:25.857522 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.857509    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-kserve-provision-location\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:02:25.857558 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.857524    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-dshm\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:02:25.857558 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.857534    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzdtw\" (UniqueName: \"kubernetes.io/projected/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-kube-api-access-rzdtw\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:02:25.857558 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.857543    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-tmp-dir\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:02:25.857558 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.857554    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-tls-certs\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:02:25.857682 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.857566    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd-model-cache\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:02:25.857682 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.857579    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41be4739-3dec-4533-b6f4-197042aab719-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "41be4739-3dec-4533-b6f4-197042aab719" (UID: "41be4739-3dec-4533-b6f4-197042aab719"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:02:25.859308 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.859287    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41be4739-3dec-4533-b6f4-197042aab719-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "41be4739-3dec-4533-b6f4-197042aab719" (UID: "41be4739-3dec-4533-b6f4-197042aab719"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:02:25.912910 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.912871    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41be4739-3dec-4533-b6f4-197042aab719-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "41be4739-3dec-4533-b6f4-197042aab719" (UID: "41be4739-3dec-4533-b6f4-197042aab719"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:02:25.958536 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.958452    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41be4739-3dec-4533-b6f4-197042aab719-tls-certs\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:02:25.958536 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.958480    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41be4739-3dec-4533-b6f4-197042aab719-kserve-provision-location\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:02:25.958536 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:25.958490    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41be4739-3dec-4533-b6f4-197042aab719-tmp-dir\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:02:26.058306 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:26.058270    2557 generic.go:356] "Generic (PLEG): container finished" podID="bd34a9fa-a8d2-4e06-8adf-55ef85e210bd" containerID="6d38ac86074f7bc7977fe7c49b0f8fbc0cd83f54326665f3e96da170811cd46c" exitCode=0
Jul 27 18:02:26.058473 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:26.058346    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"
Jul 27 18:02:26.058473 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:26.058346    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd" event={"ID":"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd","Type":"ContainerDied","Data":"6d38ac86074f7bc7977fe7c49b0f8fbc0cd83f54326665f3e96da170811cd46c"}
Jul 27 18:02:26.058473 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:26.058400    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd" event={"ID":"bd34a9fa-a8d2-4e06-8adf-55ef85e210bd","Type":"ContainerDied","Data":"764b725e1fdb7d666f2049e8a7eae2e9dbe0a65209bee00c72b3284b6082c487"}
Jul 27 18:02:26.058473 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:26.058431    2557 scope.go:117] "RemoveContainer" containerID="6d38ac86074f7bc7977fe7c49b0f8fbc0cd83f54326665f3e96da170811cd46c"
Jul 27 18:02:26.060329 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:26.060293    2557 generic.go:356] "Generic (PLEG): container finished" podID="41be4739-3dec-4533-b6f4-197042aab719" containerID="ded85c683e5a89b5b6bece932ad81af2b7694db86550491abfb2ea452a5dd241" exitCode=0
Jul 27 18:02:26.060463 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:26.060338    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm"
Jul 27 18:02:26.060463 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:26.060385    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm" event={"ID":"41be4739-3dec-4533-b6f4-197042aab719","Type":"ContainerDied","Data":"ded85c683e5a89b5b6bece932ad81af2b7694db86550491abfb2ea452a5dd241"}
Jul 27 18:02:26.060463 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:26.060441    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm" event={"ID":"41be4739-3dec-4533-b6f4-197042aab719","Type":"ContainerDied","Data":"e349262c502360cdfdf849d0ecfa5657893b9b7c389e969ec618a1b250be259f"}
Jul 27 18:02:26.069009 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:26.068990    2557 scope.go:117] "RemoveContainer" containerID="6e14baf8399b7eb98bba474c19a280cb450487bff2b7afe38c3e5ed80c389909"
Jul 27 18:02:26.077215 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:26.077198    2557 scope.go:117] "RemoveContainer" containerID="6d38ac86074f7bc7977fe7c49b0f8fbc0cd83f54326665f3e96da170811cd46c"
Jul 27 18:02:26.077459 ip-10-0-131-114 kubenswrapper[2557]: E0727 18:02:26.077440    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d38ac86074f7bc7977fe7c49b0f8fbc0cd83f54326665f3e96da170811cd46c\": container with ID starting with 6d38ac86074f7bc7977fe7c49b0f8fbc0cd83f54326665f3e96da170811cd46c not found: ID does not exist" containerID="6d38ac86074f7bc7977fe7c49b0f8fbc0cd83f54326665f3e96da170811cd46c"
Jul 27 18:02:26.077517 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:26.077466    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d38ac86074f7bc7977fe7c49b0f8fbc0cd83f54326665f3e96da170811cd46c"} err="failed to get container status \"6d38ac86074f7bc7977fe7c49b0f8fbc0cd83f54326665f3e96da170811cd46c\": rpc error: code = NotFound desc = could not find container \"6d38ac86074f7bc7977fe7c49b0f8fbc0cd83f54326665f3e96da170811cd46c\": container with ID starting with 6d38ac86074f7bc7977fe7c49b0f8fbc0cd83f54326665f3e96da170811cd46c not found: ID does not exist"
Jul 27 18:02:26.077517 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:26.077482    2557 scope.go:117] "RemoveContainer" containerID="6e14baf8399b7eb98bba474c19a280cb450487bff2b7afe38c3e5ed80c389909"
Jul 27 18:02:26.077707 ip-10-0-131-114 kubenswrapper[2557]: E0727 18:02:26.077691    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e14baf8399b7eb98bba474c19a280cb450487bff2b7afe38c3e5ed80c389909\": container with ID starting with 6e14baf8399b7eb98bba474c19a280cb450487bff2b7afe38c3e5ed80c389909 not found: ID does not exist" containerID="6e14baf8399b7eb98bba474c19a280cb450487bff2b7afe38c3e5ed80c389909"
Jul 27 18:02:26.077773 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:26.077712    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e14baf8399b7eb98bba474c19a280cb450487bff2b7afe38c3e5ed80c389909"} err="failed to get container status \"6e14baf8399b7eb98bba474c19a280cb450487bff2b7afe38c3e5ed80c389909\": rpc error: code = NotFound desc = could not find container \"6e14baf8399b7eb98bba474c19a280cb450487bff2b7afe38c3e5ed80c389909\": container with ID starting with 6e14baf8399b7eb98bba474c19a280cb450487bff2b7afe38c3e5ed80c389909 not found: ID does not exist"
Jul 27 18:02:26.077773 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:26.077726    2557 scope.go:117] "RemoveContainer" containerID="ded85c683e5a89b5b6bece932ad81af2b7694db86550491abfb2ea452a5dd241"
Jul 27 18:02:26.080746 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:26.080723    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"]
Jul 27 18:02:26.086546 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:26.086530    2557 scope.go:117] "RemoveContainer" containerID="5816f80c352e2b04b5fe8df56a1ace5f668c37cc6326cc9d7ed420e976dc0214"
Jul 27 18:02:26.087144 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:26.087123    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd"]
Jul 27 18:02:26.094103 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:26.094076    2557 scope.go:117] "RemoveContainer" containerID="ded85c683e5a89b5b6bece932ad81af2b7694db86550491abfb2ea452a5dd241"
Jul 27 18:02:26.094369 ip-10-0-131-114 kubenswrapper[2557]: E0727 18:02:26.094343    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ded85c683e5a89b5b6bece932ad81af2b7694db86550491abfb2ea452a5dd241\": container with ID starting with ded85c683e5a89b5b6bece932ad81af2b7694db86550491abfb2ea452a5dd241 not found: ID does not exist" containerID="ded85c683e5a89b5b6bece932ad81af2b7694db86550491abfb2ea452a5dd241"
Jul 27 18:02:26.094434 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:26.094378    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ded85c683e5a89b5b6bece932ad81af2b7694db86550491abfb2ea452a5dd241"} err="failed to get container status \"ded85c683e5a89b5b6bece932ad81af2b7694db86550491abfb2ea452a5dd241\": rpc error: code = NotFound desc = could not find container \"ded85c683e5a89b5b6bece932ad81af2b7694db86550491abfb2ea452a5dd241\": container with ID starting with ded85c683e5a89b5b6bece932ad81af2b7694db86550491abfb2ea452a5dd241 not found: ID does not exist"
Jul 27 18:02:26.094434 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:26.094401    2557 scope.go:117] "RemoveContainer" containerID="5816f80c352e2b04b5fe8df56a1ace5f668c37cc6326cc9d7ed420e976dc0214"
Jul 27 18:02:26.094684 ip-10-0-131-114 kubenswrapper[2557]: E0727 18:02:26.094666    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5816f80c352e2b04b5fe8df56a1ace5f668c37cc6326cc9d7ed420e976dc0214\": container with ID starting with 5816f80c352e2b04b5fe8df56a1ace5f668c37cc6326cc9d7ed420e976dc0214 not found: ID does not exist" containerID="5816f80c352e2b04b5fe8df56a1ace5f668c37cc6326cc9d7ed420e976dc0214"
Jul 27 18:02:26.094744 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:26.094690    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5816f80c352e2b04b5fe8df56a1ace5f668c37cc6326cc9d7ed420e976dc0214"} err="failed to get container status \"5816f80c352e2b04b5fe8df56a1ace5f668c37cc6326cc9d7ed420e976dc0214\": rpc error: code = NotFound desc = could not find container \"5816f80c352e2b04b5fe8df56a1ace5f668c37cc6326cc9d7ed420e976dc0214\": container with ID starting with 5816f80c352e2b04b5fe8df56a1ace5f668c37cc6326cc9d7ed420e976dc0214 not found: ID does not exist"
Jul 27 18:02:26.098227 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:26.098207    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm"]
Jul 27 18:02:26.102073 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:26.102054    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-7bf4c45ddc-ptbxm"]
Jul 27 18:02:26.117928 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:26.117908    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41be4739-3dec-4533-b6f4-197042aab719" path="/var/lib/kubelet/pods/41be4739-3dec-4533-b6f4-197042aab719/volumes"
Jul 27 18:02:26.118341 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:26.118328    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd34a9fa-a8d2-4e06-8adf-55ef85e210bd" path="/var/lib/kubelet/pods/bd34a9fa-a8d2-4e06-8adf-55ef85e210bd/volumes"
Jul 27 18:02:26.503813 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:02:26.503772    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-66b9cd7b4b-6xgbd" podUID="bd34a9fa-a8d2-4e06-8adf-55ef85e210bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 27 18:11:29.756549 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:29.756516    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj"]
Jul 27 18:11:29.766454 ip-10-0-131-114 kubenswrapper[2557]: E0727 18:11:29.765851    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 18:11:29.766454 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:11:29.766454 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 18:11:29.766454 ip-10-0-131-114 kubenswrapper[2557]:  > containerID="759ed79e41c385c6dcbd43f06088ae245ad0eeff25d9642331f86b9f1fe1abe4" cmd=["/bin/sleep","15"]
Jul 27 18:11:29.766454 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:29.765913    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj" message="" err=<
Jul 27 18:11:29.766454 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:11:29.766454 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 18:11:29.766454 ip-10-0-131-114 kubenswrapper[2557]:  >
Jul 27 18:11:29.766454 ip-10-0-131-114 kubenswrapper[2557]: E0727 18:11:29.765929    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 18:11:29.766454 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:11:29.766454 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 18:11:29.766454 ip-10-0-131-114 kubenswrapper[2557]:  > pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj" podUID="fb913b33-5cd4-490e-aca7-35d3e56b6a03" containerName="main" containerID="cri-o://759ed79e41c385c6dcbd43f06088ae245ad0eeff25d9642331f86b9f1fe1abe4"
Jul 27 18:11:29.766454 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:29.765980    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj" podUID="fb913b33-5cd4-490e-aca7-35d3e56b6a03" containerName="main" containerID="cri-o://759ed79e41c385c6dcbd43f06088ae245ad0eeff25d9642331f86b9f1fe1abe4" gracePeriod=60
Jul 27 18:11:30.524972 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:30.524952    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj"
Jul 27 18:11:30.537922 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:30.537899    2557 generic.go:356] "Generic (PLEG): container finished" podID="fb913b33-5cd4-490e-aca7-35d3e56b6a03" containerID="759ed79e41c385c6dcbd43f06088ae245ad0eeff25d9642331f86b9f1fe1abe4" exitCode=0
Jul 27 18:11:30.538022 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:30.537938    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj" event={"ID":"fb913b33-5cd4-490e-aca7-35d3e56b6a03","Type":"ContainerDied","Data":"759ed79e41c385c6dcbd43f06088ae245ad0eeff25d9642331f86b9f1fe1abe4"}
Jul 27 18:11:30.538022 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:30.537946    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj"
Jul 27 18:11:30.538022 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:30.537980    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj" event={"ID":"fb913b33-5cd4-490e-aca7-35d3e56b6a03","Type":"ContainerDied","Data":"b67f2fdf309072b4c667c1b1a589adfe47c9598a7296caab3424f2f6f375ab16"}
Jul 27 18:11:30.538022 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:30.538004    2557 scope.go:117] "RemoveContainer" containerID="759ed79e41c385c6dcbd43f06088ae245ad0eeff25d9642331f86b9f1fe1abe4"
Jul 27 18:11:30.547019 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:30.547003    2557 scope.go:117] "RemoveContainer" containerID="759ed79e41c385c6dcbd43f06088ae245ad0eeff25d9642331f86b9f1fe1abe4"
Jul 27 18:11:30.547319 ip-10-0-131-114 kubenswrapper[2557]: E0727 18:11:30.547301    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"759ed79e41c385c6dcbd43f06088ae245ad0eeff25d9642331f86b9f1fe1abe4\": container with ID starting with 759ed79e41c385c6dcbd43f06088ae245ad0eeff25d9642331f86b9f1fe1abe4 not found: ID does not exist" containerID="759ed79e41c385c6dcbd43f06088ae245ad0eeff25d9642331f86b9f1fe1abe4"
Jul 27 18:11:30.547401 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:30.547327    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"759ed79e41c385c6dcbd43f06088ae245ad0eeff25d9642331f86b9f1fe1abe4"} err="failed to get container status \"759ed79e41c385c6dcbd43f06088ae245ad0eeff25d9642331f86b9f1fe1abe4\": rpc error: code = NotFound desc = could not find container \"759ed79e41c385c6dcbd43f06088ae245ad0eeff25d9642331f86b9f1fe1abe4\": container with ID starting with 759ed79e41c385c6dcbd43f06088ae245ad0eeff25d9642331f86b9f1fe1abe4 not found: ID does not exist"
Jul 27 18:11:30.655441 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:30.655353    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb913b33-5cd4-490e-aca7-35d3e56b6a03-tmp-dir\") pod \"fb913b33-5cd4-490e-aca7-35d3e56b6a03\" (UID: \"fb913b33-5cd4-490e-aca7-35d3e56b6a03\") "
Jul 27 18:11:30.655441 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:30.655393    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb913b33-5cd4-490e-aca7-35d3e56b6a03-model-cache\") pod \"fb913b33-5cd4-490e-aca7-35d3e56b6a03\" (UID: \"fb913b33-5cd4-490e-aca7-35d3e56b6a03\") "
Jul 27 18:11:30.655441 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:30.655428    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb913b33-5cd4-490e-aca7-35d3e56b6a03-tls-certs\") pod \"fb913b33-5cd4-490e-aca7-35d3e56b6a03\" (UID: \"fb913b33-5cd4-490e-aca7-35d3e56b6a03\") "
Jul 27 18:11:30.655701 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:30.655474    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb913b33-5cd4-490e-aca7-35d3e56b6a03-dshm\") pod \"fb913b33-5cd4-490e-aca7-35d3e56b6a03\" (UID: \"fb913b33-5cd4-490e-aca7-35d3e56b6a03\") "
Jul 27 18:11:30.655701 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:30.655498    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4md5\" (UniqueName: \"kubernetes.io/projected/fb913b33-5cd4-490e-aca7-35d3e56b6a03-kube-api-access-s4md5\") pod \"fb913b33-5cd4-490e-aca7-35d3e56b6a03\" (UID: \"fb913b33-5cd4-490e-aca7-35d3e56b6a03\") "
Jul 27 18:11:30.655701 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:30.655577    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb913b33-5cd4-490e-aca7-35d3e56b6a03-home\") pod \"fb913b33-5cd4-490e-aca7-35d3e56b6a03\" (UID: \"fb913b33-5cd4-490e-aca7-35d3e56b6a03\") "
Jul 27 18:11:30.655701 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:30.655676    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb913b33-5cd4-490e-aca7-35d3e56b6a03-model-cache" (OuterVolumeSpecName: "model-cache") pod "fb913b33-5cd4-490e-aca7-35d3e56b6a03" (UID: "fb913b33-5cd4-490e-aca7-35d3e56b6a03"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:11:30.655930 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:30.655692    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb913b33-5cd4-490e-aca7-35d3e56b6a03-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fb913b33-5cd4-490e-aca7-35d3e56b6a03" (UID: "fb913b33-5cd4-490e-aca7-35d3e56b6a03"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:11:30.655930 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:30.655877    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb913b33-5cd4-490e-aca7-35d3e56b6a03-home" (OuterVolumeSpecName: "home") pod "fb913b33-5cd4-490e-aca7-35d3e56b6a03" (UID: "fb913b33-5cd4-490e-aca7-35d3e56b6a03"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:11:30.655930 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:30.655929    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb913b33-5cd4-490e-aca7-35d3e56b6a03-home\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:11:30.656071 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:30.655941    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb913b33-5cd4-490e-aca7-35d3e56b6a03-tmp-dir\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:11:30.656071 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:30.655952    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb913b33-5cd4-490e-aca7-35d3e56b6a03-model-cache\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:11:30.657791 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:30.657748    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb913b33-5cd4-490e-aca7-35d3e56b6a03-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fb913b33-5cd4-490e-aca7-35d3e56b6a03" (UID: "fb913b33-5cd4-490e-aca7-35d3e56b6a03"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:11:30.657904 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:30.657824    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb913b33-5cd4-490e-aca7-35d3e56b6a03-dshm" (OuterVolumeSpecName: "dshm") pod "fb913b33-5cd4-490e-aca7-35d3e56b6a03" (UID: "fb913b33-5cd4-490e-aca7-35d3e56b6a03"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:11:30.657904 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:30.657848    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb913b33-5cd4-490e-aca7-35d3e56b6a03-kube-api-access-s4md5" (OuterVolumeSpecName: "kube-api-access-s4md5") pod "fb913b33-5cd4-490e-aca7-35d3e56b6a03" (UID: "fb913b33-5cd4-490e-aca7-35d3e56b6a03"). InnerVolumeSpecName "kube-api-access-s4md5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:11:30.756319 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:30.756290    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb913b33-5cd4-490e-aca7-35d3e56b6a03-tls-certs\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:11:30.756319 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:30.756317    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb913b33-5cd4-490e-aca7-35d3e56b6a03-dshm\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:11:30.756498 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:30.756331    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4md5\" (UniqueName: \"kubernetes.io/projected/fb913b33-5cd4-490e-aca7-35d3e56b6a03-kube-api-access-s4md5\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:11:30.858840 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:30.858813    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj"]
Jul 27 18:11:30.863862 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:30.863840    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj"]
Jul 27 18:11:31.496014 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:31.495977    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-zdbfj" podUID="fb913b33-5cd4-490e-aca7-35d3e56b6a03" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": context deadline exceeded"
Jul 27 18:11:32.118568 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:11:32.118536    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb913b33-5cd4-490e-aca7-35d3e56b6a03" path="/var/lib/kubelet/pods/fb913b33-5cd4-490e-aca7-35d3e56b6a03/volumes"
Jul 27 18:12:05.095645 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:12:05.095537    2557 prober.go:120] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-114.ec2.internal" podUID="14088cbd064fda87605e72bf989bc1ec" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 27 18:12:08.717434 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:12:08.717398    2557 generic.go:356] "Generic (PLEG): container finished" podID="3882525b-8726-46b2-b717-9736e21fd723" containerID="2cf04cbff5ae8f223137509bee405ef7baf8777ac33707517b7121761fa9fe3d" exitCode=0
Jul 27 18:12:08.717854 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:12:08.717472    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-5z76z" event={"ID":"3882525b-8726-46b2-b717-9736e21fd723","Type":"ContainerDied","Data":"2cf04cbff5ae8f223137509bee405ef7baf8777ac33707517b7121761fa9fe3d"}
Jul 27 18:12:08.717901 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:12:08.717856    2557 scope.go:117] "RemoveContainer" containerID="2cf04cbff5ae8f223137509bee405ef7baf8777ac33707517b7121761fa9fe3d"
Jul 27 18:12:09.607789 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:12:09.607743    2557 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-5z76z"
Jul 27 18:12:09.723616 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:12:09.723581    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-5z76z" event={"ID":"3882525b-8726-46b2-b717-9736e21fd723","Type":"ContainerStarted","Data":"726f787abe5bd6ae4b0f9a316bdb7e098dae668c85d566e57079d449c2134927"}
Jul 27 18:12:09.724033 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:12:09.723648    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-5z76z"
Jul 27 18:12:20.731225 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:12:20.731197    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-5z76z"
Jul 27 18:18:41.604474 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.604438    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"]
Jul 27 18:18:41.607045 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.604796    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd34a9fa-a8d2-4e06-8adf-55ef85e210bd" containerName="main"
Jul 27 18:18:41.607045 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.604807    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd34a9fa-a8d2-4e06-8adf-55ef85e210bd" containerName="main"
Jul 27 18:18:41.607045 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.604816    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb913b33-5cd4-490e-aca7-35d3e56b6a03" containerName="main"
Jul 27 18:18:41.607045 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.604821    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb913b33-5cd4-490e-aca7-35d3e56b6a03" containerName="main"
Jul 27 18:18:41.607045 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.604828    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41be4739-3dec-4533-b6f4-197042aab719" containerName="main"
Jul 27 18:18:41.607045 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.604834    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="41be4739-3dec-4533-b6f4-197042aab719" containerName="main"
Jul 27 18:18:41.607045 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.604850    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41be4739-3dec-4533-b6f4-197042aab719" containerName="storage-initializer"
Jul 27 18:18:41.607045 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.604855    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="41be4739-3dec-4533-b6f4-197042aab719" containerName="storage-initializer"
Jul 27 18:18:41.607045 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.604862    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd34a9fa-a8d2-4e06-8adf-55ef85e210bd" containerName="storage-initializer"
Jul 27 18:18:41.607045 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.604868    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd34a9fa-a8d2-4e06-8adf-55ef85e210bd" containerName="storage-initializer"
Jul 27 18:18:41.607045 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.604923    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd34a9fa-a8d2-4e06-8adf-55ef85e210bd" containerName="main"
Jul 27 18:18:41.607045 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.604929    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="41be4739-3dec-4533-b6f4-197042aab719" containerName="main"
Jul 27 18:18:41.607045 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.604935    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb913b33-5cd4-490e-aca7-35d3e56b6a03" containerName="main"
Jul 27 18:18:41.608064 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.608046    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"
Jul 27 18:18:41.610386 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.610366    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Jul 27 18:18:41.610386 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.610379    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Jul 27 18:18:41.610675 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.610658    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-44gpj\""
Jul 27 18:18:41.612305 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.612289    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Jul 27 18:18:41.612485 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.612471    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Jul 27 18:18:41.636389 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.636364    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"]
Jul 27 18:18:41.723141 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.723110    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6906b36-1c83-4dd6-8be9-05c4422921fd-dshm\") pod \"tokenizer-migration-path-test-kserve-569c688787-9p66s\" (UID: \"a6906b36-1c83-4dd6-8be9-05c4422921fd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"
Jul 27 18:18:41.723286 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.723156    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6906b36-1c83-4dd6-8be9-05c4422921fd-tls-certs\") pod \"tokenizer-migration-path-test-kserve-569c688787-9p66s\" (UID: \"a6906b36-1c83-4dd6-8be9-05c4422921fd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"
Jul 27 18:18:41.723286 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.723176    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6906b36-1c83-4dd6-8be9-05c4422921fd-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-569c688787-9p66s\" (UID: \"a6906b36-1c83-4dd6-8be9-05c4422921fd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"
Jul 27 18:18:41.723286 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.723208    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6906b36-1c83-4dd6-8be9-05c4422921fd-model-cache\") pod \"tokenizer-migration-path-test-kserve-569c688787-9p66s\" (UID: \"a6906b36-1c83-4dd6-8be9-05c4422921fd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"
Jul 27 18:18:41.723286 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.723227    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t9qd\" (UniqueName: \"kubernetes.io/projected/a6906b36-1c83-4dd6-8be9-05c4422921fd-kube-api-access-8t9qd\") pod \"tokenizer-migration-path-test-kserve-569c688787-9p66s\" (UID: \"a6906b36-1c83-4dd6-8be9-05c4422921fd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"
Jul 27 18:18:41.723286 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.723284    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6906b36-1c83-4dd6-8be9-05c4422921fd-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-569c688787-9p66s\" (UID: \"a6906b36-1c83-4dd6-8be9-05c4422921fd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"
Jul 27 18:18:41.723462 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.723316    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6906b36-1c83-4dd6-8be9-05c4422921fd-home\") pod \"tokenizer-migration-path-test-kserve-569c688787-9p66s\" (UID: \"a6906b36-1c83-4dd6-8be9-05c4422921fd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"
Jul 27 18:18:41.824286 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.824258    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6906b36-1c83-4dd6-8be9-05c4422921fd-home\") pod \"tokenizer-migration-path-test-kserve-569c688787-9p66s\" (UID: \"a6906b36-1c83-4dd6-8be9-05c4422921fd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"
Jul 27 18:18:41.824286 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.824286    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6906b36-1c83-4dd6-8be9-05c4422921fd-dshm\") pod \"tokenizer-migration-path-test-kserve-569c688787-9p66s\" (UID: \"a6906b36-1c83-4dd6-8be9-05c4422921fd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"
Jul 27 18:18:41.824483 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.824316    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6906b36-1c83-4dd6-8be9-05c4422921fd-tls-certs\") pod \"tokenizer-migration-path-test-kserve-569c688787-9p66s\" (UID: \"a6906b36-1c83-4dd6-8be9-05c4422921fd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"
Jul 27 18:18:41.824483 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.824334    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6906b36-1c83-4dd6-8be9-05c4422921fd-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-569c688787-9p66s\" (UID: \"a6906b36-1c83-4dd6-8be9-05c4422921fd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"
Jul 27 18:18:41.824483 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.824356    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6906b36-1c83-4dd6-8be9-05c4422921fd-model-cache\") pod \"tokenizer-migration-path-test-kserve-569c688787-9p66s\" (UID: \"a6906b36-1c83-4dd6-8be9-05c4422921fd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"
Jul 27 18:18:41.824483 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.824454    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t9qd\" (UniqueName: \"kubernetes.io/projected/a6906b36-1c83-4dd6-8be9-05c4422921fd-kube-api-access-8t9qd\") pod \"tokenizer-migration-path-test-kserve-569c688787-9p66s\" (UID: \"a6906b36-1c83-4dd6-8be9-05c4422921fd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"
Jul 27 18:18:41.824701 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.824543    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6906b36-1c83-4dd6-8be9-05c4422921fd-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-569c688787-9p66s\" (UID: \"a6906b36-1c83-4dd6-8be9-05c4422921fd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"
Jul 27 18:18:41.824701 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.824668    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6906b36-1c83-4dd6-8be9-05c4422921fd-home\") pod \"tokenizer-migration-path-test-kserve-569c688787-9p66s\" (UID: \"a6906b36-1c83-4dd6-8be9-05c4422921fd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"
Jul 27 18:18:41.824837 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.824709    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6906b36-1c83-4dd6-8be9-05c4422921fd-model-cache\") pod \"tokenizer-migration-path-test-kserve-569c688787-9p66s\" (UID: \"a6906b36-1c83-4dd6-8be9-05c4422921fd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"
Jul 27 18:18:41.824837 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.824773    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6906b36-1c83-4dd6-8be9-05c4422921fd-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-569c688787-9p66s\" (UID: \"a6906b36-1c83-4dd6-8be9-05c4422921fd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"
Jul 27 18:18:41.824905 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.824885    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6906b36-1c83-4dd6-8be9-05c4422921fd-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-569c688787-9p66s\" (UID: \"a6906b36-1c83-4dd6-8be9-05c4422921fd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"
Jul 27 18:18:41.826641 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.826615    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6906b36-1c83-4dd6-8be9-05c4422921fd-dshm\") pod \"tokenizer-migration-path-test-kserve-569c688787-9p66s\" (UID: \"a6906b36-1c83-4dd6-8be9-05c4422921fd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"
Jul 27 18:18:41.827060 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.827038    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6906b36-1c83-4dd6-8be9-05c4422921fd-tls-certs\") pod \"tokenizer-migration-path-test-kserve-569c688787-9p66s\" (UID: \"a6906b36-1c83-4dd6-8be9-05c4422921fd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"
Jul 27 18:18:41.836587 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.836568    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t9qd\" (UniqueName: \"kubernetes.io/projected/a6906b36-1c83-4dd6-8be9-05c4422921fd-kube-api-access-8t9qd\") pod \"tokenizer-migration-path-test-kserve-569c688787-9p66s\" (UID: \"a6906b36-1c83-4dd6-8be9-05c4422921fd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"
Jul 27 18:18:41.914701 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.914676    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g"]
Jul 27 18:18:41.918642 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.918615    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g"
Jul 27 18:18:41.918785 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.918694    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"
Jul 27 18:18:41.939971 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:41.939904    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g"]
Jul 27 18:18:42.026926 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:42.026895    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7650e20-fd5f-4bfa-9028-3465aa597121-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g\" (UID: \"b7650e20-fd5f-4bfa-9028-3465aa597121\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g"
Jul 27 18:18:42.027100 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:42.026959    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7650e20-fd5f-4bfa-9028-3465aa597121-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g\" (UID: \"b7650e20-fd5f-4bfa-9028-3465aa597121\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g"
Jul 27 18:18:42.027100 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:42.027050    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7650e20-fd5f-4bfa-9028-3465aa597121-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g\" (UID: \"b7650e20-fd5f-4bfa-9028-3465aa597121\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g"
Jul 27 18:18:42.064238 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:42.064214    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"]
Jul 27 18:18:42.066639 ip-10-0-131-114 kubenswrapper[2557]: W0727 18:18:42.066612    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6906b36_1c83_4dd6_8be9_05c4422921fd.slice/crio-6266bd9bb82f670827b8e42c93b52dc75b93563ee9b3375849c42517b54e5342 WatchSource:0}: Error finding container 6266bd9bb82f670827b8e42c93b52dc75b93563ee9b3375849c42517b54e5342: Status 404 returned error can't find the container with id 6266bd9bb82f670827b8e42c93b52dc75b93563ee9b3375849c42517b54e5342
Jul 27 18:18:42.127627 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:42.127600    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7650e20-fd5f-4bfa-9028-3465aa597121-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g\" (UID: \"b7650e20-fd5f-4bfa-9028-3465aa597121\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g"
Jul 27 18:18:42.127720 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:42.127649    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7650e20-fd5f-4bfa-9028-3465aa597121-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g\" (UID: \"b7650e20-fd5f-4bfa-9028-3465aa597121\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g"
Jul 27 18:18:42.127720 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:42.127703    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7650e20-fd5f-4bfa-9028-3465aa597121-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g\" (UID: \"b7650e20-fd5f-4bfa-9028-3465aa597121\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g"
Jul 27 18:18:42.128052 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:42.128035    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7650e20-fd5f-4bfa-9028-3465aa597121-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g\" (UID: \"b7650e20-fd5f-4bfa-9028-3465aa597121\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g"
Jul 27 18:18:42.128096 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:42.128033    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7650e20-fd5f-4bfa-9028-3465aa597121-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g\" (UID: \"b7650e20-fd5f-4bfa-9028-3465aa597121\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g"
Jul 27 18:18:42.129923 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:42.129907    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7650e20-fd5f-4bfa-9028-3465aa597121-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g\" (UID: \"b7650e20-fd5f-4bfa-9028-3465aa597121\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g"
Jul 27 18:18:42.230631 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:42.230553    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g"
Jul 27 18:18:42.381169 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:42.381141    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g"]
Jul 27 18:18:42.383466 ip-10-0-131-114 kubenswrapper[2557]: W0727 18:18:42.383427    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7650e20_fd5f_4bfa_9028_3465aa597121.slice/crio-b53928919d13815c391d76551f7608aabee994ae634d68f916457c1c864e7d0d WatchSource:0}: Error finding container b53928919d13815c391d76551f7608aabee994ae634d68f916457c1c864e7d0d: Status 404 returned error can't find the container with id b53928919d13815c391d76551f7608aabee994ae634d68f916457c1c864e7d0d
Jul 27 18:18:42.498992 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:42.498891    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s" event={"ID":"a6906b36-1c83-4dd6-8be9-05c4422921fd","Type":"ContainerStarted","Data":"80aa6a528cad483bc0c2a7d23c92361a5b5cb48758ad29dd220174ccb2ff646f"}
Jul 27 18:18:42.498992 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:42.498964    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s" event={"ID":"a6906b36-1c83-4dd6-8be9-05c4422921fd","Type":"ContainerStarted","Data":"6266bd9bb82f670827b8e42c93b52dc75b93563ee9b3375849c42517b54e5342"}
Jul 27 18:18:42.501159 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:42.501134    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g" event={"ID":"b7650e20-fd5f-4bfa-9028-3465aa597121","Type":"ContainerStarted","Data":"1971ec239b5046548461ff1e1dbfe2ecac012e7a0b425f8d9ad989f2c1580bac"}
Jul 27 18:18:42.501292 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:42.501203    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g" event={"ID":"b7650e20-fd5f-4bfa-9028-3465aa597121","Type":"ContainerStarted","Data":"b53928919d13815c391d76551f7608aabee994ae634d68f916457c1c864e7d0d"}
Jul 27 18:18:50.541433 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:50.541388    2557 generic.go:356] "Generic (PLEG): container finished" podID="a6906b36-1c83-4dd6-8be9-05c4422921fd" containerID="80aa6a528cad483bc0c2a7d23c92361a5b5cb48758ad29dd220174ccb2ff646f" exitCode=0
Jul 27 18:18:50.541951 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:50.541446    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s" event={"ID":"a6906b36-1c83-4dd6-8be9-05c4422921fd","Type":"ContainerDied","Data":"80aa6a528cad483bc0c2a7d23c92361a5b5cb48758ad29dd220174ccb2ff646f"}
Jul 27 18:18:51.548094 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:51.548053    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s" event={"ID":"a6906b36-1c83-4dd6-8be9-05c4422921fd","Type":"ContainerStarted","Data":"cff665589ef86ac4277764afea151bd8a14b99973a05878a7102de4555f94c48"}
Jul 27 18:18:51.567337 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:51.567288    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s" podStartSLOduration=2.785570948 podStartE2EDuration="10.567272509s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.781701561s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:18:51.566155435 +0000 UTC m=+3024.052966748" watchObservedRunningTime="2026-07-27 18:18:51.567272509 +0000 UTC m=+3024.054083796"
Jul 27 18:18:51.919837 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:51.919809    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"
Jul 27 18:18:51.920008 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:51.919862    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"
Jul 27 18:18:51.932992 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:51.932964    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"
Jul 27 18:18:52.564823 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:52.564798    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"
Jul 27 18:18:53.560181 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:53.560141    2557 generic.go:356] "Generic (PLEG): container finished" podID="b7650e20-fd5f-4bfa-9028-3465aa597121" containerID="1971ec239b5046548461ff1e1dbfe2ecac012e7a0b425f8d9ad989f2c1580bac" exitCode=0
Jul 27 18:18:53.560378 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:53.560213    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g" event={"ID":"b7650e20-fd5f-4bfa-9028-3465aa597121","Type":"ContainerDied","Data":"1971ec239b5046548461ff1e1dbfe2ecac012e7a0b425f8d9ad989f2c1580bac"}
Jul 27 18:18:54.567220 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:54.567182    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g" event={"ID":"b7650e20-fd5f-4bfa-9028-3465aa597121","Type":"ContainerStarted","Data":"84cceb8f0db56aec208f50201c4e08acd1a614bfd3d0ec80c28a759af1f52e2f"}
Jul 27 18:18:54.567724 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:54.567491    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g"
Jul 27 18:18:54.583033 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:18:54.582951    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g" podStartSLOduration=3.0228827 podStartE2EDuration="13.582932602s" totalImagesPullingTime="0s" totalInitContainerRuntime="10.560049902s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:18:54.582499853 +0000 UTC m=+3027.069311213" watchObservedRunningTime="2026-07-27 18:18:54.582932602 +0000 UTC m=+3027.069743890"
Jul 27 18:19:25.583162 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:25.583085    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g"
Jul 27 18:19:27.301576 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:27.301538    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g_b7650e20-fd5f-4bfa-9028-3465aa597121/main/0.log"
Jul 27 18:19:27.481284 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:27.481250    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g"]
Jul 27 18:19:27.481489 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:27.481462    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g" podUID="b7650e20-fd5f-4bfa-9028-3465aa597121" containerName="main" containerID="cri-o://84cceb8f0db56aec208f50201c4e08acd1a614bfd3d0ec80c28a759af1f52e2f" gracePeriod=30
Jul 27 18:19:27.489125 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:27.489099    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"]
Jul 27 18:19:27.499600 ip-10-0-131-114 kubenswrapper[2557]: E0727 18:19:27.499566    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 18:19:27.499600 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:19:27.499600 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 18:19:27.499600 ip-10-0-131-114 kubenswrapper[2557]:  > containerID="cff665589ef86ac4277764afea151bd8a14b99973a05878a7102de4555f94c48" cmd=["/bin/sleep","15"]
Jul 27 18:19:27.499884 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:27.499618    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s" message="" err=<
Jul 27 18:19:27.499884 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:19:27.499884 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 18:19:27.499884 ip-10-0-131-114 kubenswrapper[2557]:  >
Jul 27 18:19:27.499884 ip-10-0-131-114 kubenswrapper[2557]: E0727 18:19:27.499631    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 18:19:27.499884 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:19:27.499884 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 18:19:27.499884 ip-10-0-131-114 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s" podUID="a6906b36-1c83-4dd6-8be9-05c4422921fd" containerName="main" containerID="cri-o://cff665589ef86ac4277764afea151bd8a14b99973a05878a7102de4555f94c48"
Jul 27 18:19:27.499884 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:27.499666    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s" podUID="a6906b36-1c83-4dd6-8be9-05c4422921fd" containerName="main" containerID="cri-o://cff665589ef86ac4277764afea151bd8a14b99973a05878a7102de4555f94c48" gracePeriod=60
Jul 27 18:19:27.553455 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:27.553380    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s" podUID="a6906b36-1c83-4dd6-8be9-05c4422921fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Jul 27 18:19:27.730056 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:27.730021    2557 generic.go:356] "Generic (PLEG): container finished" podID="a6906b36-1c83-4dd6-8be9-05c4422921fd" containerID="cff665589ef86ac4277764afea151bd8a14b99973a05878a7102de4555f94c48" exitCode=0
Jul 27 18:19:27.730210 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:27.730103    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s" event={"ID":"a6906b36-1c83-4dd6-8be9-05c4422921fd","Type":"ContainerDied","Data":"cff665589ef86ac4277764afea151bd8a14b99973a05878a7102de4555f94c48"}
Jul 27 18:19:27.860626 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:27.860603    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"
Jul 27 18:19:27.935990 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:27.935960    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6906b36-1c83-4dd6-8be9-05c4422921fd-home\") pod \"a6906b36-1c83-4dd6-8be9-05c4422921fd\" (UID: \"a6906b36-1c83-4dd6-8be9-05c4422921fd\") "
Jul 27 18:19:27.936151 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:27.936006    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8t9qd\" (UniqueName: \"kubernetes.io/projected/a6906b36-1c83-4dd6-8be9-05c4422921fd-kube-api-access-8t9qd\") pod \"a6906b36-1c83-4dd6-8be9-05c4422921fd\" (UID: \"a6906b36-1c83-4dd6-8be9-05c4422921fd\") "
Jul 27 18:19:27.936151 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:27.936052    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6906b36-1c83-4dd6-8be9-05c4422921fd-model-cache\") pod \"a6906b36-1c83-4dd6-8be9-05c4422921fd\" (UID: \"a6906b36-1c83-4dd6-8be9-05c4422921fd\") "
Jul 27 18:19:27.936151 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:27.936115    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6906b36-1c83-4dd6-8be9-05c4422921fd-dshm\") pod \"a6906b36-1c83-4dd6-8be9-05c4422921fd\" (UID: \"a6906b36-1c83-4dd6-8be9-05c4422921fd\") "
Jul 27 18:19:27.936151 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:27.936143    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6906b36-1c83-4dd6-8be9-05c4422921fd-tls-certs\") pod \"a6906b36-1c83-4dd6-8be9-05c4422921fd\" (UID: \"a6906b36-1c83-4dd6-8be9-05c4422921fd\") "
Jul 27 18:19:27.936371 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:27.936178    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6906b36-1c83-4dd6-8be9-05c4422921fd-tmp-dir\") pod \"a6906b36-1c83-4dd6-8be9-05c4422921fd\" (UID: \"a6906b36-1c83-4dd6-8be9-05c4422921fd\") "
Jul 27 18:19:27.936371 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:27.936177    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6906b36-1c83-4dd6-8be9-05c4422921fd-home" (OuterVolumeSpecName: "home") pod "a6906b36-1c83-4dd6-8be9-05c4422921fd" (UID: "a6906b36-1c83-4dd6-8be9-05c4422921fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:19:27.936371 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:27.936247    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6906b36-1c83-4dd6-8be9-05c4422921fd-kserve-provision-location\") pod \"a6906b36-1c83-4dd6-8be9-05c4422921fd\" (UID: \"a6906b36-1c83-4dd6-8be9-05c4422921fd\") "
Jul 27 18:19:27.936371 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:27.936281    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6906b36-1c83-4dd6-8be9-05c4422921fd-model-cache" (OuterVolumeSpecName: "model-cache") pod "a6906b36-1c83-4dd6-8be9-05c4422921fd" (UID: "a6906b36-1c83-4dd6-8be9-05c4422921fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:19:27.936576 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:27.936482    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6906b36-1c83-4dd6-8be9-05c4422921fd-home\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:19:27.936576 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:27.936497    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6906b36-1c83-4dd6-8be9-05c4422921fd-model-cache\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:19:27.936687 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:27.936642    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6906b36-1c83-4dd6-8be9-05c4422921fd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a6906b36-1c83-4dd6-8be9-05c4422921fd" (UID: "a6906b36-1c83-4dd6-8be9-05c4422921fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:19:27.938644 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:27.938607    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6906b36-1c83-4dd6-8be9-05c4422921fd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a6906b36-1c83-4dd6-8be9-05c4422921fd" (UID: "a6906b36-1c83-4dd6-8be9-05c4422921fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:19:27.938745 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:27.938655    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6906b36-1c83-4dd6-8be9-05c4422921fd-dshm" (OuterVolumeSpecName: "dshm") pod "a6906b36-1c83-4dd6-8be9-05c4422921fd" (UID: "a6906b36-1c83-4dd6-8be9-05c4422921fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:19:27.938745 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:27.938662    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6906b36-1c83-4dd6-8be9-05c4422921fd-kube-api-access-8t9qd" (OuterVolumeSpecName: "kube-api-access-8t9qd") pod "a6906b36-1c83-4dd6-8be9-05c4422921fd" (UID: "a6906b36-1c83-4dd6-8be9-05c4422921fd"). InnerVolumeSpecName "kube-api-access-8t9qd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:19:27.996358 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:27.996318    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6906b36-1c83-4dd6-8be9-05c4422921fd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a6906b36-1c83-4dd6-8be9-05c4422921fd" (UID: "a6906b36-1c83-4dd6-8be9-05c4422921fd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:19:28.037087 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.037052    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6906b36-1c83-4dd6-8be9-05c4422921fd-kserve-provision-location\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:19:28.037087 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.037091    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t9qd\" (UniqueName: \"kubernetes.io/projected/a6906b36-1c83-4dd6-8be9-05c4422921fd-kube-api-access-8t9qd\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:19:28.037272 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.037107    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6906b36-1c83-4dd6-8be9-05c4422921fd-dshm\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:19:28.037272 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.037121    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6906b36-1c83-4dd6-8be9-05c4422921fd-tls-certs\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:19:28.037272 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.037136    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6906b36-1c83-4dd6-8be9-05c4422921fd-tmp-dir\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:19:28.120339 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.120314    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g"
Jul 27 18:19:28.238640 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.238614    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7650e20-fd5f-4bfa-9028-3465aa597121-tls-certs\") pod \"b7650e20-fd5f-4bfa-9028-3465aa597121\" (UID: \"b7650e20-fd5f-4bfa-9028-3465aa597121\") "
Jul 27 18:19:28.238820 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.238648    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7650e20-fd5f-4bfa-9028-3465aa597121-kserve-provision-location\") pod \"b7650e20-fd5f-4bfa-9028-3465aa597121\" (UID: \"b7650e20-fd5f-4bfa-9028-3465aa597121\") "
Jul 27 18:19:28.238820 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.238700    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7650e20-fd5f-4bfa-9028-3465aa597121-tmp-dir\") pod \"b7650e20-fd5f-4bfa-9028-3465aa597121\" (UID: \"b7650e20-fd5f-4bfa-9028-3465aa597121\") "
Jul 27 18:19:28.239083 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.239061    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7650e20-fd5f-4bfa-9028-3465aa597121-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b7650e20-fd5f-4bfa-9028-3465aa597121" (UID: "b7650e20-fd5f-4bfa-9028-3465aa597121"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:19:28.240778 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.240739    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7650e20-fd5f-4bfa-9028-3465aa597121-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b7650e20-fd5f-4bfa-9028-3465aa597121" (UID: "b7650e20-fd5f-4bfa-9028-3465aa597121"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:19:28.294269 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.294232    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7650e20-fd5f-4bfa-9028-3465aa597121-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b7650e20-fd5f-4bfa-9028-3465aa597121" (UID: "b7650e20-fd5f-4bfa-9028-3465aa597121"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:19:28.340220 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.340195    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7650e20-fd5f-4bfa-9028-3465aa597121-tls-certs\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:19:28.340220 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.340218    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b7650e20-fd5f-4bfa-9028-3465aa597121-kserve-provision-location\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:19:28.340597 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.340245    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7650e20-fd5f-4bfa-9028-3465aa597121-tmp-dir\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:19:28.736635 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.736604    2557 generic.go:356] "Generic (PLEG): container finished" podID="b7650e20-fd5f-4bfa-9028-3465aa597121" containerID="84cceb8f0db56aec208f50201c4e08acd1a614bfd3d0ec80c28a759af1f52e2f" exitCode=0
Jul 27 18:19:28.736802 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.736669    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g"
Jul 27 18:19:28.736802 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.736691    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g" event={"ID":"b7650e20-fd5f-4bfa-9028-3465aa597121","Type":"ContainerDied","Data":"84cceb8f0db56aec208f50201c4e08acd1a614bfd3d0ec80c28a759af1f52e2f"}
Jul 27 18:19:28.736920 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.736800    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g" event={"ID":"b7650e20-fd5f-4bfa-9028-3465aa597121","Type":"ContainerDied","Data":"b53928919d13815c391d76551f7608aabee994ae634d68f916457c1c864e7d0d"}
Jul 27 18:19:28.736920 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.736835    2557 scope.go:117] "RemoveContainer" containerID="84cceb8f0db56aec208f50201c4e08acd1a614bfd3d0ec80c28a759af1f52e2f"
Jul 27 18:19:28.738502 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.738482    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s" event={"ID":"a6906b36-1c83-4dd6-8be9-05c4422921fd","Type":"ContainerDied","Data":"6266bd9bb82f670827b8e42c93b52dc75b93563ee9b3375849c42517b54e5342"}
Jul 27 18:19:28.738619 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.738525    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"
Jul 27 18:19:28.747653 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.747637    2557 scope.go:117] "RemoveContainer" containerID="1971ec239b5046548461ff1e1dbfe2ecac012e7a0b425f8d9ad989f2c1580bac"
Jul 27 18:19:28.757925 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.757909    2557 scope.go:117] "RemoveContainer" containerID="84cceb8f0db56aec208f50201c4e08acd1a614bfd3d0ec80c28a759af1f52e2f"
Jul 27 18:19:28.758134 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.758115    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"]
Jul 27 18:19:28.758212 ip-10-0-131-114 kubenswrapper[2557]: E0727 18:19:28.758194    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84cceb8f0db56aec208f50201c4e08acd1a614bfd3d0ec80c28a759af1f52e2f\": container with ID starting with 84cceb8f0db56aec208f50201c4e08acd1a614bfd3d0ec80c28a759af1f52e2f not found: ID does not exist" containerID="84cceb8f0db56aec208f50201c4e08acd1a614bfd3d0ec80c28a759af1f52e2f"
Jul 27 18:19:28.758251 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.758221    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84cceb8f0db56aec208f50201c4e08acd1a614bfd3d0ec80c28a759af1f52e2f"} err="failed to get container status \"84cceb8f0db56aec208f50201c4e08acd1a614bfd3d0ec80c28a759af1f52e2f\": rpc error: code = NotFound desc = could not find container \"84cceb8f0db56aec208f50201c4e08acd1a614bfd3d0ec80c28a759af1f52e2f\": container with ID starting with 84cceb8f0db56aec208f50201c4e08acd1a614bfd3d0ec80c28a759af1f52e2f not found: ID does not exist"
Jul 27 18:19:28.758251 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.758237    2557 scope.go:117] "RemoveContainer" containerID="1971ec239b5046548461ff1e1dbfe2ecac012e7a0b425f8d9ad989f2c1580bac"
Jul 27 18:19:28.758487 ip-10-0-131-114 kubenswrapper[2557]: E0727 18:19:28.758470    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1971ec239b5046548461ff1e1dbfe2ecac012e7a0b425f8d9ad989f2c1580bac\": container with ID starting with 1971ec239b5046548461ff1e1dbfe2ecac012e7a0b425f8d9ad989f2c1580bac not found: ID does not exist" containerID="1971ec239b5046548461ff1e1dbfe2ecac012e7a0b425f8d9ad989f2c1580bac"
Jul 27 18:19:28.758547 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.758491    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1971ec239b5046548461ff1e1dbfe2ecac012e7a0b425f8d9ad989f2c1580bac"} err="failed to get container status \"1971ec239b5046548461ff1e1dbfe2ecac012e7a0b425f8d9ad989f2c1580bac\": rpc error: code = NotFound desc = could not find container \"1971ec239b5046548461ff1e1dbfe2ecac012e7a0b425f8d9ad989f2c1580bac\": container with ID starting with 1971ec239b5046548461ff1e1dbfe2ecac012e7a0b425f8d9ad989f2c1580bac not found: ID does not exist"
Jul 27 18:19:28.758547 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.758508    2557 scope.go:117] "RemoveContainer" containerID="cff665589ef86ac4277764afea151bd8a14b99973a05878a7102de4555f94c48"
Jul 27 18:19:28.762314 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.762290    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-569c688787-9p66s"]
Jul 27 18:19:28.767130 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.767109    2557 scope.go:117] "RemoveContainer" containerID="80aa6a528cad483bc0c2a7d23c92361a5b5cb48758ad29dd220174ccb2ff646f"
Jul 27 18:19:28.771457 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.771437    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g"]
Jul 27 18:19:28.775237 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:28.775216    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-f9c56bf65-72j2g"]
Jul 27 18:19:30.118293 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:30.118262    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6906b36-1c83-4dd6-8be9-05c4422921fd" path="/var/lib/kubelet/pods/a6906b36-1c83-4dd6-8be9-05c4422921fd/volumes"
Jul 27 18:19:30.118678 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:30.118654    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7650e20-fd5f-4bfa-9028-3465aa597121" path="/var/lib/kubelet/pods/b7650e20-fd5f-4bfa-9028-3465aa597121/volumes"
Jul 27 18:19:53.704667 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.704630    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx"]
Jul 27 18:19:53.705180 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.705063    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7650e20-fd5f-4bfa-9028-3465aa597121" containerName="storage-initializer"
Jul 27 18:19:53.705180 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.705076    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7650e20-fd5f-4bfa-9028-3465aa597121" containerName="storage-initializer"
Jul 27 18:19:53.705180 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.705089    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6906b36-1c83-4dd6-8be9-05c4422921fd" containerName="main"
Jul 27 18:19:53.705180 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.705095    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6906b36-1c83-4dd6-8be9-05c4422921fd" containerName="main"
Jul 27 18:19:53.705180 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.705101    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7650e20-fd5f-4bfa-9028-3465aa597121" containerName="main"
Jul 27 18:19:53.705180 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.705106    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7650e20-fd5f-4bfa-9028-3465aa597121" containerName="main"
Jul 27 18:19:53.705180 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.705123    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6906b36-1c83-4dd6-8be9-05c4422921fd" containerName="storage-initializer"
Jul 27 18:19:53.705180 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.705131    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6906b36-1c83-4dd6-8be9-05c4422921fd" containerName="storage-initializer"
Jul 27 18:19:53.705430 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.705188    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6906b36-1c83-4dd6-8be9-05c4422921fd" containerName="main"
Jul 27 18:19:53.705430 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.705197    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7650e20-fd5f-4bfa-9028-3465aa597121" containerName="main"
Jul 27 18:19:53.707775 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.707741    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx"
Jul 27 18:19:53.711744 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.711723    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Jul 27 18:19:53.712415 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.712397    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-4t6g8\""
Jul 27 18:19:53.712415 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.712409    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Jul 27 18:19:53.712566 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.712398    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 27 18:19:53.718962 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.718942    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx"]
Jul 27 18:19:53.853026 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.852998    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdc7d484-0389-4ff4-95af-388686a1dcb8-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx\" (UID: \"fdc7d484-0389-4ff4-95af-388686a1dcb8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx"
Jul 27 18:19:53.853215 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.853036    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdc7d484-0389-4ff4-95af-388686a1dcb8-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx\" (UID: \"fdc7d484-0389-4ff4-95af-388686a1dcb8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx"
Jul 27 18:19:53.853215 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.853064    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9php\" (UniqueName: \"kubernetes.io/projected/fdc7d484-0389-4ff4-95af-388686a1dcb8-kube-api-access-v9php\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx\" (UID: \"fdc7d484-0389-4ff4-95af-388686a1dcb8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx"
Jul 27 18:19:53.853215 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.853170    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdc7d484-0389-4ff4-95af-388686a1dcb8-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx\" (UID: \"fdc7d484-0389-4ff4-95af-388686a1dcb8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx"
Jul 27 18:19:53.853215 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.853196    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdc7d484-0389-4ff4-95af-388686a1dcb8-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx\" (UID: \"fdc7d484-0389-4ff4-95af-388686a1dcb8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx"
Jul 27 18:19:53.853358 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.853295    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdc7d484-0389-4ff4-95af-388686a1dcb8-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx\" (UID: \"fdc7d484-0389-4ff4-95af-388686a1dcb8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx"
Jul 27 18:19:53.954810 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.954694    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdc7d484-0389-4ff4-95af-388686a1dcb8-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx\" (UID: \"fdc7d484-0389-4ff4-95af-388686a1dcb8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx"
Jul 27 18:19:53.954810 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.954740    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdc7d484-0389-4ff4-95af-388686a1dcb8-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx\" (UID: \"fdc7d484-0389-4ff4-95af-388686a1dcb8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx"
Jul 27 18:19:53.954810 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.954781    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdc7d484-0389-4ff4-95af-388686a1dcb8-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx\" (UID: \"fdc7d484-0389-4ff4-95af-388686a1dcb8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx"
Jul 27 18:19:53.954810 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.954801    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9php\" (UniqueName: \"kubernetes.io/projected/fdc7d484-0389-4ff4-95af-388686a1dcb8-kube-api-access-v9php\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx\" (UID: \"fdc7d484-0389-4ff4-95af-388686a1dcb8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx"
Jul 27 18:19:53.955171 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.954849    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdc7d484-0389-4ff4-95af-388686a1dcb8-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx\" (UID: \"fdc7d484-0389-4ff4-95af-388686a1dcb8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx"
Jul 27 18:19:53.955171 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.954864    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdc7d484-0389-4ff4-95af-388686a1dcb8-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx\" (UID: \"fdc7d484-0389-4ff4-95af-388686a1dcb8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx"
Jul 27 18:19:53.955171 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.955147    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdc7d484-0389-4ff4-95af-388686a1dcb8-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx\" (UID: \"fdc7d484-0389-4ff4-95af-388686a1dcb8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx"
Jul 27 18:19:53.955284 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.955215    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdc7d484-0389-4ff4-95af-388686a1dcb8-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx\" (UID: \"fdc7d484-0389-4ff4-95af-388686a1dcb8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx"
Jul 27 18:19:53.955284 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.955276    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdc7d484-0389-4ff4-95af-388686a1dcb8-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx\" (UID: \"fdc7d484-0389-4ff4-95af-388686a1dcb8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx"
Jul 27 18:19:53.957054 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.957025    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdc7d484-0389-4ff4-95af-388686a1dcb8-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx\" (UID: \"fdc7d484-0389-4ff4-95af-388686a1dcb8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx"
Jul 27 18:19:53.957358 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.957338    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdc7d484-0389-4ff4-95af-388686a1dcb8-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx\" (UID: \"fdc7d484-0389-4ff4-95af-388686a1dcb8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx"
Jul 27 18:19:53.963234 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:53.963209    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9php\" (UniqueName: \"kubernetes.io/projected/fdc7d484-0389-4ff4-95af-388686a1dcb8-kube-api-access-v9php\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx\" (UID: \"fdc7d484-0389-4ff4-95af-388686a1dcb8\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx"
Jul 27 18:19:54.018534 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:54.018495    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx"
Jul 27 18:19:54.154047 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:54.154008    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx"]
Jul 27 18:19:54.156983 ip-10-0-131-114 kubenswrapper[2557]: W0727 18:19:54.156950    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc7d484_0389_4ff4_95af_388686a1dcb8.slice/crio-33c6a54b289499c436c4e72e0856750b8c6ea6bc8712a01d352732f2179765cd WatchSource:0}: Error finding container 33c6a54b289499c436c4e72e0856750b8c6ea6bc8712a01d352732f2179765cd: Status 404 returned error can't find the container with id 33c6a54b289499c436c4e72e0856750b8c6ea6bc8712a01d352732f2179765cd
Jul 27 18:19:54.861655 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:54.861615    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx" event={"ID":"fdc7d484-0389-4ff4-95af-388686a1dcb8","Type":"ContainerStarted","Data":"e54311dfd7e06efec10ea4f2eb9271edc35791d0a1415f04726094ce4e911c15"}
Jul 27 18:19:54.862091 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:54.861682    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx" event={"ID":"fdc7d484-0389-4ff4-95af-388686a1dcb8","Type":"ContainerStarted","Data":"33c6a54b289499c436c4e72e0856750b8c6ea6bc8712a01d352732f2179765cd"}
Jul 27 18:19:54.878071 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:54.878022    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx" podStartSLOduration=1.8780079939999998 podStartE2EDuration="1.878007994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:19:54.876110976 +0000 UTC m=+3087.362922288" watchObservedRunningTime="2026-07-27 18:19:54.878007994 +0000 UTC m=+3087.364819281"
Jul 27 18:19:55.019006 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:19:55.018972    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx"
Jul 27 18:20:04.018948 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:04.018906    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx"
Jul 27 18:20:04.031731 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:04.031698    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx"
Jul 27 18:20:04.918605 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:04.918576    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx"
Jul 27 18:20:27.234223 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:27.234183    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx"]
Jul 27 18:20:27.259998 ip-10-0-131-114 kubenswrapper[2557]: E0727 18:20:27.259949    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 18:20:27.259998 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:20:27.259998 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 18:20:27.259998 ip-10-0-131-114 kubenswrapper[2557]:  > containerID="e54311dfd7e06efec10ea4f2eb9271edc35791d0a1415f04726094ce4e911c15" cmd=["/bin/sleep","15"]
Jul 27 18:20:27.260260 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:27.260022    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx" message="" err=<
Jul 27 18:20:27.260260 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:20:27.260260 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 18:20:27.260260 ip-10-0-131-114 kubenswrapper[2557]:  >
Jul 27 18:20:27.260260 ip-10-0-131-114 kubenswrapper[2557]: E0727 18:20:27.260039    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 18:20:27.260260 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:20:27.260260 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 18:20:27.260260 ip-10-0-131-114 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx" podUID="fdc7d484-0389-4ff4-95af-388686a1dcb8" containerName="main" containerID="cri-o://e54311dfd7e06efec10ea4f2eb9271edc35791d0a1415f04726094ce4e911c15"
Jul 27 18:20:27.260260 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:27.260084    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx" podUID="fdc7d484-0389-4ff4-95af-388686a1dcb8" containerName="main" containerID="cri-o://e54311dfd7e06efec10ea4f2eb9271edc35791d0a1415f04726094ce4e911c15" gracePeriod=60
Jul 27 18:20:27.614984 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:27.614956    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx"
Jul 27 18:20:27.760047 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:27.760010    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9php\" (UniqueName: \"kubernetes.io/projected/fdc7d484-0389-4ff4-95af-388686a1dcb8-kube-api-access-v9php\") pod \"fdc7d484-0389-4ff4-95af-388686a1dcb8\" (UID: \"fdc7d484-0389-4ff4-95af-388686a1dcb8\") "
Jul 27 18:20:27.760047 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:27.760053    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdc7d484-0389-4ff4-95af-388686a1dcb8-tmp-dir\") pod \"fdc7d484-0389-4ff4-95af-388686a1dcb8\" (UID: \"fdc7d484-0389-4ff4-95af-388686a1dcb8\") "
Jul 27 18:20:27.760307 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:27.760074    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdc7d484-0389-4ff4-95af-388686a1dcb8-dshm\") pod \"fdc7d484-0389-4ff4-95af-388686a1dcb8\" (UID: \"fdc7d484-0389-4ff4-95af-388686a1dcb8\") "
Jul 27 18:20:27.760307 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:27.760097    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdc7d484-0389-4ff4-95af-388686a1dcb8-tls-certs\") pod \"fdc7d484-0389-4ff4-95af-388686a1dcb8\" (UID: \"fdc7d484-0389-4ff4-95af-388686a1dcb8\") "
Jul 27 18:20:27.760307 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:27.760149    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdc7d484-0389-4ff4-95af-388686a1dcb8-home\") pod \"fdc7d484-0389-4ff4-95af-388686a1dcb8\" (UID: \"fdc7d484-0389-4ff4-95af-388686a1dcb8\") "
Jul 27 18:20:27.760307 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:27.760190    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdc7d484-0389-4ff4-95af-388686a1dcb8-model-cache\") pod \"fdc7d484-0389-4ff4-95af-388686a1dcb8\" (UID: \"fdc7d484-0389-4ff4-95af-388686a1dcb8\") "
Jul 27 18:20:27.760521 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:27.760345    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc7d484-0389-4ff4-95af-388686a1dcb8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fdc7d484-0389-4ff4-95af-388686a1dcb8" (UID: "fdc7d484-0389-4ff4-95af-388686a1dcb8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:20:27.760521 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:27.760422    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc7d484-0389-4ff4-95af-388686a1dcb8-home" (OuterVolumeSpecName: "home") pod "fdc7d484-0389-4ff4-95af-388686a1dcb8" (UID: "fdc7d484-0389-4ff4-95af-388686a1dcb8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:20:27.760521 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:27.760470    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdc7d484-0389-4ff4-95af-388686a1dcb8-tmp-dir\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:20:27.760521 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:27.760485    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc7d484-0389-4ff4-95af-388686a1dcb8-model-cache" (OuterVolumeSpecName: "model-cache") pod "fdc7d484-0389-4ff4-95af-388686a1dcb8" (UID: "fdc7d484-0389-4ff4-95af-388686a1dcb8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:20:27.762546 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:27.762516    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc7d484-0389-4ff4-95af-388686a1dcb8-dshm" (OuterVolumeSpecName: "dshm") pod "fdc7d484-0389-4ff4-95af-388686a1dcb8" (UID: "fdc7d484-0389-4ff4-95af-388686a1dcb8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:20:27.762546 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:27.762544    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdc7d484-0389-4ff4-95af-388686a1dcb8-kube-api-access-v9php" (OuterVolumeSpecName: "kube-api-access-v9php") pod "fdc7d484-0389-4ff4-95af-388686a1dcb8" (UID: "fdc7d484-0389-4ff4-95af-388686a1dcb8"). InnerVolumeSpecName "kube-api-access-v9php". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:20:27.762705 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:27.762575    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdc7d484-0389-4ff4-95af-388686a1dcb8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fdc7d484-0389-4ff4-95af-388686a1dcb8" (UID: "fdc7d484-0389-4ff4-95af-388686a1dcb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:20:27.861523 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:27.861443    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9php\" (UniqueName: \"kubernetes.io/projected/fdc7d484-0389-4ff4-95af-388686a1dcb8-kube-api-access-v9php\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:20:27.861523 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:27.861475    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdc7d484-0389-4ff4-95af-388686a1dcb8-dshm\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:20:27.861523 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:27.861487    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdc7d484-0389-4ff4-95af-388686a1dcb8-tls-certs\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:20:27.861523 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:27.861496    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdc7d484-0389-4ff4-95af-388686a1dcb8-home\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:20:27.861523 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:27.861503    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdc7d484-0389-4ff4-95af-388686a1dcb8-model-cache\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:20:28.025215 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:28.025180    2557 generic.go:356] "Generic (PLEG): container finished" podID="fdc7d484-0389-4ff4-95af-388686a1dcb8" containerID="e54311dfd7e06efec10ea4f2eb9271edc35791d0a1415f04726094ce4e911c15" exitCode=0
Jul 27 18:20:28.025392 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:28.025223    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx"
Jul 27 18:20:28.025392 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:28.025272    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx" event={"ID":"fdc7d484-0389-4ff4-95af-388686a1dcb8","Type":"ContainerDied","Data":"e54311dfd7e06efec10ea4f2eb9271edc35791d0a1415f04726094ce4e911c15"}
Jul 27 18:20:28.025392 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:28.025347    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx" event={"ID":"fdc7d484-0389-4ff4-95af-388686a1dcb8","Type":"ContainerDied","Data":"33c6a54b289499c436c4e72e0856750b8c6ea6bc8712a01d352732f2179765cd"}
Jul 27 18:20:28.025392 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:28.025369    2557 scope.go:117] "RemoveContainer" containerID="e54311dfd7e06efec10ea4f2eb9271edc35791d0a1415f04726094ce4e911c15"
Jul 27 18:20:28.034625 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:28.034592    2557 scope.go:117] "RemoveContainer" containerID="e54311dfd7e06efec10ea4f2eb9271edc35791d0a1415f04726094ce4e911c15"
Jul 27 18:20:28.034853 ip-10-0-131-114 kubenswrapper[2557]: E0727 18:20:28.034835    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e54311dfd7e06efec10ea4f2eb9271edc35791d0a1415f04726094ce4e911c15\": container with ID starting with e54311dfd7e06efec10ea4f2eb9271edc35791d0a1415f04726094ce4e911c15 not found: ID does not exist" containerID="e54311dfd7e06efec10ea4f2eb9271edc35791d0a1415f04726094ce4e911c15"
Jul 27 18:20:28.034909 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:28.034861    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e54311dfd7e06efec10ea4f2eb9271edc35791d0a1415f04726094ce4e911c15"} err="failed to get container status \"e54311dfd7e06efec10ea4f2eb9271edc35791d0a1415f04726094ce4e911c15\": rpc error: code = NotFound desc = could not find container \"e54311dfd7e06efec10ea4f2eb9271edc35791d0a1415f04726094ce4e911c15\": container with ID starting with e54311dfd7e06efec10ea4f2eb9271edc35791d0a1415f04726094ce4e911c15 not found: ID does not exist"
Jul 27 18:20:28.046617 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:28.046595    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx"]
Jul 27 18:20:28.054898 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:28.054875    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-6n2sx"]
Jul 27 18:20:28.119557 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:28.119472    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdc7d484-0389-4ff4-95af-388686a1dcb8" path="/var/lib/kubelet/pods/fdc7d484-0389-4ff4-95af-388686a1dcb8/volumes"
Jul 27 18:20:52.995546 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:52.995509    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7"]
Jul 27 18:20:52.996102 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:52.996050    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdc7d484-0389-4ff4-95af-388686a1dcb8" containerName="main"
Jul 27 18:20:52.996102 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:52.996071    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdc7d484-0389-4ff4-95af-388686a1dcb8" containerName="main"
Jul 27 18:20:52.996219 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:52.996195    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdc7d484-0389-4ff4-95af-388686a1dcb8" containerName="main"
Jul 27 18:20:52.999234 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:52.999215    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7"
Jul 27 18:20:53.001390 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:53.001368    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Jul 27 18:20:53.002017 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:53.001996    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 27 18:20:53.002138 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:53.002064    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-n86fh\""
Jul 27 18:20:53.002138 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:53.002119    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Jul 27 18:20:53.007935 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:53.007913    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7"]
Jul 27 18:20:53.068692 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:53.068657    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/057db77e-7092-42e0-9d5c-217c2f172b6d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7\" (UID: \"057db77e-7092-42e0-9d5c-217c2f172b6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7"
Jul 27 18:20:53.068692 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:53.068697    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw467\" (UniqueName: \"kubernetes.io/projected/057db77e-7092-42e0-9d5c-217c2f172b6d-kube-api-access-cw467\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7\" (UID: \"057db77e-7092-42e0-9d5c-217c2f172b6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7"
Jul 27 18:20:53.068955 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:53.068724    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/057db77e-7092-42e0-9d5c-217c2f172b6d-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7\" (UID: \"057db77e-7092-42e0-9d5c-217c2f172b6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7"
Jul 27 18:20:53.068955 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:53.068847    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/057db77e-7092-42e0-9d5c-217c2f172b6d-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7\" (UID: \"057db77e-7092-42e0-9d5c-217c2f172b6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7"
Jul 27 18:20:53.068955 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:53.068894    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/057db77e-7092-42e0-9d5c-217c2f172b6d-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7\" (UID: \"057db77e-7092-42e0-9d5c-217c2f172b6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7"
Jul 27 18:20:53.068955 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:53.068913    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/057db77e-7092-42e0-9d5c-217c2f172b6d-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7\" (UID: \"057db77e-7092-42e0-9d5c-217c2f172b6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7"
Jul 27 18:20:53.169921 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:53.169874    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/057db77e-7092-42e0-9d5c-217c2f172b6d-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7\" (UID: \"057db77e-7092-42e0-9d5c-217c2f172b6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7"
Jul 27 18:20:53.170130 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:53.169956    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/057db77e-7092-42e0-9d5c-217c2f172b6d-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7\" (UID: \"057db77e-7092-42e0-9d5c-217c2f172b6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7"
Jul 27 18:20:53.170130 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:53.169991    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/057db77e-7092-42e0-9d5c-217c2f172b6d-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7\" (UID: \"057db77e-7092-42e0-9d5c-217c2f172b6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7"
Jul 27 18:20:53.170130 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:53.170049    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/057db77e-7092-42e0-9d5c-217c2f172b6d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7\" (UID: \"057db77e-7092-42e0-9d5c-217c2f172b6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7"
Jul 27 18:20:53.170130 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:53.170089    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw467\" (UniqueName: \"kubernetes.io/projected/057db77e-7092-42e0-9d5c-217c2f172b6d-kube-api-access-cw467\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7\" (UID: \"057db77e-7092-42e0-9d5c-217c2f172b6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7"
Jul 27 18:20:53.170353 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:53.170131    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/057db77e-7092-42e0-9d5c-217c2f172b6d-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7\" (UID: \"057db77e-7092-42e0-9d5c-217c2f172b6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7"
Jul 27 18:20:53.170681 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:53.170656    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/057db77e-7092-42e0-9d5c-217c2f172b6d-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7\" (UID: \"057db77e-7092-42e0-9d5c-217c2f172b6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7"
Jul 27 18:20:53.170955 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:53.170936    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/057db77e-7092-42e0-9d5c-217c2f172b6d-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7\" (UID: \"057db77e-7092-42e0-9d5c-217c2f172b6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7"
Jul 27 18:20:53.173577 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:53.171870    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/057db77e-7092-42e0-9d5c-217c2f172b6d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7\" (UID: \"057db77e-7092-42e0-9d5c-217c2f172b6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7"
Jul 27 18:20:53.174178 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:53.174153    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/057db77e-7092-42e0-9d5c-217c2f172b6d-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7\" (UID: \"057db77e-7092-42e0-9d5c-217c2f172b6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7"
Jul 27 18:20:53.178065 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:53.178041    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/057db77e-7092-42e0-9d5c-217c2f172b6d-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7\" (UID: \"057db77e-7092-42e0-9d5c-217c2f172b6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7"
Jul 27 18:20:53.180085 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:53.180067    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw467\" (UniqueName: \"kubernetes.io/projected/057db77e-7092-42e0-9d5c-217c2f172b6d-kube-api-access-cw467\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7\" (UID: \"057db77e-7092-42e0-9d5c-217c2f172b6d\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7"
Jul 27 18:20:53.311069 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:53.310976    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7"
Jul 27 18:20:53.448813 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:53.448788    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7"]
Jul 27 18:20:53.451972 ip-10-0-131-114 kubenswrapper[2557]: W0727 18:20:53.451938    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod057db77e_7092_42e0_9d5c_217c2f172b6d.slice/crio-9f824873c5afd83f06db68f32b40e8a30912a97da856f51b4e5e5783e64ad2d3 WatchSource:0}: Error finding container 9f824873c5afd83f06db68f32b40e8a30912a97da856f51b4e5e5783e64ad2d3: Status 404 returned error can't find the container with id 9f824873c5afd83f06db68f32b40e8a30912a97da856f51b4e5e5783e64ad2d3
Jul 27 18:20:54.144979 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:54.144941    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7" event={"ID":"057db77e-7092-42e0-9d5c-217c2f172b6d","Type":"ContainerStarted","Data":"615cb71f87c386e9f87d10a2eaa982311b0e689479cb5a8f4657737136d7ccfd"}
Jul 27 18:20:54.145394 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:54.145013    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7" event={"ID":"057db77e-7092-42e0-9d5c-217c2f172b6d","Type":"ContainerStarted","Data":"9f824873c5afd83f06db68f32b40e8a30912a97da856f51b4e5e5783e64ad2d3"}
Jul 27 18:20:54.161862 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:54.161812    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7" podStartSLOduration=2.161796158 podStartE2EDuration="2.161796158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:20:54.159944285 +0000 UTC m=+3146.646755572" watchObservedRunningTime="2026-07-27 18:20:54.161796158 +0000 UTC m=+3146.648607444"
Jul 27 18:20:54.312086 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:20:54.312046    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7"
Jul 27 18:21:03.311588 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:21:03.311550    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7"
Jul 27 18:21:03.323964 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:21:03.323938    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7"
Jul 27 18:21:04.201578 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:21:04.201546    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7"
Jul 27 18:22:24.622312 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:24.622223    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7"]
Jul 27 18:22:24.632655 ip-10-0-131-114 kubenswrapper[2557]: E0727 18:22:24.632616    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 18:22:24.632655 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:22:24.632655 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 18:22:24.632655 ip-10-0-131-114 kubenswrapper[2557]:  > containerID="615cb71f87c386e9f87d10a2eaa982311b0e689479cb5a8f4657737136d7ccfd" cmd=["/bin/sleep","15"]
Jul 27 18:22:24.632996 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:24.632679    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7" message="" err=<
Jul 27 18:22:24.632996 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:22:24.632996 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 18:22:24.632996 ip-10-0-131-114 kubenswrapper[2557]:  >
Jul 27 18:22:24.632996 ip-10-0-131-114 kubenswrapper[2557]: E0727 18:22:24.632698    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 18:22:24.632996 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:22:24.632996 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 18:22:24.632996 ip-10-0-131-114 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7" podUID="057db77e-7092-42e0-9d5c-217c2f172b6d" containerName="main" containerID="cri-o://615cb71f87c386e9f87d10a2eaa982311b0e689479cb5a8f4657737136d7ccfd"
Jul 27 18:22:24.632996 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:24.632747    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7" podUID="057db77e-7092-42e0-9d5c-217c2f172b6d" containerName="main" containerID="cri-o://615cb71f87c386e9f87d10a2eaa982311b0e689479cb5a8f4657737136d7ccfd" gracePeriod=60
Jul 27 18:22:24.994165 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:24.994136    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7"
Jul 27 18:22:25.173557 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:25.173530    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/057db77e-7092-42e0-9d5c-217c2f172b6d-dshm\") pod \"057db77e-7092-42e0-9d5c-217c2f172b6d\" (UID: \"057db77e-7092-42e0-9d5c-217c2f172b6d\") "
Jul 27 18:22:25.173748 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:25.173569    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/057db77e-7092-42e0-9d5c-217c2f172b6d-model-cache\") pod \"057db77e-7092-42e0-9d5c-217c2f172b6d\" (UID: \"057db77e-7092-42e0-9d5c-217c2f172b6d\") "
Jul 27 18:22:25.173748 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:25.173590    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/057db77e-7092-42e0-9d5c-217c2f172b6d-tmp-dir\") pod \"057db77e-7092-42e0-9d5c-217c2f172b6d\" (UID: \"057db77e-7092-42e0-9d5c-217c2f172b6d\") "
Jul 27 18:22:25.173748 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:25.173649    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/057db77e-7092-42e0-9d5c-217c2f172b6d-home\") pod \"057db77e-7092-42e0-9d5c-217c2f172b6d\" (UID: \"057db77e-7092-42e0-9d5c-217c2f172b6d\") "
Jul 27 18:22:25.173748 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:25.173672    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cw467\" (UniqueName: \"kubernetes.io/projected/057db77e-7092-42e0-9d5c-217c2f172b6d-kube-api-access-cw467\") pod \"057db77e-7092-42e0-9d5c-217c2f172b6d\" (UID: \"057db77e-7092-42e0-9d5c-217c2f172b6d\") "
Jul 27 18:22:25.173748 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:25.173686    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/057db77e-7092-42e0-9d5c-217c2f172b6d-tls-certs\") pod \"057db77e-7092-42e0-9d5c-217c2f172b6d\" (UID: \"057db77e-7092-42e0-9d5c-217c2f172b6d\") "
Jul 27 18:22:25.174087 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:25.173829    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/057db77e-7092-42e0-9d5c-217c2f172b6d-model-cache" (OuterVolumeSpecName: "model-cache") pod "057db77e-7092-42e0-9d5c-217c2f172b6d" (UID: "057db77e-7092-42e0-9d5c-217c2f172b6d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:22:25.174087 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:25.173900    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/057db77e-7092-42e0-9d5c-217c2f172b6d-home" (OuterVolumeSpecName: "home") pod "057db77e-7092-42e0-9d5c-217c2f172b6d" (UID: "057db77e-7092-42e0-9d5c-217c2f172b6d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:22:25.174087 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:25.173914    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/057db77e-7092-42e0-9d5c-217c2f172b6d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "057db77e-7092-42e0-9d5c-217c2f172b6d" (UID: "057db77e-7092-42e0-9d5c-217c2f172b6d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:22:25.174087 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:25.174038    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/057db77e-7092-42e0-9d5c-217c2f172b6d-model-cache\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:22:25.174087 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:25.174055    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/057db77e-7092-42e0-9d5c-217c2f172b6d-tmp-dir\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:22:25.174087 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:25.174067    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/057db77e-7092-42e0-9d5c-217c2f172b6d-home\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:22:25.175929 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:25.175908    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/057db77e-7092-42e0-9d5c-217c2f172b6d-dshm" (OuterVolumeSpecName: "dshm") pod "057db77e-7092-42e0-9d5c-217c2f172b6d" (UID: "057db77e-7092-42e0-9d5c-217c2f172b6d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:22:25.175998 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:25.175936    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/057db77e-7092-42e0-9d5c-217c2f172b6d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "057db77e-7092-42e0-9d5c-217c2f172b6d" (UID: "057db77e-7092-42e0-9d5c-217c2f172b6d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:22:25.176059 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:25.176037    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/057db77e-7092-42e0-9d5c-217c2f172b6d-kube-api-access-cw467" (OuterVolumeSpecName: "kube-api-access-cw467") pod "057db77e-7092-42e0-9d5c-217c2f172b6d" (UID: "057db77e-7092-42e0-9d5c-217c2f172b6d"). InnerVolumeSpecName "kube-api-access-cw467". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:22:25.275180 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:25.275147    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw467\" (UniqueName: \"kubernetes.io/projected/057db77e-7092-42e0-9d5c-217c2f172b6d-kube-api-access-cw467\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:22:25.275180 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:25.275177    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/057db77e-7092-42e0-9d5c-217c2f172b6d-tls-certs\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:22:25.275370 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:25.275192    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/057db77e-7092-42e0-9d5c-217c2f172b6d-dshm\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:22:25.573696 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:25.573604    2557 generic.go:356] "Generic (PLEG): container finished" podID="057db77e-7092-42e0-9d5c-217c2f172b6d" containerID="615cb71f87c386e9f87d10a2eaa982311b0e689479cb5a8f4657737136d7ccfd" exitCode=0
Jul 27 18:22:25.573696 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:25.573647    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7" event={"ID":"057db77e-7092-42e0-9d5c-217c2f172b6d","Type":"ContainerDied","Data":"615cb71f87c386e9f87d10a2eaa982311b0e689479cb5a8f4657737136d7ccfd"}
Jul 27 18:22:25.573696 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:25.573657    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7"
Jul 27 18:22:25.573696 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:25.573697    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7" event={"ID":"057db77e-7092-42e0-9d5c-217c2f172b6d","Type":"ContainerDied","Data":"9f824873c5afd83f06db68f32b40e8a30912a97da856f51b4e5e5783e64ad2d3"}
Jul 27 18:22:25.574262 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:25.573722    2557 scope.go:117] "RemoveContainer" containerID="615cb71f87c386e9f87d10a2eaa982311b0e689479cb5a8f4657737136d7ccfd"
Jul 27 18:22:25.582925 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:25.582907    2557 scope.go:117] "RemoveContainer" containerID="615cb71f87c386e9f87d10a2eaa982311b0e689479cb5a8f4657737136d7ccfd"
Jul 27 18:22:25.583169 ip-10-0-131-114 kubenswrapper[2557]: E0727 18:22:25.583151    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"615cb71f87c386e9f87d10a2eaa982311b0e689479cb5a8f4657737136d7ccfd\": container with ID starting with 615cb71f87c386e9f87d10a2eaa982311b0e689479cb5a8f4657737136d7ccfd not found: ID does not exist" containerID="615cb71f87c386e9f87d10a2eaa982311b0e689479cb5a8f4657737136d7ccfd"
Jul 27 18:22:25.583213 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:25.583179    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"615cb71f87c386e9f87d10a2eaa982311b0e689479cb5a8f4657737136d7ccfd"} err="failed to get container status \"615cb71f87c386e9f87d10a2eaa982311b0e689479cb5a8f4657737136d7ccfd\": rpc error: code = NotFound desc = could not find container \"615cb71f87c386e9f87d10a2eaa982311b0e689479cb5a8f4657737136d7ccfd\": container with ID starting with 615cb71f87c386e9f87d10a2eaa982311b0e689479cb5a8f4657737136d7ccfd not found: ID does not exist"
Jul 27 18:22:25.594931 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:25.594902    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7"]
Jul 27 18:22:25.598787 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:25.598743    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-s2xv7"]
Jul 27 18:22:26.119553 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:26.119510    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="057db77e-7092-42e0-9d5c-217c2f172b6d" path="/var/lib/kubelet/pods/057db77e-7092-42e0-9d5c-217c2f172b6d/volumes"
Jul 27 18:22:51.894551 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:51.894452    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2"]
Jul 27 18:22:51.895088 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:51.894971    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="057db77e-7092-42e0-9d5c-217c2f172b6d" containerName="main"
Jul 27 18:22:51.895088 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:51.894988    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="057db77e-7092-42e0-9d5c-217c2f172b6d" containerName="main"
Jul 27 18:22:51.895209 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:51.895100    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="057db77e-7092-42e0-9d5c-217c2f172b6d" containerName="main"
Jul 27 18:22:51.898036 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:51.898016    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2"
Jul 27 18:22:51.900253 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:51.900235    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Jul 27 18:22:51.900827 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:51.900811    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 27 18:22:51.900921 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:51.900839    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Jul 27 18:22:51.900921 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:51.900845    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-r7d8l\""
Jul 27 18:22:51.905960 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:51.905938    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2"]
Jul 27 18:22:51.996073 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:51.996041    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37f661d9-1d71-423c-b330-b5d25ae41662-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2\" (UID: \"37f661d9-1d71-423c-b330-b5d25ae41662\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2"
Jul 27 18:22:51.996234 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:51.996079    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbm4c\" (UniqueName: \"kubernetes.io/projected/37f661d9-1d71-423c-b330-b5d25ae41662-kube-api-access-rbm4c\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2\" (UID: \"37f661d9-1d71-423c-b330-b5d25ae41662\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2"
Jul 27 18:22:51.996234 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:51.996109    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37f661d9-1d71-423c-b330-b5d25ae41662-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2\" (UID: \"37f661d9-1d71-423c-b330-b5d25ae41662\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2"
Jul 27 18:22:51.996234 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:51.996199    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37f661d9-1d71-423c-b330-b5d25ae41662-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2\" (UID: \"37f661d9-1d71-423c-b330-b5d25ae41662\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2"
Jul 27 18:22:51.996234 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:51.996229    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f661d9-1d71-423c-b330-b5d25ae41662-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2\" (UID: \"37f661d9-1d71-423c-b330-b5d25ae41662\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2"
Jul 27 18:22:51.996421 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:51.996307    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f661d9-1d71-423c-b330-b5d25ae41662-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2\" (UID: \"37f661d9-1d71-423c-b330-b5d25ae41662\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2"
Jul 27 18:22:52.097307 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:52.097277    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37f661d9-1d71-423c-b330-b5d25ae41662-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2\" (UID: \"37f661d9-1d71-423c-b330-b5d25ae41662\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2"
Jul 27 18:22:52.097511 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:52.097314    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbm4c\" (UniqueName: \"kubernetes.io/projected/37f661d9-1d71-423c-b330-b5d25ae41662-kube-api-access-rbm4c\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2\" (UID: \"37f661d9-1d71-423c-b330-b5d25ae41662\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2"
Jul 27 18:22:52.097511 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:52.097338    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37f661d9-1d71-423c-b330-b5d25ae41662-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2\" (UID: \"37f661d9-1d71-423c-b330-b5d25ae41662\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2"
Jul 27 18:22:52.097511 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:52.097361    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37f661d9-1d71-423c-b330-b5d25ae41662-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2\" (UID: \"37f661d9-1d71-423c-b330-b5d25ae41662\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2"
Jul 27 18:22:52.097511 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:52.097378    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f661d9-1d71-423c-b330-b5d25ae41662-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2\" (UID: \"37f661d9-1d71-423c-b330-b5d25ae41662\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2"
Jul 27 18:22:52.097511 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:52.097427    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f661d9-1d71-423c-b330-b5d25ae41662-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2\" (UID: \"37f661d9-1d71-423c-b330-b5d25ae41662\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2"
Jul 27 18:22:52.097839 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:52.097720    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37f661d9-1d71-423c-b330-b5d25ae41662-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2\" (UID: \"37f661d9-1d71-423c-b330-b5d25ae41662\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2"
Jul 27 18:22:52.097839 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:52.097747    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37f661d9-1d71-423c-b330-b5d25ae41662-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2\" (UID: \"37f661d9-1d71-423c-b330-b5d25ae41662\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2"
Jul 27 18:22:52.097839 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:52.097831    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f661d9-1d71-423c-b330-b5d25ae41662-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2\" (UID: \"37f661d9-1d71-423c-b330-b5d25ae41662\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2"
Jul 27 18:22:52.099745 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:52.099723    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37f661d9-1d71-423c-b330-b5d25ae41662-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2\" (UID: \"37f661d9-1d71-423c-b330-b5d25ae41662\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2"
Jul 27 18:22:52.099981 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:52.099966    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f661d9-1d71-423c-b330-b5d25ae41662-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2\" (UID: \"37f661d9-1d71-423c-b330-b5d25ae41662\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2"
Jul 27 18:22:52.105005 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:52.104987    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbm4c\" (UniqueName: \"kubernetes.io/projected/37f661d9-1d71-423c-b330-b5d25ae41662-kube-api-access-rbm4c\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2\" (UID: \"37f661d9-1d71-423c-b330-b5d25ae41662\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2"
Jul 27 18:22:52.208930 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:52.208897    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2"
Jul 27 18:22:52.355786 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:52.355739    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2"]
Jul 27 18:22:52.359296 ip-10-0-131-114 kubenswrapper[2557]: W0727 18:22:52.359248    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f661d9_1d71_423c_b330_b5d25ae41662.slice/crio-bbe9374556280f54f0f1a3a00039e9cc47890f3c6e3e394ddd5906bb09b5353b WatchSource:0}: Error finding container bbe9374556280f54f0f1a3a00039e9cc47890f3c6e3e394ddd5906bb09b5353b: Status 404 returned error can't find the container with id bbe9374556280f54f0f1a3a00039e9cc47890f3c6e3e394ddd5906bb09b5353b
Jul 27 18:22:52.701327 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:52.701288    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2" event={"ID":"37f661d9-1d71-423c-b330-b5d25ae41662","Type":"ContainerStarted","Data":"6f41b3b6ad482f49476278eac282ede9f705f63fa7acceb550edfe2d07206065"}
Jul 27 18:22:52.701532 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:52.701349    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2" event={"ID":"37f661d9-1d71-423c-b330-b5d25ae41662","Type":"ContainerStarted","Data":"bbe9374556280f54f0f1a3a00039e9cc47890f3c6e3e394ddd5906bb09b5353b"}
Jul 27 18:22:52.718983 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:52.718925    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2" podStartSLOduration=1.718907861 podStartE2EDuration="1.718907861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:22:52.717134628 +0000 UTC m=+3265.203945915" watchObservedRunningTime="2026-07-27 18:22:52.718907861 +0000 UTC m=+3265.205719147"
Jul 27 18:22:53.209406 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:22:53.209369    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2"
Jul 27 18:23:02.209189 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:23:02.209148    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2"
Jul 27 18:23:02.221499 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:23:02.221470    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2"
Jul 27 18:23:02.759569 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:23:02.759541    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2"
Jul 27 18:24:24.007488 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:24.007450    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2"]
Jul 27 18:24:24.016879 ip-10-0-131-114 kubenswrapper[2557]: E0727 18:24:24.016845    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 18:24:24.016879 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:24:24.016879 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 18:24:24.016879 ip-10-0-131-114 kubenswrapper[2557]:  > containerID="6f41b3b6ad482f49476278eac282ede9f705f63fa7acceb550edfe2d07206065" cmd=["/bin/sleep","15"]
Jul 27 18:24:24.017221 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:24.016897    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2" message="" err=<
Jul 27 18:24:24.017221 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:24:24.017221 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 18:24:24.017221 ip-10-0-131-114 kubenswrapper[2557]:  >
Jul 27 18:24:24.017221 ip-10-0-131-114 kubenswrapper[2557]: E0727 18:24:24.016910    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 18:24:24.017221 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:24:24.017221 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 18:24:24.017221 ip-10-0-131-114 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2" podUID="37f661d9-1d71-423c-b330-b5d25ae41662" containerName="main" containerID="cri-o://6f41b3b6ad482f49476278eac282ede9f705f63fa7acceb550edfe2d07206065"
Jul 27 18:24:24.017221 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:24.016944    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2" podUID="37f661d9-1d71-423c-b330-b5d25ae41662" containerName="main" containerID="cri-o://6f41b3b6ad482f49476278eac282ede9f705f63fa7acceb550edfe2d07206065" gracePeriod=60
Jul 27 18:24:24.369814 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:24.369791    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2"
Jul 27 18:24:24.397551 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:24.397527    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbm4c\" (UniqueName: \"kubernetes.io/projected/37f661d9-1d71-423c-b330-b5d25ae41662-kube-api-access-rbm4c\") pod \"37f661d9-1d71-423c-b330-b5d25ae41662\" (UID: \"37f661d9-1d71-423c-b330-b5d25ae41662\") "
Jul 27 18:24:24.397693 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:24.397582    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f661d9-1d71-423c-b330-b5d25ae41662-tmp-dir\") pod \"37f661d9-1d71-423c-b330-b5d25ae41662\" (UID: \"37f661d9-1d71-423c-b330-b5d25ae41662\") "
Jul 27 18:24:24.397736 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:24.397689    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37f661d9-1d71-423c-b330-b5d25ae41662-home\") pod \"37f661d9-1d71-423c-b330-b5d25ae41662\" (UID: \"37f661d9-1d71-423c-b330-b5d25ae41662\") "
Jul 27 18:24:24.397791 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:24.397738    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37f661d9-1d71-423c-b330-b5d25ae41662-model-cache\") pod \"37f661d9-1d71-423c-b330-b5d25ae41662\" (UID: \"37f661d9-1d71-423c-b330-b5d25ae41662\") "
Jul 27 18:24:24.397791 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:24.397782    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37f661d9-1d71-423c-b330-b5d25ae41662-dshm\") pod \"37f661d9-1d71-423c-b330-b5d25ae41662\" (UID: \"37f661d9-1d71-423c-b330-b5d25ae41662\") "
Jul 27 18:24:24.397891 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:24.397810    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f661d9-1d71-423c-b330-b5d25ae41662-tls-certs\") pod \"37f661d9-1d71-423c-b330-b5d25ae41662\" (UID: \"37f661d9-1d71-423c-b330-b5d25ae41662\") "
Jul 27 18:24:24.397891 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:24.397853    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f661d9-1d71-423c-b330-b5d25ae41662-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "37f661d9-1d71-423c-b330-b5d25ae41662" (UID: "37f661d9-1d71-423c-b330-b5d25ae41662"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:24:24.398004 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:24.397915    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f661d9-1d71-423c-b330-b5d25ae41662-home" (OuterVolumeSpecName: "home") pod "37f661d9-1d71-423c-b330-b5d25ae41662" (UID: "37f661d9-1d71-423c-b330-b5d25ae41662"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:24:24.398140 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:24.398099    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f661d9-1d71-423c-b330-b5d25ae41662-model-cache" (OuterVolumeSpecName: "model-cache") pod "37f661d9-1d71-423c-b330-b5d25ae41662" (UID: "37f661d9-1d71-423c-b330-b5d25ae41662"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:24:24.398140 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:24.398125    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37f661d9-1d71-423c-b330-b5d25ae41662-home\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:24:24.399981 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:24.399955    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f661d9-1d71-423c-b330-b5d25ae41662-kube-api-access-rbm4c" (OuterVolumeSpecName: "kube-api-access-rbm4c") pod "37f661d9-1d71-423c-b330-b5d25ae41662" (UID: "37f661d9-1d71-423c-b330-b5d25ae41662"). InnerVolumeSpecName "kube-api-access-rbm4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:24:24.400096 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:24.400007    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37f661d9-1d71-423c-b330-b5d25ae41662-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37f661d9-1d71-423c-b330-b5d25ae41662" (UID: "37f661d9-1d71-423c-b330-b5d25ae41662"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:24:24.400096 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:24.400031    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f661d9-1d71-423c-b330-b5d25ae41662-dshm" (OuterVolumeSpecName: "dshm") pod "37f661d9-1d71-423c-b330-b5d25ae41662" (UID: "37f661d9-1d71-423c-b330-b5d25ae41662"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:24:24.499277 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:24.499243    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbm4c\" (UniqueName: \"kubernetes.io/projected/37f661d9-1d71-423c-b330-b5d25ae41662-kube-api-access-rbm4c\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:24:24.499277 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:24.499273    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f661d9-1d71-423c-b330-b5d25ae41662-tmp-dir\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:24:24.499277 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:24.499285    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37f661d9-1d71-423c-b330-b5d25ae41662-model-cache\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:24:24.499506 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:24.499293    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37f661d9-1d71-423c-b330-b5d25ae41662-dshm\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:24:24.499506 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:24.499301    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f661d9-1d71-423c-b330-b5d25ae41662-tls-certs\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:24:25.126970 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:25.126940    2557 generic.go:356] "Generic (PLEG): container finished" podID="37f661d9-1d71-423c-b330-b5d25ae41662" containerID="6f41b3b6ad482f49476278eac282ede9f705f63fa7acceb550edfe2d07206065" exitCode=0
Jul 27 18:24:25.127412 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:25.126982    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2"
Jul 27 18:24:25.127412 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:25.127012    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2" event={"ID":"37f661d9-1d71-423c-b330-b5d25ae41662","Type":"ContainerDied","Data":"6f41b3b6ad482f49476278eac282ede9f705f63fa7acceb550edfe2d07206065"}
Jul 27 18:24:25.127412 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:25.127048    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2" event={"ID":"37f661d9-1d71-423c-b330-b5d25ae41662","Type":"ContainerDied","Data":"bbe9374556280f54f0f1a3a00039e9cc47890f3c6e3e394ddd5906bb09b5353b"}
Jul 27 18:24:25.127412 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:25.127068    2557 scope.go:117] "RemoveContainer" containerID="6f41b3b6ad482f49476278eac282ede9f705f63fa7acceb550edfe2d07206065"
Jul 27 18:24:25.136699 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:25.136680    2557 scope.go:117] "RemoveContainer" containerID="6f41b3b6ad482f49476278eac282ede9f705f63fa7acceb550edfe2d07206065"
Jul 27 18:24:25.136989 ip-10-0-131-114 kubenswrapper[2557]: E0727 18:24:25.136971    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f41b3b6ad482f49476278eac282ede9f705f63fa7acceb550edfe2d07206065\": container with ID starting with 6f41b3b6ad482f49476278eac282ede9f705f63fa7acceb550edfe2d07206065 not found: ID does not exist" containerID="6f41b3b6ad482f49476278eac282ede9f705f63fa7acceb550edfe2d07206065"
Jul 27 18:24:25.137039 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:25.136998    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f41b3b6ad482f49476278eac282ede9f705f63fa7acceb550edfe2d07206065"} err="failed to get container status \"6f41b3b6ad482f49476278eac282ede9f705f63fa7acceb550edfe2d07206065\": rpc error: code = NotFound desc = could not find container \"6f41b3b6ad482f49476278eac282ede9f705f63fa7acceb550edfe2d07206065\": container with ID starting with 6f41b3b6ad482f49476278eac282ede9f705f63fa7acceb550edfe2d07206065 not found: ID does not exist"
Jul 27 18:24:25.146600 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:25.146575    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2"]
Jul 27 18:24:25.150227 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:25.150208    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969q2hq2"]
Jul 27 18:24:26.119286 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:26.119257    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37f661d9-1d71-423c-b330-b5d25ae41662" path="/var/lib/kubelet/pods/37f661d9-1d71-423c-b330-b5d25ae41662/volumes"
Jul 27 18:24:47.902374 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:47.902337    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"]
Jul 27 18:24:47.902899 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:47.902814    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37f661d9-1d71-423c-b330-b5d25ae41662" containerName="main"
Jul 27 18:24:47.902899 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:47.902829    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="37f661d9-1d71-423c-b330-b5d25ae41662" containerName="main"
Jul 27 18:24:47.903024 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:47.902951    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="37f661d9-1d71-423c-b330-b5d25ae41662" containerName="main"
Jul 27 18:24:47.906279 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:47.906259    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"
Jul 27 18:24:47.908390 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:47.908368    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 18:24:47.908532 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:47.908470    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-wbgw5\""
Jul 27 18:24:47.908532 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:47.908470    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Jul 27 18:24:47.908532 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:47.908510    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 27 18:24:47.908813 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:47.908605    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Jul 27 18:24:47.908813 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:47.908743    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Jul 27 18:24:47.918285 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:47.918265    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"]
Jul 27 18:24:48.003103 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:48.003075    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62ksm\" (UniqueName: \"kubernetes.io/projected/ad36a931-b94d-4e96-9b5e-370409c5207e-kube-api-access-62ksm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn\" (UID: \"ad36a931-b94d-4e96-9b5e-370409c5207e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"
Jul 27 18:24:48.003296 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:48.003111    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad36a931-b94d-4e96-9b5e-370409c5207e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn\" (UID: \"ad36a931-b94d-4e96-9b5e-370409c5207e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"
Jul 27 18:24:48.003296 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:48.003134    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad36a931-b94d-4e96-9b5e-370409c5207e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn\" (UID: \"ad36a931-b94d-4e96-9b5e-370409c5207e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"
Jul 27 18:24:48.003296 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:48.003153    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad36a931-b94d-4e96-9b5e-370409c5207e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn\" (UID: \"ad36a931-b94d-4e96-9b5e-370409c5207e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"
Jul 27 18:24:48.003444 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:48.003287    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad36a931-b94d-4e96-9b5e-370409c5207e-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn\" (UID: \"ad36a931-b94d-4e96-9b5e-370409c5207e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"
Jul 27 18:24:48.003444 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:48.003328    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad36a931-b94d-4e96-9b5e-370409c5207e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn\" (UID: \"ad36a931-b94d-4e96-9b5e-370409c5207e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"
Jul 27 18:24:48.003444 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:48.003385    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad36a931-b94d-4e96-9b5e-370409c5207e-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn\" (UID: \"ad36a931-b94d-4e96-9b5e-370409c5207e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"
Jul 27 18:24:48.003544 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:48.003445    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad36a931-b94d-4e96-9b5e-370409c5207e-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn\" (UID: \"ad36a931-b94d-4e96-9b5e-370409c5207e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"
Jul 27 18:24:48.104627 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:48.104586    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad36a931-b94d-4e96-9b5e-370409c5207e-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn\" (UID: \"ad36a931-b94d-4e96-9b5e-370409c5207e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"
Jul 27 18:24:48.104627 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:48.104628    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad36a931-b94d-4e96-9b5e-370409c5207e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn\" (UID: \"ad36a931-b94d-4e96-9b5e-370409c5207e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"
Jul 27 18:24:48.104946 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:48.104652    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad36a931-b94d-4e96-9b5e-370409c5207e-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn\" (UID: \"ad36a931-b94d-4e96-9b5e-370409c5207e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"
Jul 27 18:24:48.104946 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:48.104687    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad36a931-b94d-4e96-9b5e-370409c5207e-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn\" (UID: \"ad36a931-b94d-4e96-9b5e-370409c5207e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"
Jul 27 18:24:48.104946 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:48.104741    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62ksm\" (UniqueName: \"kubernetes.io/projected/ad36a931-b94d-4e96-9b5e-370409c5207e-kube-api-access-62ksm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn\" (UID: \"ad36a931-b94d-4e96-9b5e-370409c5207e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"
Jul 27 18:24:48.104946 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:48.104807    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad36a931-b94d-4e96-9b5e-370409c5207e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn\" (UID: \"ad36a931-b94d-4e96-9b5e-370409c5207e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"
Jul 27 18:24:48.104946 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:48.104834    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad36a931-b94d-4e96-9b5e-370409c5207e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn\" (UID: \"ad36a931-b94d-4e96-9b5e-370409c5207e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"
Jul 27 18:24:48.104946 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:48.104859    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad36a931-b94d-4e96-9b5e-370409c5207e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn\" (UID: \"ad36a931-b94d-4e96-9b5e-370409c5207e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"
Jul 27 18:24:48.105258 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:48.105114    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad36a931-b94d-4e96-9b5e-370409c5207e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn\" (UID: \"ad36a931-b94d-4e96-9b5e-370409c5207e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"
Jul 27 18:24:48.105258 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:48.105148    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad36a931-b94d-4e96-9b5e-370409c5207e-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn\" (UID: \"ad36a931-b94d-4e96-9b5e-370409c5207e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"
Jul 27 18:24:48.105258 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:48.105184    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad36a931-b94d-4e96-9b5e-370409c5207e-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn\" (UID: \"ad36a931-b94d-4e96-9b5e-370409c5207e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"
Jul 27 18:24:48.105356 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:48.105262    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad36a931-b94d-4e96-9b5e-370409c5207e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn\" (UID: \"ad36a931-b94d-4e96-9b5e-370409c5207e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"
Jul 27 18:24:48.105728 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:48.105708    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad36a931-b94d-4e96-9b5e-370409c5207e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn\" (UID: \"ad36a931-b94d-4e96-9b5e-370409c5207e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"
Jul 27 18:24:48.107008 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:48.106991    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad36a931-b94d-4e96-9b5e-370409c5207e-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn\" (UID: \"ad36a931-b94d-4e96-9b5e-370409c5207e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"
Jul 27 18:24:48.107437 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:48.107415    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad36a931-b94d-4e96-9b5e-370409c5207e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn\" (UID: \"ad36a931-b94d-4e96-9b5e-370409c5207e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"
Jul 27 18:24:48.112247 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:48.112224    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62ksm\" (UniqueName: \"kubernetes.io/projected/ad36a931-b94d-4e96-9b5e-370409c5207e-kube-api-access-62ksm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn\" (UID: \"ad36a931-b94d-4e96-9b5e-370409c5207e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"
Jul 27 18:24:48.216847 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:48.216815    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"
Jul 27 18:24:48.352089 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:48.352065    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"]
Jul 27 18:24:48.354605 ip-10-0-131-114 kubenswrapper[2557]: W0727 18:24:48.354577    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad36a931_b94d_4e96_9b5e_370409c5207e.slice/crio-5ea9c482c69743318cb9461347743982d7d336895879639b4e2db91115eb09a3 WatchSource:0}: Error finding container 5ea9c482c69743318cb9461347743982d7d336895879639b4e2db91115eb09a3: Status 404 returned error can't find the container with id 5ea9c482c69743318cb9461347743982d7d336895879639b4e2db91115eb09a3
Jul 27 18:24:49.248019 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:49.247979    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn" event={"ID":"ad36a931-b94d-4e96-9b5e-370409c5207e","Type":"ContainerStarted","Data":"c8afe77e74c528d55a65b3b90d73605094f3bf37477e10201f29cc08aa694094"}
Jul 27 18:24:49.248406 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:49.248047    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn" event={"ID":"ad36a931-b94d-4e96-9b5e-370409c5207e","Type":"ContainerStarted","Data":"5ea9c482c69743318cb9461347743982d7d336895879639b4e2db91115eb09a3"}
Jul 27 18:24:56.283520 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:56.283483    2557 generic.go:356] "Generic (PLEG): container finished" podID="ad36a931-b94d-4e96-9b5e-370409c5207e" containerID="c8afe77e74c528d55a65b3b90d73605094f3bf37477e10201f29cc08aa694094" exitCode=0
Jul 27 18:24:56.283937 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:56.283549    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn" event={"ID":"ad36a931-b94d-4e96-9b5e-370409c5207e","Type":"ContainerDied","Data":"c8afe77e74c528d55a65b3b90d73605094f3bf37477e10201f29cc08aa694094"}
Jul 27 18:24:57.290241 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:57.290209    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn" event={"ID":"ad36a931-b94d-4e96-9b5e-370409c5207e","Type":"ContainerStarted","Data":"10c6c2416b820eb4f843de5c0ed8c8a23cefab5611ea7d6ecea432539c469360"}
Jul 27 18:24:57.314118 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:57.314065    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn" podStartSLOduration=2.615479459 podStartE2EDuration="10.314049236s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.698569777s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:24:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:24:57.312693462 +0000 UTC m=+3389.799504861" watchObservedRunningTime="2026-07-27 18:24:57.314049236 +0000 UTC m=+3389.800860523"
Jul 27 18:24:58.217826 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:58.217788    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"
Jul 27 18:24:58.218036 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:58.217863    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"
Jul 27 18:24:58.219277 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:24:58.219246    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn" podUID="ad36a931-b94d-4e96-9b5e-370409c5207e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 27 18:25:08.217686 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:25:08.217650    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn" podUID="ad36a931-b94d-4e96-9b5e-370409c5207e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 27 18:25:18.217497 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:25:18.217451    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn" podUID="ad36a931-b94d-4e96-9b5e-370409c5207e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 27 18:25:28.217350 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:25:28.217256    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn" podUID="ad36a931-b94d-4e96-9b5e-370409c5207e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 27 18:25:38.217891 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:25:38.217841    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn" podUID="ad36a931-b94d-4e96-9b5e-370409c5207e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 27 18:25:48.218137 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:25:48.218098    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn" podUID="ad36a931-b94d-4e96-9b5e-370409c5207e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 27 18:25:58.217266 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:25:58.217226    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn" podUID="ad36a931-b94d-4e96-9b5e-370409c5207e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 27 18:26:08.218547 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:08.218461    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn" podUID="ad36a931-b94d-4e96-9b5e-370409c5207e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 27 18:26:18.228016 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:18.227980    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"
Jul 27 18:26:18.236174 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:18.236153    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"
Jul 27 18:26:43.295692 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:43.295655    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"]
Jul 27 18:26:58.320438 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:58.320398    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn" podUID="ad36a931-b94d-4e96-9b5e-370409c5207e" containerName="main" containerID="cri-o://10c6c2416b820eb4f843de5c0ed8c8a23cefab5611ea7d6ecea432539c469360" gracePeriod=45
Jul 27 18:26:58.882173 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:58.882133    2557 generic.go:356] "Generic (PLEG): container finished" podID="ad36a931-b94d-4e96-9b5e-370409c5207e" containerID="10c6c2416b820eb4f843de5c0ed8c8a23cefab5611ea7d6ecea432539c469360" exitCode=0
Jul 27 18:26:58.882331 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:58.882201    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn" event={"ID":"ad36a931-b94d-4e96-9b5e-370409c5207e","Type":"ContainerDied","Data":"10c6c2416b820eb4f843de5c0ed8c8a23cefab5611ea7d6ecea432539c469360"}
Jul 27 18:26:58.980495 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:58.980469    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"
Jul 27 18:26:59.057471 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:59.057394    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad36a931-b94d-4e96-9b5e-370409c5207e-kserve-provision-location\") pod \"ad36a931-b94d-4e96-9b5e-370409c5207e\" (UID: \"ad36a931-b94d-4e96-9b5e-370409c5207e\") "
Jul 27 18:26:59.057471 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:59.057448    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad36a931-b94d-4e96-9b5e-370409c5207e-dshm\") pod \"ad36a931-b94d-4e96-9b5e-370409c5207e\" (UID: \"ad36a931-b94d-4e96-9b5e-370409c5207e\") "
Jul 27 18:26:59.057780 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:59.057482    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad36a931-b94d-4e96-9b5e-370409c5207e-model-cache\") pod \"ad36a931-b94d-4e96-9b5e-370409c5207e\" (UID: \"ad36a931-b94d-4e96-9b5e-370409c5207e\") "
Jul 27 18:26:59.057780 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:59.057510    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62ksm\" (UniqueName: \"kubernetes.io/projected/ad36a931-b94d-4e96-9b5e-370409c5207e-kube-api-access-62ksm\") pod \"ad36a931-b94d-4e96-9b5e-370409c5207e\" (UID: \"ad36a931-b94d-4e96-9b5e-370409c5207e\") "
Jul 27 18:26:59.057780 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:59.057527    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad36a931-b94d-4e96-9b5e-370409c5207e-tls-certs\") pod \"ad36a931-b94d-4e96-9b5e-370409c5207e\" (UID: \"ad36a931-b94d-4e96-9b5e-370409c5207e\") "
Jul 27 18:26:59.057780 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:59.057542    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad36a931-b94d-4e96-9b5e-370409c5207e-tmp-dir\") pod \"ad36a931-b94d-4e96-9b5e-370409c5207e\" (UID: \"ad36a931-b94d-4e96-9b5e-370409c5207e\") "
Jul 27 18:26:59.057780 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:59.057592    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad36a931-b94d-4e96-9b5e-370409c5207e-home\") pod \"ad36a931-b94d-4e96-9b5e-370409c5207e\" (UID: \"ad36a931-b94d-4e96-9b5e-370409c5207e\") "
Jul 27 18:26:59.057780 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:59.057629    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad36a931-b94d-4e96-9b5e-370409c5207e-cabundle-cert\") pod \"ad36a931-b94d-4e96-9b5e-370409c5207e\" (UID: \"ad36a931-b94d-4e96-9b5e-370409c5207e\") "
Jul 27 18:26:59.058119 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:59.057836    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad36a931-b94d-4e96-9b5e-370409c5207e-model-cache" (OuterVolumeSpecName: "model-cache") pod "ad36a931-b94d-4e96-9b5e-370409c5207e" (UID: "ad36a931-b94d-4e96-9b5e-370409c5207e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:26:59.058119 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:59.057984    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad36a931-b94d-4e96-9b5e-370409c5207e-model-cache\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:26:59.058229 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:59.058115    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad36a931-b94d-4e96-9b5e-370409c5207e-home" (OuterVolumeSpecName: "home") pod "ad36a931-b94d-4e96-9b5e-370409c5207e" (UID: "ad36a931-b94d-4e96-9b5e-370409c5207e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:26:59.058229 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:59.058130    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad36a931-b94d-4e96-9b5e-370409c5207e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ad36a931-b94d-4e96-9b5e-370409c5207e" (UID: "ad36a931-b94d-4e96-9b5e-370409c5207e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:26:59.059889 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:59.059861    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad36a931-b94d-4e96-9b5e-370409c5207e-dshm" (OuterVolumeSpecName: "dshm") pod "ad36a931-b94d-4e96-9b5e-370409c5207e" (UID: "ad36a931-b94d-4e96-9b5e-370409c5207e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:26:59.060109 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:59.060075    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad36a931-b94d-4e96-9b5e-370409c5207e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ad36a931-b94d-4e96-9b5e-370409c5207e" (UID: "ad36a931-b94d-4e96-9b5e-370409c5207e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:26:59.060228 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:59.060202    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad36a931-b94d-4e96-9b5e-370409c5207e-kube-api-access-62ksm" (OuterVolumeSpecName: "kube-api-access-62ksm") pod "ad36a931-b94d-4e96-9b5e-370409c5207e" (UID: "ad36a931-b94d-4e96-9b5e-370409c5207e"). InnerVolumeSpecName "kube-api-access-62ksm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:26:59.068270 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:59.068244    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad36a931-b94d-4e96-9b5e-370409c5207e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ad36a931-b94d-4e96-9b5e-370409c5207e" (UID: "ad36a931-b94d-4e96-9b5e-370409c5207e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:26:59.117165 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:59.117137    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad36a931-b94d-4e96-9b5e-370409c5207e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ad36a931-b94d-4e96-9b5e-370409c5207e" (UID: "ad36a931-b94d-4e96-9b5e-370409c5207e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:26:59.158376 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:59.158342    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62ksm\" (UniqueName: \"kubernetes.io/projected/ad36a931-b94d-4e96-9b5e-370409c5207e-kube-api-access-62ksm\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:26:59.158376 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:59.158375    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad36a931-b94d-4e96-9b5e-370409c5207e-tls-certs\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:26:59.158505 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:59.158403    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad36a931-b94d-4e96-9b5e-370409c5207e-tmp-dir\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:26:59.158505 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:59.158412    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad36a931-b94d-4e96-9b5e-370409c5207e-home\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:26:59.158505 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:59.158420    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad36a931-b94d-4e96-9b5e-370409c5207e-cabundle-cert\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:26:59.158505 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:59.158428    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad36a931-b94d-4e96-9b5e-370409c5207e-kserve-provision-location\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:26:59.158505 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:59.158435    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad36a931-b94d-4e96-9b5e-370409c5207e-dshm\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:26:59.888362 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:59.888327    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn" event={"ID":"ad36a931-b94d-4e96-9b5e-370409c5207e","Type":"ContainerDied","Data":"5ea9c482c69743318cb9461347743982d7d336895879639b4e2db91115eb09a3"}
Jul 27 18:26:59.888827 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:59.888386    2557 scope.go:117] "RemoveContainer" containerID="10c6c2416b820eb4f843de5c0ed8c8a23cefab5611ea7d6ecea432539c469360"
Jul 27 18:26:59.888827 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:59.888343    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"
Jul 27 18:26:59.898552 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:59.898535    2557 scope.go:117] "RemoveContainer" containerID="c8afe77e74c528d55a65b3b90d73605094f3bf37477e10201f29cc08aa694094"
Jul 27 18:26:59.910844 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:59.910822    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"]
Jul 27 18:26:59.916288 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:26:59.916267    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-85bf74c68-8v6tn"]
Jul 27 18:27:00.118918 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:00.118886    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad36a931-b94d-4e96-9b5e-370409c5207e" path="/var/lib/kubelet/pods/ad36a931-b94d-4e96-9b5e-370409c5207e/volumes"
Jul 27 18:27:13.386703 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.386669    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"]
Jul 27 18:27:13.387269 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.387199    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad36a931-b94d-4e96-9b5e-370409c5207e" containerName="main"
Jul 27 18:27:13.387269 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.387215    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad36a931-b94d-4e96-9b5e-370409c5207e" containerName="main"
Jul 27 18:27:13.387269 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.387238    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad36a931-b94d-4e96-9b5e-370409c5207e" containerName="storage-initializer"
Jul 27 18:27:13.387269 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.387248    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad36a931-b94d-4e96-9b5e-370409c5207e" containerName="storage-initializer"
Jul 27 18:27:13.387494 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.387341    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad36a931-b94d-4e96-9b5e-370409c5207e" containerName="main"
Jul 27 18:27:13.390638 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.390614    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"
Jul 27 18:27:13.393030 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.393004    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 27 18:27:13.393158 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.393044    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-s78vq\""
Jul 27 18:27:13.393158 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.393140    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Jul 27 18:27:13.393483 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.393464    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Jul 27 18:27:13.393609 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.393471    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Jul 27 18:27:13.393609 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.393566    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 18:27:13.403616 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.403588    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"]
Jul 27 18:27:13.483392 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.483361    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96cb17eb-1c38-4131-987a-67dc993ae687-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z\" (UID: \"96cb17eb-1c38-4131-987a-67dc993ae687\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"
Jul 27 18:27:13.483578 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.483405    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96cb17eb-1c38-4131-987a-67dc993ae687-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z\" (UID: \"96cb17eb-1c38-4131-987a-67dc993ae687\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"
Jul 27 18:27:13.483578 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.483439    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96cb17eb-1c38-4131-987a-67dc993ae687-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z\" (UID: \"96cb17eb-1c38-4131-987a-67dc993ae687\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"
Jul 27 18:27:13.483578 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.483461    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96cb17eb-1c38-4131-987a-67dc993ae687-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z\" (UID: \"96cb17eb-1c38-4131-987a-67dc993ae687\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"
Jul 27 18:27:13.483578 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.483483    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96cb17eb-1c38-4131-987a-67dc993ae687-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z\" (UID: \"96cb17eb-1c38-4131-987a-67dc993ae687\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"
Jul 27 18:27:13.483578 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.483511    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96cb17eb-1c38-4131-987a-67dc993ae687-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z\" (UID: \"96cb17eb-1c38-4131-987a-67dc993ae687\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"
Jul 27 18:27:13.483578 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.483535    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96cb17eb-1c38-4131-987a-67dc993ae687-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z\" (UID: \"96cb17eb-1c38-4131-987a-67dc993ae687\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"
Jul 27 18:27:13.483843 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.483619    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tddzp\" (UniqueName: \"kubernetes.io/projected/96cb17eb-1c38-4131-987a-67dc993ae687-kube-api-access-tddzp\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z\" (UID: \"96cb17eb-1c38-4131-987a-67dc993ae687\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"
Jul 27 18:27:13.584775 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.584727    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96cb17eb-1c38-4131-987a-67dc993ae687-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z\" (UID: \"96cb17eb-1c38-4131-987a-67dc993ae687\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"
Jul 27 18:27:13.584775 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.584775    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96cb17eb-1c38-4131-987a-67dc993ae687-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z\" (UID: \"96cb17eb-1c38-4131-987a-67dc993ae687\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"
Jul 27 18:27:13.585010 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.584805    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96cb17eb-1c38-4131-987a-67dc993ae687-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z\" (UID: \"96cb17eb-1c38-4131-987a-67dc993ae687\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"
Jul 27 18:27:13.585010 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.584836    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96cb17eb-1c38-4131-987a-67dc993ae687-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z\" (UID: \"96cb17eb-1c38-4131-987a-67dc993ae687\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"
Jul 27 18:27:13.585010 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.584862    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96cb17eb-1c38-4131-987a-67dc993ae687-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z\" (UID: \"96cb17eb-1c38-4131-987a-67dc993ae687\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"
Jul 27 18:27:13.585174 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.585007    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tddzp\" (UniqueName: \"kubernetes.io/projected/96cb17eb-1c38-4131-987a-67dc993ae687-kube-api-access-tddzp\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z\" (UID: \"96cb17eb-1c38-4131-987a-67dc993ae687\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"
Jul 27 18:27:13.585174 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.585077    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96cb17eb-1c38-4131-987a-67dc993ae687-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z\" (UID: \"96cb17eb-1c38-4131-987a-67dc993ae687\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"
Jul 27 18:27:13.585174 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.585129    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96cb17eb-1c38-4131-987a-67dc993ae687-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z\" (UID: \"96cb17eb-1c38-4131-987a-67dc993ae687\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"
Jul 27 18:27:13.585174 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.585144    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96cb17eb-1c38-4131-987a-67dc993ae687-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z\" (UID: \"96cb17eb-1c38-4131-987a-67dc993ae687\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"
Jul 27 18:27:13.585391 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.585186    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96cb17eb-1c38-4131-987a-67dc993ae687-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z\" (UID: \"96cb17eb-1c38-4131-987a-67dc993ae687\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"
Jul 27 18:27:13.585468 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.585447    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96cb17eb-1c38-4131-987a-67dc993ae687-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z\" (UID: \"96cb17eb-1c38-4131-987a-67dc993ae687\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"
Jul 27 18:27:13.585532 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.585494    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96cb17eb-1c38-4131-987a-67dc993ae687-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z\" (UID: \"96cb17eb-1c38-4131-987a-67dc993ae687\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"
Jul 27 18:27:13.585618 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.585601    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96cb17eb-1c38-4131-987a-67dc993ae687-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z\" (UID: \"96cb17eb-1c38-4131-987a-67dc993ae687\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"
Jul 27 18:27:13.587202 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.587181    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96cb17eb-1c38-4131-987a-67dc993ae687-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z\" (UID: \"96cb17eb-1c38-4131-987a-67dc993ae687\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"
Jul 27 18:27:13.587589 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.587568    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96cb17eb-1c38-4131-987a-67dc993ae687-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z\" (UID: \"96cb17eb-1c38-4131-987a-67dc993ae687\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"
Jul 27 18:27:13.593077 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.593045    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tddzp\" (UniqueName: \"kubernetes.io/projected/96cb17eb-1c38-4131-987a-67dc993ae687-kube-api-access-tddzp\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z\" (UID: \"96cb17eb-1c38-4131-987a-67dc993ae687\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"
Jul 27 18:27:13.703188 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.703149    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"
Jul 27 18:27:13.835433 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.835410    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"]
Jul 27 18:27:13.837492 ip-10-0-131-114 kubenswrapper[2557]: W0727 18:27:13.837464    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96cb17eb_1c38_4131_987a_67dc993ae687.slice/crio-9c01df470d197296fb6d01711b8b46096a763546855d28fba60a36b62674e5e5 WatchSource:0}: Error finding container 9c01df470d197296fb6d01711b8b46096a763546855d28fba60a36b62674e5e5: Status 404 returned error can't find the container with id 9c01df470d197296fb6d01711b8b46096a763546855d28fba60a36b62674e5e5
Jul 27 18:27:13.961154 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.961077    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z" event={"ID":"96cb17eb-1c38-4131-987a-67dc993ae687","Type":"ContainerStarted","Data":"989c8b8e4e0679da4688bd93d6e7da7c78bc3a2ad303b946d137079a6d764b10"}
Jul 27 18:27:13.961154 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:13.961135    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z" event={"ID":"96cb17eb-1c38-4131-987a-67dc993ae687","Type":"ContainerStarted","Data":"9c01df470d197296fb6d01711b8b46096a763546855d28fba60a36b62674e5e5"}
Jul 27 18:27:20.997496 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:20.997460    2557 generic.go:356] "Generic (PLEG): container finished" podID="96cb17eb-1c38-4131-987a-67dc993ae687" containerID="989c8b8e4e0679da4688bd93d6e7da7c78bc3a2ad303b946d137079a6d764b10" exitCode=0
Jul 27 18:27:20.997949 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:20.997541    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z" event={"ID":"96cb17eb-1c38-4131-987a-67dc993ae687","Type":"ContainerDied","Data":"989c8b8e4e0679da4688bd93d6e7da7c78bc3a2ad303b946d137079a6d764b10"}
Jul 27 18:27:22.004797 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:22.004747    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z" event={"ID":"96cb17eb-1c38-4131-987a-67dc993ae687","Type":"ContainerStarted","Data":"5cdb5d1270caf6161a2dc13d02e7f7441fcdb4b9389b57982e829513690ffde6"}
Jul 27 18:27:22.025651 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:22.025588    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z" podStartSLOduration=2.280851327 podStartE2EDuration="9.02556608s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.744714753s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:27:22.023500259 +0000 UTC m=+3534.510311580" watchObservedRunningTime="2026-07-27 18:27:22.02556608 +0000 UTC m=+3534.512377368"
Jul 27 18:27:22.703846 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:22.703811    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"
Jul 27 18:27:23.704274 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:23.704245    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"
Jul 27 18:27:23.706120 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:23.706090    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z" podUID="96cb17eb-1c38-4131-987a-67dc993ae687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 27 18:27:33.703505 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:33.703461    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z" podUID="96cb17eb-1c38-4131-987a-67dc993ae687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 27 18:27:43.703848 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:43.703805    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z" podUID="96cb17eb-1c38-4131-987a-67dc993ae687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 27 18:27:53.704572 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:27:53.704525    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z" podUID="96cb17eb-1c38-4131-987a-67dc993ae687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 27 18:28:03.704315 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:28:03.704274    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z" podUID="96cb17eb-1c38-4131-987a-67dc993ae687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 27 18:28:13.703663 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:28:13.703612    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z" podUID="96cb17eb-1c38-4131-987a-67dc993ae687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 27 18:28:23.703667 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:28:23.703577    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z" podUID="96cb17eb-1c38-4131-987a-67dc993ae687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 27 18:28:33.704364 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:28:33.704321    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z" podUID="96cb17eb-1c38-4131-987a-67dc993ae687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 27 18:28:43.713853 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:28:43.713819    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"
Jul 27 18:28:43.722999 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:28:43.722975    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"
Jul 27 18:28:45.486805 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:28:45.486770    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"]
Jul 27 18:29:00.510445 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:00.510389    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z" podUID="96cb17eb-1c38-4131-987a-67dc993ae687" containerName="main" containerID="cri-o://5cdb5d1270caf6161a2dc13d02e7f7441fcdb4b9389b57982e829513690ffde6" gracePeriod=45
Jul 27 18:29:00.714963 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:00.714914    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z" podUID="96cb17eb-1c38-4131-987a-67dc993ae687" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Jul 27 18:29:01.061115 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.061092    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"
Jul 27 18:29:01.170289 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.170268    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96cb17eb-1c38-4131-987a-67dc993ae687-model-cache\") pod \"96cb17eb-1c38-4131-987a-67dc993ae687\" (UID: \"96cb17eb-1c38-4131-987a-67dc993ae687\") "
Jul 27 18:29:01.170415 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.170331    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96cb17eb-1c38-4131-987a-67dc993ae687-dshm\") pod \"96cb17eb-1c38-4131-987a-67dc993ae687\" (UID: \"96cb17eb-1c38-4131-987a-67dc993ae687\") "
Jul 27 18:29:01.170415 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.170367    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96cb17eb-1c38-4131-987a-67dc993ae687-cabundle-cert\") pod \"96cb17eb-1c38-4131-987a-67dc993ae687\" (UID: \"96cb17eb-1c38-4131-987a-67dc993ae687\") "
Jul 27 18:29:01.170415 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.170383    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tddzp\" (UniqueName: \"kubernetes.io/projected/96cb17eb-1c38-4131-987a-67dc993ae687-kube-api-access-tddzp\") pod \"96cb17eb-1c38-4131-987a-67dc993ae687\" (UID: \"96cb17eb-1c38-4131-987a-67dc993ae687\") "
Jul 27 18:29:01.170415 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.170407    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96cb17eb-1c38-4131-987a-67dc993ae687-tmp-dir\") pod \"96cb17eb-1c38-4131-987a-67dc993ae687\" (UID: \"96cb17eb-1c38-4131-987a-67dc993ae687\") "
Jul 27 18:29:01.170642 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.170428    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96cb17eb-1c38-4131-987a-67dc993ae687-kserve-provision-location\") pod \"96cb17eb-1c38-4131-987a-67dc993ae687\" (UID: \"96cb17eb-1c38-4131-987a-67dc993ae687\") "
Jul 27 18:29:01.170642 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.170460    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96cb17eb-1c38-4131-987a-67dc993ae687-tls-certs\") pod \"96cb17eb-1c38-4131-987a-67dc993ae687\" (UID: \"96cb17eb-1c38-4131-987a-67dc993ae687\") "
Jul 27 18:29:01.170642 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.170502    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96cb17eb-1c38-4131-987a-67dc993ae687-home\") pod \"96cb17eb-1c38-4131-987a-67dc993ae687\" (UID: \"96cb17eb-1c38-4131-987a-67dc993ae687\") "
Jul 27 18:29:01.170642 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.170532    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cb17eb-1c38-4131-987a-67dc993ae687-model-cache" (OuterVolumeSpecName: "model-cache") pod "96cb17eb-1c38-4131-987a-67dc993ae687" (UID: "96cb17eb-1c38-4131-987a-67dc993ae687"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:29:01.170895 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.170792    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96cb17eb-1c38-4131-987a-67dc993ae687-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "96cb17eb-1c38-4131-987a-67dc993ae687" (UID: "96cb17eb-1c38-4131-987a-67dc993ae687"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:29:01.170895 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.170849    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96cb17eb-1c38-4131-987a-67dc993ae687-model-cache\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:29:01.170895 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.170845    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cb17eb-1c38-4131-987a-67dc993ae687-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "96cb17eb-1c38-4131-987a-67dc993ae687" (UID: "96cb17eb-1c38-4131-987a-67dc993ae687"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:29:01.171163 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.171145    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cb17eb-1c38-4131-987a-67dc993ae687-home" (OuterVolumeSpecName: "home") pod "96cb17eb-1c38-4131-987a-67dc993ae687" (UID: "96cb17eb-1c38-4131-987a-67dc993ae687"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:29:01.172674 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.172648    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96cb17eb-1c38-4131-987a-67dc993ae687-kube-api-access-tddzp" (OuterVolumeSpecName: "kube-api-access-tddzp") pod "96cb17eb-1c38-4131-987a-67dc993ae687" (UID: "96cb17eb-1c38-4131-987a-67dc993ae687"). InnerVolumeSpecName "kube-api-access-tddzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:29:01.172998 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.172978    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96cb17eb-1c38-4131-987a-67dc993ae687-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "96cb17eb-1c38-4131-987a-67dc993ae687" (UID: "96cb17eb-1c38-4131-987a-67dc993ae687"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:29:01.173083 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.173006    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cb17eb-1c38-4131-987a-67dc993ae687-dshm" (OuterVolumeSpecName: "dshm") pod "96cb17eb-1c38-4131-987a-67dc993ae687" (UID: "96cb17eb-1c38-4131-987a-67dc993ae687"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:29:01.231355 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.231316    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cb17eb-1c38-4131-987a-67dc993ae687-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "96cb17eb-1c38-4131-987a-67dc993ae687" (UID: "96cb17eb-1c38-4131-987a-67dc993ae687"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:29:01.272937 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.272791    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96cb17eb-1c38-4131-987a-67dc993ae687-cabundle-cert\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:29:01.272937 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.272817    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tddzp\" (UniqueName: \"kubernetes.io/projected/96cb17eb-1c38-4131-987a-67dc993ae687-kube-api-access-tddzp\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:29:01.272937 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.272831    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96cb17eb-1c38-4131-987a-67dc993ae687-tmp-dir\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:29:01.272937 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.272847    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96cb17eb-1c38-4131-987a-67dc993ae687-kserve-provision-location\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:29:01.272937 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.272860    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96cb17eb-1c38-4131-987a-67dc993ae687-tls-certs\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:29:01.272937 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.272874    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96cb17eb-1c38-4131-987a-67dc993ae687-home\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:29:01.272937 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.272886    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96cb17eb-1c38-4131-987a-67dc993ae687-dshm\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:29:01.488841 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.488738    2557 generic.go:356] "Generic (PLEG): container finished" podID="96cb17eb-1c38-4131-987a-67dc993ae687" containerID="5cdb5d1270caf6161a2dc13d02e7f7441fcdb4b9389b57982e829513690ffde6" exitCode=0
Jul 27 18:29:01.488841 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.488806    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"
Jul 27 18:29:01.488841 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.488822    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z" event={"ID":"96cb17eb-1c38-4131-987a-67dc993ae687","Type":"ContainerDied","Data":"5cdb5d1270caf6161a2dc13d02e7f7441fcdb4b9389b57982e829513690ffde6"}
Jul 27 18:29:01.489106 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.488879    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z" event={"ID":"96cb17eb-1c38-4131-987a-67dc993ae687","Type":"ContainerDied","Data":"9c01df470d197296fb6d01711b8b46096a763546855d28fba60a36b62674e5e5"}
Jul 27 18:29:01.489106 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.488902    2557 scope.go:117] "RemoveContainer" containerID="5cdb5d1270caf6161a2dc13d02e7f7441fcdb4b9389b57982e829513690ffde6"
Jul 27 18:29:01.497890 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.497871    2557 scope.go:117] "RemoveContainer" containerID="989c8b8e4e0679da4688bd93d6e7da7c78bc3a2ad303b946d137079a6d764b10"
Jul 27 18:29:01.505935 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.505919    2557 scope.go:117] "RemoveContainer" containerID="5cdb5d1270caf6161a2dc13d02e7f7441fcdb4b9389b57982e829513690ffde6"
Jul 27 18:29:01.506171 ip-10-0-131-114 kubenswrapper[2557]: E0727 18:29:01.506153    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cdb5d1270caf6161a2dc13d02e7f7441fcdb4b9389b57982e829513690ffde6\": container with ID starting with 5cdb5d1270caf6161a2dc13d02e7f7441fcdb4b9389b57982e829513690ffde6 not found: ID does not exist" containerID="5cdb5d1270caf6161a2dc13d02e7f7441fcdb4b9389b57982e829513690ffde6"
Jul 27 18:29:01.506210 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.506180    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cdb5d1270caf6161a2dc13d02e7f7441fcdb4b9389b57982e829513690ffde6"} err="failed to get container status \"5cdb5d1270caf6161a2dc13d02e7f7441fcdb4b9389b57982e829513690ffde6\": rpc error: code = NotFound desc = could not find container \"5cdb5d1270caf6161a2dc13d02e7f7441fcdb4b9389b57982e829513690ffde6\": container with ID starting with 5cdb5d1270caf6161a2dc13d02e7f7441fcdb4b9389b57982e829513690ffde6 not found: ID does not exist"
Jul 27 18:29:01.506210 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.506197    2557 scope.go:117] "RemoveContainer" containerID="989c8b8e4e0679da4688bd93d6e7da7c78bc3a2ad303b946d137079a6d764b10"
Jul 27 18:29:01.506378 ip-10-0-131-114 kubenswrapper[2557]: E0727 18:29:01.506363    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"989c8b8e4e0679da4688bd93d6e7da7c78bc3a2ad303b946d137079a6d764b10\": container with ID starting with 989c8b8e4e0679da4688bd93d6e7da7c78bc3a2ad303b946d137079a6d764b10 not found: ID does not exist" containerID="989c8b8e4e0679da4688bd93d6e7da7c78bc3a2ad303b946d137079a6d764b10"
Jul 27 18:29:01.506426 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.506381    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"989c8b8e4e0679da4688bd93d6e7da7c78bc3a2ad303b946d137079a6d764b10"} err="failed to get container status \"989c8b8e4e0679da4688bd93d6e7da7c78bc3a2ad303b946d137079a6d764b10\": rpc error: code = NotFound desc = could not find container \"989c8b8e4e0679da4688bd93d6e7da7c78bc3a2ad303b946d137079a6d764b10\": container with ID starting with 989c8b8e4e0679da4688bd93d6e7da7c78bc3a2ad303b946d137079a6d764b10 not found: ID does not exist"
Jul 27 18:29:01.509718 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.509697    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"]
Jul 27 18:29:01.515180 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:01.515155    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-76484c89d5-k549z"]
Jul 27 18:29:02.118591 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:02.118552    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96cb17eb-1c38-4131-987a-67dc993ae687" path="/var/lib/kubelet/pods/96cb17eb-1c38-4131-987a-67dc993ae687/volumes"
Jul 27 18:29:08.647665 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:08.647620    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-vmqtz"]
Jul 27 18:29:08.648176 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:08.648156    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96cb17eb-1c38-4131-987a-67dc993ae687" containerName="storage-initializer"
Jul 27 18:29:08.648225 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:08.648181    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="96cb17eb-1c38-4131-987a-67dc993ae687" containerName="storage-initializer"
Jul 27 18:29:08.648225 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:08.648216    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96cb17eb-1c38-4131-987a-67dc993ae687" containerName="main"
Jul 27 18:29:08.648304 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:08.648225    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="96cb17eb-1c38-4131-987a-67dc993ae687" containerName="main"
Jul 27 18:29:08.648345 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:08.648313    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="96cb17eb-1c38-4131-987a-67dc993ae687" containerName="main"
Jul 27 18:29:08.655744 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:08.655721    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-vmqtz"
Jul 27 18:29:08.657106 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:08.657071    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-vmqtz"]
Jul 27 18:29:08.657922 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:08.657830    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"kube-root-ca.crt\""
Jul 27 18:29:08.658017 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:08.657935    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"openshift-service-ca.crt\""
Jul 27 18:29:08.658017 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:08.657967    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"default-dockercfg-gz7n6\""
Jul 27 18:29:08.732851 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:08.732820    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzzxr\" (UniqueName: \"kubernetes.io/projected/b59d1545-ea1d-4592-a659-d89e11e74b32-kube-api-access-mzzxr\") pod \"e2e-pvc-model-download-vmqtz\" (UID: \"b59d1545-ea1d-4592-a659-d89e11e74b32\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-vmqtz"
Jul 27 18:29:08.833493 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:08.833458    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzzxr\" (UniqueName: \"kubernetes.io/projected/b59d1545-ea1d-4592-a659-d89e11e74b32-kube-api-access-mzzxr\") pod \"e2e-pvc-model-download-vmqtz\" (UID: \"b59d1545-ea1d-4592-a659-d89e11e74b32\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-vmqtz"
Jul 27 18:29:08.844507 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:08.844478    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzzxr\" (UniqueName: \"kubernetes.io/projected/b59d1545-ea1d-4592-a659-d89e11e74b32-kube-api-access-mzzxr\") pod \"e2e-pvc-model-download-vmqtz\" (UID: \"b59d1545-ea1d-4592-a659-d89e11e74b32\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-vmqtz"
Jul 27 18:29:11.154633 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:11.154593    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-fa64c5a0-6125-4afb-a25e-65b95b8d77df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0836558a9ec859b57\") pod \"e2e-pvc-model-download-vmqtz\" (UID: \"b59d1545-ea1d-4592-a659-d89e11e74b32\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-vmqtz"
Jul 27 18:29:11.159441 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:11.159420    2557 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-fa64c5a0-6125-4afb-a25e-65b95b8d77df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0836558a9ec859b57\") pod \"e2e-pvc-model-download-vmqtz\" (UID: \"b59d1545-ea1d-4592-a659-d89e11e74b32\") device path: \"\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-vmqtz"
Jul 27 18:29:11.255025 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:11.254998    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-fa64c5a0-6125-4afb-a25e-65b95b8d77df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0836558a9ec859b57\") pod \"e2e-pvc-model-download-vmqtz\" (UID: \"b59d1545-ea1d-4592-a659-d89e11e74b32\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-vmqtz"
Jul 27 18:29:11.255154 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:11.255125    2557 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-fa64c5a0-6125-4afb-a25e-65b95b8d77df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0836558a9ec859b57\") pod \"e2e-pvc-model-download-vmqtz\" (UID: \"b59d1545-ea1d-4592-a659-d89e11e74b32\") DevicePath \"\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-vmqtz"
Jul 27 18:29:11.257208 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:11.257191    2557 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-fa64c5a0-6125-4afb-a25e-65b95b8d77df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0836558a9ec859b57\") pod \"e2e-pvc-model-download-vmqtz\" (UID: \"b59d1545-ea1d-4592-a659-d89e11e74b32\") DevicePath \"csi-7c51cd07b865de6f0f6ab6589f6ed36f4d3288357e2f30aa692c6def4a0263a0\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-vmqtz"
Jul 27 18:29:11.427510 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:11.427474    2557 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-fa64c5a0-6125-4afb-a25e-65b95b8d77df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0836558a9ec859b57\") pod \"e2e-pvc-model-download-vmqtz\" (UID: \"b59d1545-ea1d-4592-a659-d89e11e74b32\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/44241717332f650a7709e393031c2bc78e3bf793b7e34b4c5be5ae6a20d88f00/globalmount\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-vmqtz"
Jul 27 18:29:11.440221 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:11.440198    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-fa64c5a0-6125-4afb-a25e-65b95b8d77df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0836558a9ec859b57\") pod \"e2e-pvc-model-download-vmqtz\" (UID: \"b59d1545-ea1d-4592-a659-d89e11e74b32\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-vmqtz"
Jul 27 18:29:11.666328 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:11.666247    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-vmqtz"
Jul 27 18:29:11.791565 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:11.791446    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-vmqtz"]
Jul 27 18:29:11.793821 ip-10-0-131-114 kubenswrapper[2557]: W0727 18:29:11.793788    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb59d1545_ea1d_4592_a659_d89e11e74b32.slice/crio-aff80fa652dc236284219a974b91dea4a8a3faf1354ffc6b614194517d28d388 WatchSource:0}: Error finding container aff80fa652dc236284219a974b91dea4a8a3faf1354ffc6b614194517d28d388: Status 404 returned error can't find the container with id aff80fa652dc236284219a974b91dea4a8a3faf1354ffc6b614194517d28d388
Jul 27 18:29:12.546645 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:12.546615    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-vmqtz" event={"ID":"b59d1545-ea1d-4592-a659-d89e11e74b32","Type":"ContainerStarted","Data":"af9b3e230b375ba0e2a98e6244a8a2c356c2466f5791a650dfde9baabb91fcfc"}
Jul 27 18:29:12.547090 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:12.546654    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-vmqtz" event={"ID":"b59d1545-ea1d-4592-a659-d89e11e74b32","Type":"ContainerStarted","Data":"aff80fa652dc236284219a974b91dea4a8a3faf1354ffc6b614194517d28d388"}
Jul 27 18:29:12.561393 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:12.561348    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-vmqtz" podStartSLOduration=7.56133331 podStartE2EDuration="7.56133331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:12.559229197 +0000 UTC m=+3645.046040485" watchObservedRunningTime="2026-07-27 18:29:12.56133331 +0000 UTC m=+3645.048144612"
Jul 27 18:29:20.585844 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:20.585807    2557 generic.go:356] "Generic (PLEG): container finished" podID="b59d1545-ea1d-4592-a659-d89e11e74b32" containerID="af9b3e230b375ba0e2a98e6244a8a2c356c2466f5791a650dfde9baabb91fcfc" exitCode=0
Jul 27 18:29:20.585844 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:20.585848    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-vmqtz" event={"ID":"b59d1545-ea1d-4592-a659-d89e11e74b32","Type":"ContainerDied","Data":"af9b3e230b375ba0e2a98e6244a8a2c356c2466f5791a650dfde9baabb91fcfc"}
Jul 27 18:29:21.743109 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:21.743086    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-vmqtz"
Jul 27 18:29:21.844374 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:21.844303    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzzxr\" (UniqueName: \"kubernetes.io/projected/b59d1545-ea1d-4592-a659-d89e11e74b32-kube-api-access-mzzxr\") pod \"b59d1545-ea1d-4592-a659-d89e11e74b32\" (UID: \"b59d1545-ea1d-4592-a659-d89e11e74b32\") "
Jul 27 18:29:21.844495 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:21.844459    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0836558a9ec859b57\") pod \"b59d1545-ea1d-4592-a659-d89e11e74b32\" (UID: \"b59d1545-ea1d-4592-a659-d89e11e74b32\") "
Jul 27 18:29:21.846580 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:21.846552    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b59d1545-ea1d-4592-a659-d89e11e74b32-kube-api-access-mzzxr" (OuterVolumeSpecName: "kube-api-access-mzzxr") pod "b59d1545-ea1d-4592-a659-d89e11e74b32" (UID: "b59d1545-ea1d-4592-a659-d89e11e74b32"). InnerVolumeSpecName "kube-api-access-mzzxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:29:21.848409 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:21.848391    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0836558a9ec859b57" (OuterVolumeSpecName: "model-storage") pod "b59d1545-ea1d-4592-a659-d89e11e74b32" (UID: "b59d1545-ea1d-4592-a659-d89e11e74b32"). InnerVolumeSpecName "pvc-fa64c5a0-6125-4afb-a25e-65b95b8d77df". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 27 18:29:21.944968 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:21.944939    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzzxr\" (UniqueName: \"kubernetes.io/projected/b59d1545-ea1d-4592-a659-d89e11e74b32-kube-api-access-mzzxr\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:29:21.945099 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:21.944984    2557 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-fa64c5a0-6125-4afb-a25e-65b95b8d77df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0836558a9ec859b57\") on node \"ip-10-0-131-114.ec2.internal\" "
Jul 27 18:29:22.059305 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:22.059285    2557 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-fa64c5a0-6125-4afb-a25e-65b95b8d77df" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0836558a9ec859b57") on node "ip-10-0-131-114.ec2.internal"
Jul 27 18:29:22.146339 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:22.146313    2557 reconciler_common.go:299] "Volume detached for volume \"pvc-fa64c5a0-6125-4afb-a25e-65b95b8d77df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0836558a9ec859b57\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"csi-7c51cd07b865de6f0f6ab6589f6ed36f4d3288357e2f30aa692c6def4a0263a0\""
Jul 27 18:29:22.597003 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:22.596969    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-vmqtz" event={"ID":"b59d1545-ea1d-4592-a659-d89e11e74b32","Type":"ContainerDied","Data":"aff80fa652dc236284219a974b91dea4a8a3faf1354ffc6b614194517d28d388"}
Jul 27 18:29:22.597003 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:22.597005    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-vmqtz"
Jul 27 18:29:22.597247 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:22.597006    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aff80fa652dc236284219a974b91dea4a8a3faf1354ffc6b614194517d28d388"
Jul 27 18:29:31.485710 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:31.485677    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"]
Jul 27 18:29:31.486333 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:31.486074    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b59d1545-ea1d-4592-a659-d89e11e74b32" containerName="storage-initializer"
Jul 27 18:29:31.486333 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:31.486088    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b59d1545-ea1d-4592-a659-d89e11e74b32" containerName="storage-initializer"
Jul 27 18:29:31.486333 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:31.486145    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b59d1545-ea1d-4592-a659-d89e11e74b32" containerName="storage-initializer"
Jul 27 18:29:31.490408 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:31.490386    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:29:31.492612 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:31.492590    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 27 18:29:31.492612 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:31.492600    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"kube-root-ca.crt\""
Jul 27 18:29:31.492941 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:31.492918    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"default-dockercfg-gz7n6\""
Jul 27 18:29:31.493081 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:31.493001    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"openshift-service-ca.crt\""
Jul 27 18:29:31.498292 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:31.498264    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"]
Jul 27 18:29:31.624899 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:31.624869    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt\" (UID: \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:29:31.625089 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:31.624911    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt\" (UID: \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:29:31.625089 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:31.624954    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpvt4\" (UniqueName: \"kubernetes.io/projected/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-kube-api-access-vpvt4\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt\" (UID: \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:29:31.625089 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:31.624984    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt\" (UID: \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:29:31.625089 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:31.625013    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt\" (UID: \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:29:31.625089 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:31.625052    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt\" (UID: \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:29:31.725557 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:31.725521    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt\" (UID: \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:29:31.725557 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:31.725562    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpvt4\" (UniqueName: \"kubernetes.io/projected/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-kube-api-access-vpvt4\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt\" (UID: \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:29:31.725798 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:31.725603    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt\" (UID: \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:29:31.725798 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:31.725634    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt\" (UID: \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:29:31.725798 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:31.725689    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt\" (UID: \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:29:31.725798 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:31.725731    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt\" (UID: \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:29:31.726094 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:31.726070    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt\" (UID: \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:29:31.726156 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:31.726092    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt\" (UID: \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:29:31.726156 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:31.726126    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt\" (UID: \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:29:31.727941 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:31.727916    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt\" (UID: \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:29:31.728102 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:31.728081    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt\" (UID: \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:29:31.733179 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:31.733160    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpvt4\" (UniqueName: \"kubernetes.io/projected/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-kube-api-access-vpvt4\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt\" (UID: \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:29:41.202653 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:41.202615    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-fa64c5a0-6125-4afb-a25e-65b95b8d77df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0836558a9ec859b57\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt\" (UID: \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:29:41.207458 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:41.207429    2557 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-fa64c5a0-6125-4afb-a25e-65b95b8d77df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0836558a9ec859b57\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt\" (UID: \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\") device path: \"\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:29:41.303712 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:41.303683    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-fa64c5a0-6125-4afb-a25e-65b95b8d77df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0836558a9ec859b57\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt\" (UID: \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:29:41.303885 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:41.303818    2557 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-fa64c5a0-6125-4afb-a25e-65b95b8d77df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0836558a9ec859b57\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt\" (UID: \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\") DevicePath \"\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:29:41.306429 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:41.306405    2557 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-fa64c5a0-6125-4afb-a25e-65b95b8d77df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0836558a9ec859b57\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt\" (UID: \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\") DevicePath \"csi-7c51cd07b865de6f0f6ab6589f6ed36f4d3288357e2f30aa692c6def4a0263a0\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:29:41.394458 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:41.394418    2557 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-fa64c5a0-6125-4afb-a25e-65b95b8d77df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0836558a9ec859b57\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt\" (UID: \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/44241717332f650a7709e393031c2bc78e3bf793b7e34b4c5be5ae6a20d88f00/globalmount\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:29:41.411215 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:41.411186    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-fa64c5a0-6125-4afb-a25e-65b95b8d77df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0836558a9ec859b57\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt\" (UID: \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:29:41.702877 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:41.702850    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:29:41.834096 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:41.834068    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"]
Jul 27 18:29:41.836102 ip-10-0-131-114 kubenswrapper[2557]: W0727 18:29:41.836076    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ead8300_1a3e_48ac_a68d_3c57b5b06df0.slice/crio-518c2a46861b850d11f7a4fa7dd14698d73e07a86e677d9e7b5c3bcbc94e74bf WatchSource:0}: Error finding container 518c2a46861b850d11f7a4fa7dd14698d73e07a86e677d9e7b5c3bcbc94e74bf: Status 404 returned error can't find the container with id 518c2a46861b850d11f7a4fa7dd14698d73e07a86e677d9e7b5c3bcbc94e74bf
Jul 27 18:29:42.695576 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:42.695536    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt" event={"ID":"5ead8300-1a3e-48ac-a68d-3c57b5b06df0","Type":"ContainerStarted","Data":"2aca17a37b6924215a173567ce4e078dd7916718363bcab148fd0b04ef42635f"}
Jul 27 18:29:42.696158 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:42.695591    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt" event={"ID":"5ead8300-1a3e-48ac-a68d-3c57b5b06df0","Type":"ContainerStarted","Data":"518c2a46861b850d11f7a4fa7dd14698d73e07a86e677d9e7b5c3bcbc94e74bf"}
Jul 27 18:29:42.703732 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:42.703705    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:29:42.713557 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:42.713512    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt" podStartSLOduration=11.713499324 podStartE2EDuration="11.713499324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:42.712487729 +0000 UTC m=+3675.199299032" watchObservedRunningTime="2026-07-27 18:29:42.713499324 +0000 UTC m=+3675.200310610"
Jul 27 18:29:51.703242 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:51.703207    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:29:51.704723 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:29:51.704696    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt" podUID="5ead8300-1a3e-48ac-a68d-3c57b5b06df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 27 18:30:01.704337 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:30:01.704286    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt" podUID="5ead8300-1a3e-48ac-a68d-3c57b5b06df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 27 18:30:11.704475 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:30:11.704413    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt" podUID="5ead8300-1a3e-48ac-a68d-3c57b5b06df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 27 18:30:21.704231 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:30:21.704184    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt" podUID="5ead8300-1a3e-48ac-a68d-3c57b5b06df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 27 18:30:31.704342 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:30:31.704295    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt" podUID="5ead8300-1a3e-48ac-a68d-3c57b5b06df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 27 18:30:41.704107 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:30:41.704063    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt" podUID="5ead8300-1a3e-48ac-a68d-3c57b5b06df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 27 18:30:51.703777 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:30:51.703712    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt" podUID="5ead8300-1a3e-48ac-a68d-3c57b5b06df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 27 18:31:01.704325 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:31:01.704280    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt" podUID="5ead8300-1a3e-48ac-a68d-3c57b5b06df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 27 18:31:11.704075 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:31:11.704026    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt" podUID="5ead8300-1a3e-48ac-a68d-3c57b5b06df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 27 18:31:21.704468 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:31:21.704369    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt" podUID="5ead8300-1a3e-48ac-a68d-3c57b5b06df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 27 18:31:31.703479 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:31:31.703440    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt" podUID="5ead8300-1a3e-48ac-a68d-3c57b5b06df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 27 18:31:41.713230 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:31:41.713193    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:31:41.720632 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:31:41.720605    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:31:46.566068 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:31:46.566033    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"]
Jul 27 18:32:01.588706 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:01.588667    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt" podUID="5ead8300-1a3e-48ac-a68d-3c57b5b06df0" containerName="main" containerID="cri-o://2aca17a37b6924215a173567ce4e078dd7916718363bcab148fd0b04ef42635f" gracePeriod=45
Jul 27 18:32:01.714299 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:01.714261    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt" podUID="5ead8300-1a3e-48ac-a68d-3c57b5b06df0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Jul 27 18:32:02.339615 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.339590    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:32:02.376132 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.376067    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-tls-certs\") pod \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\" (UID: \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\") "
Jul 27 18:32:02.376132 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.376121    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-dshm\") pod \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\" (UID: \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\") "
Jul 27 18:32:02.376316 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.376201    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0836558a9ec859b57\") pod \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\" (UID: \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\") "
Jul 27 18:32:02.376316 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.376228    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-home\") pod \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\" (UID: \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\") "
Jul 27 18:32:02.376316 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.376245    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-tmp-dir\") pod \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\" (UID: \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\") "
Jul 27 18:32:02.376316 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.376277    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-model-cache\") pod \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\" (UID: \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\") "
Jul 27 18:32:02.376535 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.376323    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpvt4\" (UniqueName: \"kubernetes.io/projected/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-kube-api-access-vpvt4\") pod \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\" (UID: \"5ead8300-1a3e-48ac-a68d-3c57b5b06df0\") "
Jul 27 18:32:02.377006 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.376901    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-model-cache" (OuterVolumeSpecName: "model-cache") pod "5ead8300-1a3e-48ac-a68d-3c57b5b06df0" (UID: "5ead8300-1a3e-48ac-a68d-3c57b5b06df0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:32:02.377114 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.377004    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-home" (OuterVolumeSpecName: "home") pod "5ead8300-1a3e-48ac-a68d-3c57b5b06df0" (UID: "5ead8300-1a3e-48ac-a68d-3c57b5b06df0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:32:02.378868 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.378783    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-dshm" (OuterVolumeSpecName: "dshm") pod "5ead8300-1a3e-48ac-a68d-3c57b5b06df0" (UID: "5ead8300-1a3e-48ac-a68d-3c57b5b06df0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:32:02.379201 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.379179    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-kube-api-access-vpvt4" (OuterVolumeSpecName: "kube-api-access-vpvt4") pod "5ead8300-1a3e-48ac-a68d-3c57b5b06df0" (UID: "5ead8300-1a3e-48ac-a68d-3c57b5b06df0"). InnerVolumeSpecName "kube-api-access-vpvt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:32:02.379647 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.379614    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5ead8300-1a3e-48ac-a68d-3c57b5b06df0" (UID: "5ead8300-1a3e-48ac-a68d-3c57b5b06df0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:32:02.379850 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.379826    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0836558a9ec859b57" (OuterVolumeSpecName: "kserve-pvc-source") pod "5ead8300-1a3e-48ac-a68d-3c57b5b06df0" (UID: "5ead8300-1a3e-48ac-a68d-3c57b5b06df0"). InnerVolumeSpecName "pvc-fa64c5a0-6125-4afb-a25e-65b95b8d77df". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 27 18:32:02.391794 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.391740    2557 generic.go:356] "Generic (PLEG): container finished" podID="5ead8300-1a3e-48ac-a68d-3c57b5b06df0" containerID="2aca17a37b6924215a173567ce4e078dd7916718363bcab148fd0b04ef42635f" exitCode=0
Jul 27 18:32:02.391890 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.391795    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"
Jul 27 18:32:02.391890 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.391826    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt" event={"ID":"5ead8300-1a3e-48ac-a68d-3c57b5b06df0","Type":"ContainerDied","Data":"2aca17a37b6924215a173567ce4e078dd7916718363bcab148fd0b04ef42635f"}
Jul 27 18:32:02.391964 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.391891    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt" event={"ID":"5ead8300-1a3e-48ac-a68d-3c57b5b06df0","Type":"ContainerDied","Data":"518c2a46861b850d11f7a4fa7dd14698d73e07a86e677d9e7b5c3bcbc94e74bf"}
Jul 27 18:32:02.391964 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.391922    2557 scope.go:117] "RemoveContainer" containerID="2aca17a37b6924215a173567ce4e078dd7916718363bcab148fd0b04ef42635f"
Jul 27 18:32:02.392998 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.392937    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5ead8300-1a3e-48ac-a68d-3c57b5b06df0" (UID: "5ead8300-1a3e-48ac-a68d-3c57b5b06df0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:32:02.403853 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.403834    2557 scope.go:117] "RemoveContainer" containerID="2aca17a37b6924215a173567ce4e078dd7916718363bcab148fd0b04ef42635f"
Jul 27 18:32:02.404131 ip-10-0-131-114 kubenswrapper[2557]: E0727 18:32:02.404113    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aca17a37b6924215a173567ce4e078dd7916718363bcab148fd0b04ef42635f\": container with ID starting with 2aca17a37b6924215a173567ce4e078dd7916718363bcab148fd0b04ef42635f not found: ID does not exist" containerID="2aca17a37b6924215a173567ce4e078dd7916718363bcab148fd0b04ef42635f"
Jul 27 18:32:02.404178 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.404139    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aca17a37b6924215a173567ce4e078dd7916718363bcab148fd0b04ef42635f"} err="failed to get container status \"2aca17a37b6924215a173567ce4e078dd7916718363bcab148fd0b04ef42635f\": rpc error: code = NotFound desc = could not find container \"2aca17a37b6924215a173567ce4e078dd7916718363bcab148fd0b04ef42635f\": container with ID starting with 2aca17a37b6924215a173567ce4e078dd7916718363bcab148fd0b04ef42635f not found: ID does not exist"
Jul 27 18:32:02.477051 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.477027    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-dshm\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:32:02.477178 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.477069    2557 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-fa64c5a0-6125-4afb-a25e-65b95b8d77df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0836558a9ec859b57\") on node \"ip-10-0-131-114.ec2.internal\" "
Jul 27 18:32:02.477178 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.477083    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-home\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:32:02.477178 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.477095    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-tmp-dir\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:32:02.477178 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.477105    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-model-cache\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:32:02.477178 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.477113    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpvt4\" (UniqueName: \"kubernetes.io/projected/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-kube-api-access-vpvt4\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:32:02.477178 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.477123    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ead8300-1a3e-48ac-a68d-3c57b5b06df0-tls-certs\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:32:02.505274 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.505256    2557 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-fa64c5a0-6125-4afb-a25e-65b95b8d77df" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0836558a9ec859b57") on node "ip-10-0-131-114.ec2.internal"
Jul 27 18:32:02.578352 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.578330    2557 reconciler_common.go:299] "Volume detached for volume \"pvc-fa64c5a0-6125-4afb-a25e-65b95b8d77df\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0836558a9ec859b57\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"csi-7c51cd07b865de6f0f6ab6589f6ed36f4d3288357e2f30aa692c6def4a0263a0\""
Jul 27 18:32:02.713045 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.713013    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"]
Jul 27 18:32:02.718459 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:02.718436    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfclbtnt"]
Jul 27 18:32:04.119434 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:04.119397    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ead8300-1a3e-48ac-a68d-3c57b5b06df0" path="/var/lib/kubelet/pods/5ead8300-1a3e-48ac-a68d-3c57b5b06df0/volumes"
Jul 27 18:32:09.809084 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:09.809044    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4fpd"]
Jul 27 18:32:09.809507 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:09.809387    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ead8300-1a3e-48ac-a68d-3c57b5b06df0" containerName="main"
Jul 27 18:32:09.809507 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:09.809398    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ead8300-1a3e-48ac-a68d-3c57b5b06df0" containerName="main"
Jul 27 18:32:09.809507 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:09.809466    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ead8300-1a3e-48ac-a68d-3c57b5b06df0" containerName="main"
Jul 27 18:32:09.813829 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:09.813812    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4fpd"
Jul 27 18:32:09.816730 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:09.816707    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"kube-root-ca.crt\""
Jul 27 18:32:09.816909 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:09.816785    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"openshift-service-ca.crt\""
Jul 27 18:32:09.816909 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:09.816800    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-9xfm7\""
Jul 27 18:32:09.825366 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:09.823110    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4fpd"]
Jul 27 18:32:09.841186 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:09.841155    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltm8q\" (UniqueName: \"kubernetes.io/projected/b3ec81ca-4d1c-4edd-a2d1-85052bd55366-kube-api-access-ltm8q\") pod \"e2e-pvc-model-download-m4fpd\" (UID: \"b3ec81ca-4d1c-4edd-a2d1-85052bd55366\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4fpd"
Jul 27 18:32:09.941933 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:09.941896    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltm8q\" (UniqueName: \"kubernetes.io/projected/b3ec81ca-4d1c-4edd-a2d1-85052bd55366-kube-api-access-ltm8q\") pod \"e2e-pvc-model-download-m4fpd\" (UID: \"b3ec81ca-4d1c-4edd-a2d1-85052bd55366\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4fpd"
Jul 27 18:32:09.950777 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:09.950716    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltm8q\" (UniqueName: \"kubernetes.io/projected/b3ec81ca-4d1c-4edd-a2d1-85052bd55366-kube-api-access-ltm8q\") pod \"e2e-pvc-model-download-m4fpd\" (UID: \"b3ec81ca-4d1c-4edd-a2d1-85052bd55366\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4fpd"
Jul 27 18:32:11.551383 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:11.551350    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-vmqtz"]
Jul 27 18:32:11.553748 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:11.553717    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-vmqtz"]
Jul 27 18:32:12.118732 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:12.118698    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b59d1545-ea1d-4592-a659-d89e11e74b32" path="/var/lib/kubelet/pods/b59d1545-ea1d-4592-a659-d89e11e74b32/volumes"
Jul 27 18:32:13.876619 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:13.876579    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-47e956c4-9962-4d79-8479-81e5daeb1dba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06095df590c4fe8dd\") pod \"e2e-pvc-model-download-m4fpd\" (UID: \"b3ec81ca-4d1c-4edd-a2d1-85052bd55366\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4fpd"
Jul 27 18:32:13.881122 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:13.881098    2557 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-47e956c4-9962-4d79-8479-81e5daeb1dba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06095df590c4fe8dd\") pod \"e2e-pvc-model-download-m4fpd\" (UID: \"b3ec81ca-4d1c-4edd-a2d1-85052bd55366\") device path: \"\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4fpd"
Jul 27 18:32:13.977032 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:13.976996    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-47e956c4-9962-4d79-8479-81e5daeb1dba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06095df590c4fe8dd\") pod \"e2e-pvc-model-download-m4fpd\" (UID: \"b3ec81ca-4d1c-4edd-a2d1-85052bd55366\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4fpd"
Jul 27 18:32:13.977210 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:13.977140    2557 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-47e956c4-9962-4d79-8479-81e5daeb1dba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06095df590c4fe8dd\") pod \"e2e-pvc-model-download-m4fpd\" (UID: \"b3ec81ca-4d1c-4edd-a2d1-85052bd55366\") DevicePath \"\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4fpd"
Jul 27 18:32:13.979526 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:13.979501    2557 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-47e956c4-9962-4d79-8479-81e5daeb1dba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06095df590c4fe8dd\") pod \"e2e-pvc-model-download-m4fpd\" (UID: \"b3ec81ca-4d1c-4edd-a2d1-85052bd55366\") DevicePath \"csi-f3c1039b9aa07e2d1e308d2ee9875c79920b53784a0c9903fa040900ccbf6b83\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4fpd"
Jul 27 18:32:14.190231 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:14.190199    2557 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-47e956c4-9962-4d79-8479-81e5daeb1dba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06095df590c4fe8dd\") pod \"e2e-pvc-model-download-m4fpd\" (UID: \"b3ec81ca-4d1c-4edd-a2d1-85052bd55366\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/e04a5e62d8ceb90c66c833954a6cff8c2326bb0711db3298101555bfc38f2c5d/globalmount\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4fpd"
Jul 27 18:32:14.203649 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:14.203624    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-47e956c4-9962-4d79-8479-81e5daeb1dba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06095df590c4fe8dd\") pod \"e2e-pvc-model-download-m4fpd\" (UID: \"b3ec81ca-4d1c-4edd-a2d1-85052bd55366\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4fpd"
Jul 27 18:32:14.328517 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:14.328477    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4fpd"
Jul 27 18:32:14.457131 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:14.457106    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4fpd"]
Jul 27 18:32:14.458778 ip-10-0-131-114 kubenswrapper[2557]: W0727 18:32:14.458740    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3ec81ca_4d1c_4edd_a2d1_85052bd55366.slice/crio-e0297ebe37085f5642bb394d04bb763b9a2222a9a5feadac20f35ed03f836f77 WatchSource:0}: Error finding container e0297ebe37085f5642bb394d04bb763b9a2222a9a5feadac20f35ed03f836f77: Status 404 returned error can't find the container with id e0297ebe37085f5642bb394d04bb763b9a2222a9a5feadac20f35ed03f836f77
Jul 27 18:32:15.453671 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:15.453636    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4fpd" event={"ID":"b3ec81ca-4d1c-4edd-a2d1-85052bd55366","Type":"ContainerStarted","Data":"98b892b24a2565bdbb109fd61a7c2c8a121e2babbbb788605d69b8a7d3289b58"}
Jul 27 18:32:15.454159 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:15.453693    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4fpd" event={"ID":"b3ec81ca-4d1c-4edd-a2d1-85052bd55366","Type":"ContainerStarted","Data":"e0297ebe37085f5642bb394d04bb763b9a2222a9a5feadac20f35ed03f836f77"}
Jul 27 18:32:15.470711 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:15.470665    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4fpd" podStartSLOduration=9.470649541 podStartE2EDuration="9.470649541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:32:15.468692625 +0000 UTC m=+3827.955503911" watchObservedRunningTime="2026-07-27 18:32:15.470649541 +0000 UTC m=+3827.957460828"
Jul 27 18:32:23.498832 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:23.498792    2557 generic.go:356] "Generic (PLEG): container finished" podID="b3ec81ca-4d1c-4edd-a2d1-85052bd55366" containerID="98b892b24a2565bdbb109fd61a7c2c8a121e2babbbb788605d69b8a7d3289b58" exitCode=0
Jul 27 18:32:23.499255 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:23.498868    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4fpd" event={"ID":"b3ec81ca-4d1c-4edd-a2d1-85052bd55366","Type":"ContainerDied","Data":"98b892b24a2565bdbb109fd61a7c2c8a121e2babbbb788605d69b8a7d3289b58"}
Jul 27 18:32:24.655940 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:24.655915    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4fpd"
Jul 27 18:32:24.772695 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:24.772626    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06095df590c4fe8dd\") pod \"b3ec81ca-4d1c-4edd-a2d1-85052bd55366\" (UID: \"b3ec81ca-4d1c-4edd-a2d1-85052bd55366\") "
Jul 27 18:32:24.772695 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:24.772680    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltm8q\" (UniqueName: \"kubernetes.io/projected/b3ec81ca-4d1c-4edd-a2d1-85052bd55366-kube-api-access-ltm8q\") pod \"b3ec81ca-4d1c-4edd-a2d1-85052bd55366\" (UID: \"b3ec81ca-4d1c-4edd-a2d1-85052bd55366\") "
Jul 27 18:32:24.774961 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:24.774934    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ec81ca-4d1c-4edd-a2d1-85052bd55366-kube-api-access-ltm8q" (OuterVolumeSpecName: "kube-api-access-ltm8q") pod "b3ec81ca-4d1c-4edd-a2d1-85052bd55366" (UID: "b3ec81ca-4d1c-4edd-a2d1-85052bd55366"). InnerVolumeSpecName "kube-api-access-ltm8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:32:24.775391 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:24.775372    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-06095df590c4fe8dd" (OuterVolumeSpecName: "model-storage") pod "b3ec81ca-4d1c-4edd-a2d1-85052bd55366" (UID: "b3ec81ca-4d1c-4edd-a2d1-85052bd55366"). InnerVolumeSpecName "pvc-47e956c4-9962-4d79-8479-81e5daeb1dba". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 27 18:32:24.873925 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:24.873897    2557 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-47e956c4-9962-4d79-8479-81e5daeb1dba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06095df590c4fe8dd\") on node \"ip-10-0-131-114.ec2.internal\" "
Jul 27 18:32:24.873925 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:24.873921    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltm8q\" (UniqueName: \"kubernetes.io/projected/b3ec81ca-4d1c-4edd-a2d1-85052bd55366-kube-api-access-ltm8q\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:32:24.990240 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:24.990221    2557 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-47e956c4-9962-4d79-8479-81e5daeb1dba" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-06095df590c4fe8dd") on node "ip-10-0-131-114.ec2.internal"
Jul 27 18:32:25.075798 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:25.075721    2557 reconciler_common.go:299] "Volume detached for volume \"pvc-47e956c4-9962-4d79-8479-81e5daeb1dba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06095df590c4fe8dd\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"csi-f3c1039b9aa07e2d1e308d2ee9875c79920b53784a0c9903fa040900ccbf6b83\""
Jul 27 18:32:25.509259 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:25.509226    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4fpd" event={"ID":"b3ec81ca-4d1c-4edd-a2d1-85052bd55366","Type":"ContainerDied","Data":"e0297ebe37085f5642bb394d04bb763b9a2222a9a5feadac20f35ed03f836f77"}
Jul 27 18:32:25.509436 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:25.509267    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0297ebe37085f5642bb394d04bb763b9a2222a9a5feadac20f35ed03f836f77"
Jul 27 18:32:25.509436 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:25.509266    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-m4fpd"
Jul 27 18:32:36.328117 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:36.328087    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"]
Jul 27 18:32:36.328579 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:36.328435    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3ec81ca-4d1c-4edd-a2d1-85052bd55366" containerName="storage-initializer"
Jul 27 18:32:36.328579 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:36.328445    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3ec81ca-4d1c-4edd-a2d1-85052bd55366" containerName="storage-initializer"
Jul 27 18:32:36.328579 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:36.328507    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3ec81ca-4d1c-4edd-a2d1-85052bd55366" containerName="storage-initializer"
Jul 27 18:32:36.332783 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:36.332748    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:32:36.334895 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:36.334867    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-9xfm7\""
Jul 27 18:32:36.335122 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:36.335105    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"kube-root-ca.crt\""
Jul 27 18:32:36.335246 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:36.335223    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 27 18:32:36.335372 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:36.335225    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"openshift-service-ca.crt\""
Jul 27 18:32:36.335609 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:36.335589    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-cb9h8\""
Jul 27 18:32:36.345258 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:36.345237    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"]
Jul 27 18:32:36.475044 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:36.475002    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8f61f6b-b525-4658-ba38-b97f8631fe50-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm\" (UID: \"a8f61f6b-b525-4658-ba38-b97f8631fe50\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:32:36.475044 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:36.475047    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qjpd\" (UniqueName: \"kubernetes.io/projected/a8f61f6b-b525-4658-ba38-b97f8631fe50-kube-api-access-2qjpd\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm\" (UID: \"a8f61f6b-b525-4658-ba38-b97f8631fe50\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:32:36.475303 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:36.475088    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8f61f6b-b525-4658-ba38-b97f8631fe50-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm\" (UID: \"a8f61f6b-b525-4658-ba38-b97f8631fe50\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:32:36.475303 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:36.475147    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8f61f6b-b525-4658-ba38-b97f8631fe50-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm\" (UID: \"a8f61f6b-b525-4658-ba38-b97f8631fe50\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:32:36.475303 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:36.475190    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8f61f6b-b525-4658-ba38-b97f8631fe50-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm\" (UID: \"a8f61f6b-b525-4658-ba38-b97f8631fe50\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:32:36.475303 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:36.475252    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8f61f6b-b525-4658-ba38-b97f8631fe50-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm\" (UID: \"a8f61f6b-b525-4658-ba38-b97f8631fe50\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:32:36.575882 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:36.575845    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8f61f6b-b525-4658-ba38-b97f8631fe50-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm\" (UID: \"a8f61f6b-b525-4658-ba38-b97f8631fe50\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:32:36.576085 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:36.575897    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8f61f6b-b525-4658-ba38-b97f8631fe50-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm\" (UID: \"a8f61f6b-b525-4658-ba38-b97f8631fe50\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:32:36.576085 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:36.575927    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8f61f6b-b525-4658-ba38-b97f8631fe50-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm\" (UID: \"a8f61f6b-b525-4658-ba38-b97f8631fe50\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:32:36.576085 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:36.575995    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8f61f6b-b525-4658-ba38-b97f8631fe50-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm\" (UID: \"a8f61f6b-b525-4658-ba38-b97f8631fe50\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:32:36.576085 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:36.576022    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qjpd\" (UniqueName: \"kubernetes.io/projected/a8f61f6b-b525-4658-ba38-b97f8631fe50-kube-api-access-2qjpd\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm\" (UID: \"a8f61f6b-b525-4658-ba38-b97f8631fe50\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:32:36.576085 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:36.576066    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8f61f6b-b525-4658-ba38-b97f8631fe50-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm\" (UID: \"a8f61f6b-b525-4658-ba38-b97f8631fe50\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:32:36.576518 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:36.576491    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8f61f6b-b525-4658-ba38-b97f8631fe50-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm\" (UID: \"a8f61f6b-b525-4658-ba38-b97f8631fe50\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:32:36.576709 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:36.576531    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8f61f6b-b525-4658-ba38-b97f8631fe50-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm\" (UID: \"a8f61f6b-b525-4658-ba38-b97f8631fe50\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:32:36.576892 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:36.576783    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8f61f6b-b525-4658-ba38-b97f8631fe50-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm\" (UID: \"a8f61f6b-b525-4658-ba38-b97f8631fe50\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:32:36.578668 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:36.578619    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8f61f6b-b525-4658-ba38-b97f8631fe50-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm\" (UID: \"a8f61f6b-b525-4658-ba38-b97f8631fe50\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:32:36.578970 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:36.578951    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8f61f6b-b525-4658-ba38-b97f8631fe50-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm\" (UID: \"a8f61f6b-b525-4658-ba38-b97f8631fe50\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:32:36.593767 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:36.593727    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qjpd\" (UniqueName: \"kubernetes.io/projected/a8f61f6b-b525-4658-ba38-b97f8631fe50-kube-api-access-2qjpd\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm\" (UID: \"a8f61f6b-b525-4658-ba38-b97f8631fe50\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:32:44.547044 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:44.547008    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-47e956c4-9962-4d79-8479-81e5daeb1dba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06095df590c4fe8dd\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm\" (UID: \"a8f61f6b-b525-4658-ba38-b97f8631fe50\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:32:44.551212 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:44.551188    2557 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-47e956c4-9962-4d79-8479-81e5daeb1dba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06095df590c4fe8dd\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm\" (UID: \"a8f61f6b-b525-4658-ba38-b97f8631fe50\") device path: \"\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:32:44.648233 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:44.648196    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-47e956c4-9962-4d79-8479-81e5daeb1dba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06095df590c4fe8dd\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm\" (UID: \"a8f61f6b-b525-4658-ba38-b97f8631fe50\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:32:44.648418 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:44.648349    2557 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-47e956c4-9962-4d79-8479-81e5daeb1dba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06095df590c4fe8dd\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm\" (UID: \"a8f61f6b-b525-4658-ba38-b97f8631fe50\") DevicePath \"\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:32:44.650612 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:44.650591    2557 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-47e956c4-9962-4d79-8479-81e5daeb1dba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06095df590c4fe8dd\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm\" (UID: \"a8f61f6b-b525-4658-ba38-b97f8631fe50\") DevicePath \"csi-f3c1039b9aa07e2d1e308d2ee9875c79920b53784a0c9903fa040900ccbf6b83\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:32:44.739815 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:44.739774    2557 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-47e956c4-9962-4d79-8479-81e5daeb1dba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06095df590c4fe8dd\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm\" (UID: \"a8f61f6b-b525-4658-ba38-b97f8631fe50\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/e04a5e62d8ceb90c66c833954a6cff8c2326bb0711db3298101555bfc38f2c5d/globalmount\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:32:44.754704 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:44.754671    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-47e956c4-9962-4d79-8479-81e5daeb1dba\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06095df590c4fe8dd\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm\" (UID: \"a8f61f6b-b525-4658-ba38-b97f8631fe50\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:32:45.043857 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:45.043807    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:32:45.178021 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:45.177991    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"]
Jul 27 18:32:45.179845 ip-10-0-131-114 kubenswrapper[2557]: W0727 18:32:45.179812    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f61f6b_b525_4658_ba38_b97f8631fe50.slice/crio-8466711da3f86c740810fca80b423669a3f70a31d7546b2c4afe8a7135deabe5 WatchSource:0}: Error finding container 8466711da3f86c740810fca80b423669a3f70a31d7546b2c4afe8a7135deabe5: Status 404 returned error can't find the container with id 8466711da3f86c740810fca80b423669a3f70a31d7546b2c4afe8a7135deabe5
Jul 27 18:32:45.181809 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:45.181792    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 18:32:45.604675 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:45.604638    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm" event={"ID":"a8f61f6b-b525-4658-ba38-b97f8631fe50","Type":"ContainerStarted","Data":"8466711da3f86c740810fca80b423669a3f70a31d7546b2c4afe8a7135deabe5"}
Jul 27 18:32:47.615958 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:47.615921    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm" event={"ID":"a8f61f6b-b525-4658-ba38-b97f8631fe50","Type":"ContainerStarted","Data":"fc63be0dcb1f24bf4d01b2adc3d10611db243c729993c6cd207c2ad63e1750c0"}
Jul 27 18:32:47.616426 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:47.615989    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:32:48.622643 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:48.622604    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm" event={"ID":"a8f61f6b-b525-4658-ba38-b97f8631fe50","Type":"ContainerStarted","Data":"b8dc52b204b52518cad26ac19412802f747e01891ffe3752a484e261aea13eff"}
Jul 27 18:32:48.642989 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:48.642940    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm" podStartSLOduration=11.000284998 podStartE2EDuration="12.642925016s" totalImagesPullingTime="1.642640018s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:32:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:32:48.640880312 +0000 UTC m=+3861.127691599" watchObservedRunningTime="2026-07-27 18:32:48.642925016 +0000 UTC m=+3861.129736306"
Jul 27 18:32:55.044839 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:55.044805    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:32:55.045253 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:55.044862    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:32:55.046329 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:32:55.046295    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm" podUID="a8f61f6b-b525-4658-ba38-b97f8631fe50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8001/health\": dial tcp 10.132.0.55:8001: connect: connection refused"
Jul 27 18:33:05.044453 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:33:05.044403    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm" podUID="a8f61f6b-b525-4658-ba38-b97f8631fe50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8001/health\": dial tcp 10.132.0.55:8001: connect: connection refused"
Jul 27 18:33:05.057026 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:33:05.057001    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:33:15.044793 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:33:15.044722    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm" podUID="a8f61f6b-b525-4658-ba38-b97f8631fe50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8001/health\": dial tcp 10.132.0.55:8001: connect: connection refused"
Jul 27 18:33:25.045329 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:33:25.045280    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm" podUID="a8f61f6b-b525-4658-ba38-b97f8631fe50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8001/health\": dial tcp 10.132.0.55:8001: connect: connection refused"
Jul 27 18:33:35.044648 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:33:35.044603    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm" podUID="a8f61f6b-b525-4658-ba38-b97f8631fe50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8001/health\": dial tcp 10.132.0.55:8001: connect: connection refused"
Jul 27 18:33:45.044818 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:33:45.044747    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm" podUID="a8f61f6b-b525-4658-ba38-b97f8631fe50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8001/health\": dial tcp 10.132.0.55:8001: connect: connection refused"
Jul 27 18:33:55.044593 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:33:55.044547    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm" podUID="a8f61f6b-b525-4658-ba38-b97f8631fe50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8001/health\": dial tcp 10.132.0.55:8001: connect: connection refused"
Jul 27 18:34:05.044461 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:34:05.044413    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm" podUID="a8f61f6b-b525-4658-ba38-b97f8631fe50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8001/health\": dial tcp 10.132.0.55:8001: connect: connection refused"
Jul 27 18:34:15.044583 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:34:15.044523    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm" podUID="a8f61f6b-b525-4658-ba38-b97f8631fe50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8001/health\": dial tcp 10.132.0.55:8001: connect: connection refused"
Jul 27 18:34:25.045023 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:34:25.044922    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm" podUID="a8f61f6b-b525-4658-ba38-b97f8631fe50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8001/health\": dial tcp 10.132.0.55:8001: connect: connection refused"
Jul 27 18:34:35.045173 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:34:35.045124    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm" podUID="a8f61f6b-b525-4658-ba38-b97f8631fe50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8001/health\": dial tcp 10.132.0.55:8001: connect: connection refused"
Jul 27 18:34:45.054416 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:34:45.054383    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:34:45.067391 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:34:45.067363    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm"
Jul 27 18:35:28.483465 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:35:28.483433    2557 scope.go:117] "RemoveContainer" containerID="af9b3e230b375ba0e2a98e6244a8a2c356c2466f5791a650dfde9baabb91fcfc"
Jul 27 18:47:28.084241 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:28.084205    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-48639af5_llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm_a8f61f6b-b525-4658-ba38-b97f8631fe50/llm-d-routing-sidecar/0.log"
Jul 27 18:47:28.096055 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:28.096027    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-48639af5_llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c86qrm_a8f61f6b-b525-4658-ba38-b97f8631fe50/main/0.log"
Jul 27 18:47:32.809177 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:32.809137    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-bwgsg"]
Jul 27 18:47:32.817016 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:32.816990    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-bwgsg"
Jul 27 18:47:32.819270 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:32.819243    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"openshift-service-ca.crt\""
Jul 27 18:47:32.819671 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:32.819654    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"kube-root-ca.crt\""
Jul 27 18:47:32.819799 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:32.819660    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-g5bb7\""
Jul 27 18:47:32.824541 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:32.824522    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-bwgsg"]
Jul 27 18:47:32.938426 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:32.938396    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5zpq\" (UniqueName: \"kubernetes.io/projected/998b183e-1ab2-44c4-8ff8-96b69eb7b07b-kube-api-access-s5zpq\") pod \"e2e-pvc-model-download-bwgsg\" (UID: \"998b183e-1ab2-44c4-8ff8-96b69eb7b07b\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-bwgsg"
Jul 27 18:47:33.039955 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:33.039921    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5zpq\" (UniqueName: \"kubernetes.io/projected/998b183e-1ab2-44c4-8ff8-96b69eb7b07b-kube-api-access-s5zpq\") pod \"e2e-pvc-model-download-bwgsg\" (UID: \"998b183e-1ab2-44c4-8ff8-96b69eb7b07b\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-bwgsg"
Jul 27 18:47:33.049417 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:33.049385    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5zpq\" (UniqueName: \"kubernetes.io/projected/998b183e-1ab2-44c4-8ff8-96b69eb7b07b-kube-api-access-s5zpq\") pod \"e2e-pvc-model-download-bwgsg\" (UID: \"998b183e-1ab2-44c4-8ff8-96b69eb7b07b\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-bwgsg"
Jul 27 18:47:40.698807 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:40.698775    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-a73e9557-7c78-435b-b2fe-17034bfc65b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04f045b2a76e32ea2\") pod \"e2e-pvc-model-download-bwgsg\" (UID: \"998b183e-1ab2-44c4-8ff8-96b69eb7b07b\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-bwgsg"
Jul 27 18:47:40.704086 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:40.704059    2557 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-a73e9557-7c78-435b-b2fe-17034bfc65b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04f045b2a76e32ea2\") pod \"e2e-pvc-model-download-bwgsg\" (UID: \"998b183e-1ab2-44c4-8ff8-96b69eb7b07b\") device path: \"\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-bwgsg"
Jul 27 18:47:40.800099 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:40.800076    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-a73e9557-7c78-435b-b2fe-17034bfc65b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04f045b2a76e32ea2\") pod \"e2e-pvc-model-download-bwgsg\" (UID: \"998b183e-1ab2-44c4-8ff8-96b69eb7b07b\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-bwgsg"
Jul 27 18:47:40.800259 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:40.800237    2557 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-a73e9557-7c78-435b-b2fe-17034bfc65b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04f045b2a76e32ea2\") pod \"e2e-pvc-model-download-bwgsg\" (UID: \"998b183e-1ab2-44c4-8ff8-96b69eb7b07b\") DevicePath \"\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-bwgsg"
Jul 27 18:47:40.802922 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:40.802902    2557 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-a73e9557-7c78-435b-b2fe-17034bfc65b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04f045b2a76e32ea2\") pod \"e2e-pvc-model-download-bwgsg\" (UID: \"998b183e-1ab2-44c4-8ff8-96b69eb7b07b\") DevicePath \"csi-effe7b06b46bdf10d56898c66e78d47c7fe3a9615a8eae84b539ba087163f687\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-bwgsg"
Jul 27 18:47:41.023355 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:41.023326    2557 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-a73e9557-7c78-435b-b2fe-17034bfc65b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04f045b2a76e32ea2\") pod \"e2e-pvc-model-download-bwgsg\" (UID: \"998b183e-1ab2-44c4-8ff8-96b69eb7b07b\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/180c16756ceeec634bbc4ba862f177a0ad3e8e06202a20fd191a4d6d9c121ac6/globalmount\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-bwgsg"
Jul 27 18:47:41.036422 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:41.036397    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-a73e9557-7c78-435b-b2fe-17034bfc65b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04f045b2a76e32ea2\") pod \"e2e-pvc-model-download-bwgsg\" (UID: \"998b183e-1ab2-44c4-8ff8-96b69eb7b07b\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-bwgsg"
Jul 27 18:47:41.226297 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:41.226266    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-bwgsg"
Jul 27 18:47:41.356362 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:41.356334    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-bwgsg"]
Jul 27 18:47:41.358554 ip-10-0-131-114 kubenswrapper[2557]: W0727 18:47:41.358527    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod998b183e_1ab2_44c4_8ff8_96b69eb7b07b.slice/crio-529b75a683c8af0a9a4bc45658712502b278cc8a507bb3072ddf5494c773f434 WatchSource:0}: Error finding container 529b75a683c8af0a9a4bc45658712502b278cc8a507bb3072ddf5494c773f434: Status 404 returned error can't find the container with id 529b75a683c8af0a9a4bc45658712502b278cc8a507bb3072ddf5494c773f434
Jul 27 18:47:41.841149 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:41.841111    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-bwgsg" event={"ID":"998b183e-1ab2-44c4-8ff8-96b69eb7b07b","Type":"ContainerStarted","Data":"7066c31067b760052d6aab411fa2d74950caec9205104ab1ef356e8a8ddfc3de"}
Jul 27 18:47:41.841651 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:41.841165    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-bwgsg" event={"ID":"998b183e-1ab2-44c4-8ff8-96b69eb7b07b","Type":"ContainerStarted","Data":"529b75a683c8af0a9a4bc45658712502b278cc8a507bb3072ddf5494c773f434"}
Jul 27 18:47:41.857968 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:41.857920    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-bwgsg" podStartSLOduration=12.857904337 podStartE2EDuration="12.857904337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:47:41.857437712 +0000 UTC m=+4754.344248999" watchObservedRunningTime="2026-07-27 18:47:41.857904337 +0000 UTC m=+4754.344715623"
Jul 27 18:47:49.880573 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:49.880531    2557 generic.go:356] "Generic (PLEG): container finished" podID="998b183e-1ab2-44c4-8ff8-96b69eb7b07b" containerID="7066c31067b760052d6aab411fa2d74950caec9205104ab1ef356e8a8ddfc3de" exitCode=0
Jul 27 18:47:49.880972 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:49.880606    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-bwgsg" event={"ID":"998b183e-1ab2-44c4-8ff8-96b69eb7b07b","Type":"ContainerDied","Data":"7066c31067b760052d6aab411fa2d74950caec9205104ab1ef356e8a8ddfc3de"}
Jul 27 18:47:51.035762 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:51.035728    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-bwgsg"
Jul 27 18:47:51.079279 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:51.079255    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04f045b2a76e32ea2\") pod \"998b183e-1ab2-44c4-8ff8-96b69eb7b07b\" (UID: \"998b183e-1ab2-44c4-8ff8-96b69eb7b07b\") "
Jul 27 18:47:51.079413 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:51.079308    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5zpq\" (UniqueName: \"kubernetes.io/projected/998b183e-1ab2-44c4-8ff8-96b69eb7b07b-kube-api-access-s5zpq\") pod \"998b183e-1ab2-44c4-8ff8-96b69eb7b07b\" (UID: \"998b183e-1ab2-44c4-8ff8-96b69eb7b07b\") "
Jul 27 18:47:51.081411 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:51.081379    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998b183e-1ab2-44c4-8ff8-96b69eb7b07b-kube-api-access-s5zpq" (OuterVolumeSpecName: "kube-api-access-s5zpq") pod "998b183e-1ab2-44c4-8ff8-96b69eb7b07b" (UID: "998b183e-1ab2-44c4-8ff8-96b69eb7b07b"). InnerVolumeSpecName "kube-api-access-s5zpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:47:51.081840 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:51.081820    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-04f045b2a76e32ea2" (OuterVolumeSpecName: "model-storage") pod "998b183e-1ab2-44c4-8ff8-96b69eb7b07b" (UID: "998b183e-1ab2-44c4-8ff8-96b69eb7b07b"). InnerVolumeSpecName "pvc-a73e9557-7c78-435b-b2fe-17034bfc65b1". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 27 18:47:51.180115 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:51.180085    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5zpq\" (UniqueName: \"kubernetes.io/projected/998b183e-1ab2-44c4-8ff8-96b69eb7b07b-kube-api-access-s5zpq\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:47:51.180283 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:51.180128    2557 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-a73e9557-7c78-435b-b2fe-17034bfc65b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04f045b2a76e32ea2\") on node \"ip-10-0-131-114.ec2.internal\" "
Jul 27 18:47:51.289041 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:51.289016    2557 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-a73e9557-7c78-435b-b2fe-17034bfc65b1" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-04f045b2a76e32ea2") on node "ip-10-0-131-114.ec2.internal"
Jul 27 18:47:51.381505 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:51.381474    2557 reconciler_common.go:299] "Volume detached for volume \"pvc-a73e9557-7c78-435b-b2fe-17034bfc65b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04f045b2a76e32ea2\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"csi-effe7b06b46bdf10d56898c66e78d47c7fe3a9615a8eae84b539ba087163f687\""
Jul 27 18:47:51.890862 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:51.890828    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-bwgsg"
Jul 27 18:47:51.890862 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:51.890852    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-bwgsg" event={"ID":"998b183e-1ab2-44c4-8ff8-96b69eb7b07b","Type":"ContainerDied","Data":"529b75a683c8af0a9a4bc45658712502b278cc8a507bb3072ddf5494c773f434"}
Jul 27 18:47:51.891136 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:47:51.890883    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="529b75a683c8af0a9a4bc45658712502b278cc8a507bb3072ddf5494c773f434"
Jul 27 18:48:24.027568 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:24.027527    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 27 18:48:24.028326 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:24.028003    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="998b183e-1ab2-44c4-8ff8-96b69eb7b07b" containerName="storage-initializer"
Jul 27 18:48:24.028326 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:24.028022    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="998b183e-1ab2-44c4-8ff8-96b69eb7b07b" containerName="storage-initializer"
Jul 27 18:48:24.028326 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:24.028127    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="998b183e-1ab2-44c4-8ff8-96b69eb7b07b" containerName="storage-initializer"
Jul 27 18:48:24.032172 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:24.032141    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:48:24.036850 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:24.036824    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-g5bb7\""
Jul 27 18:48:24.036985 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:24.036909    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 27 18:48:24.036985 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:24.036956    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-s2b5n\""
Jul 27 18:48:24.036985 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:24.036966    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"kube-root-ca.crt\""
Jul 27 18:48:24.037153 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:24.037040    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"openshift-service-ca.crt\""
Jul 27 18:48:24.043051 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:24.042839    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 27 18:48:24.182082 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:24.182054    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74lvd\" (UniqueName: \"kubernetes.io/projected/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-kube-api-access-74lvd\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:48:24.182237 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:24.182193    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:48:24.182237 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:24.182232    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:48:24.182333 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:24.182264    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:48:24.182378 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:24.182343    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:48:24.182490 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:24.182468    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:48:24.283600 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:24.283525    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74lvd\" (UniqueName: \"kubernetes.io/projected/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-kube-api-access-74lvd\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:48:24.283600 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:24.283589    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:48:24.283852 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:24.283606    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:48:24.283852 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:24.283624    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:48:24.283852 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:24.283649    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:48:24.283852 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:24.283675    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:48:24.284052 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:24.284035    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:48:24.284145 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:24.284127    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:48:24.284209 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:24.284128    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:48:24.286039 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:24.286005    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:48:24.286175 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:24.286086    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:48:24.292957 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:24.292939    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74lvd\" (UniqueName: \"kubernetes.io/projected/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-kube-api-access-74lvd\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:48:32.055348 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:32.055307    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-a73e9557-7c78-435b-b2fe-17034bfc65b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04f045b2a76e32ea2\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:48:32.060516 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:32.060493    2557 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-a73e9557-7c78-435b-b2fe-17034bfc65b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04f045b2a76e32ea2\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\") device path: \"\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:48:32.156408 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:32.156370    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-a73e9557-7c78-435b-b2fe-17034bfc65b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04f045b2a76e32ea2\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:48:32.156581 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:32.156487    2557 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-a73e9557-7c78-435b-b2fe-17034bfc65b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04f045b2a76e32ea2\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\") DevicePath \"\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:48:32.158906 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:32.158879    2557 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-a73e9557-7c78-435b-b2fe-17034bfc65b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04f045b2a76e32ea2\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\") DevicePath \"csi-effe7b06b46bdf10d56898c66e78d47c7fe3a9615a8eae84b539ba087163f687\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:48:32.248002 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:32.247968    2557 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-a73e9557-7c78-435b-b2fe-17034bfc65b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04f045b2a76e32ea2\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/180c16756ceeec634bbc4ba862f177a0ad3e8e06202a20fd191a4d6d9c121ac6/globalmount\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:48:32.261114 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:32.261085    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-a73e9557-7c78-435b-b2fe-17034bfc65b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04f045b2a76e32ea2\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:48:32.449422 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:32.449386    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-g5bb7\""
Jul 27 18:48:32.459585 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:32.459555    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-s2b5n\""
Jul 27 18:48:32.467930 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:32.467910    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:48:32.606317 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:32.606290    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 27 18:48:32.608226 ip-10-0-131-114 kubenswrapper[2557]: W0727 18:48:32.608197    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e52d58a_a79c_4871_bea8_fb8a4a76bd18.slice/crio-5a5c14942fd506cf7d2546d1e65bafe49ea47fc388cecddc3ef6dfce49361776 WatchSource:0}: Error finding container 5a5c14942fd506cf7d2546d1e65bafe49ea47fc388cecddc3ef6dfce49361776: Status 404 returned error can't find the container with id 5a5c14942fd506cf7d2546d1e65bafe49ea47fc388cecddc3ef6dfce49361776
Jul 27 18:48:33.101642 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:33.101599    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"0e52d58a-a79c-4871-bea8-fb8a4a76bd18","Type":"ContainerStarted","Data":"557b92505c91bdab36f40b7b5a3e752e16cdc3baedebe2be34eea4fcd20e24c8"}
Jul 27 18:48:33.102165 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:33.101675    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"0e52d58a-a79c-4871-bea8-fb8a4a76bd18","Type":"ContainerStarted","Data":"5a5c14942fd506cf7d2546d1e65bafe49ea47fc388cecddc3ef6dfce49361776"}
Jul 27 18:48:33.119528 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:33.119462    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podStartSLOduration=10.119439434 podStartE2EDuration="10.119439434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:48:33.11907305 +0000 UTC m=+4805.605884342" watchObservedRunningTime="2026-07-27 18:48:33.119439434 +0000 UTC m=+4805.606250722"
Jul 27 18:48:33.468671 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:33.468633    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:48:42.468515 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:42.468480    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:48:42.470047 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:42.470023    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0e52d58a-a79c-4871-bea8-fb8a4a76bd18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 27 18:48:52.469019 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:48:52.468972    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0e52d58a-a79c-4871-bea8-fb8a4a76bd18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 27 18:49:02.468631 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:49:02.468586    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0e52d58a-a79c-4871-bea8-fb8a4a76bd18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 27 18:49:12.469169 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:49:12.469123    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0e52d58a-a79c-4871-bea8-fb8a4a76bd18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 27 18:49:22.469417 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:49:22.469325    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0e52d58a-a79c-4871-bea8-fb8a4a76bd18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 27 18:49:32.468360 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:49:32.468310    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0e52d58a-a79c-4871-bea8-fb8a4a76bd18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 27 18:49:42.469327 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:49:42.469274    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0e52d58a-a79c-4871-bea8-fb8a4a76bd18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 27 18:49:52.469313 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:49:52.469271    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0e52d58a-a79c-4871-bea8-fb8a4a76bd18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 27 18:50:02.469412 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:02.469359    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0e52d58a-a79c-4871-bea8-fb8a4a76bd18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 27 18:50:12.478476 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:12.478440    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:50:12.485799 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:12.485774    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:50:31.728897 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:31.728859    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 27 18:50:46.751916 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:46.751854    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0e52d58a-a79c-4871-bea8-fb8a4a76bd18" containerName="main" containerID="cri-o://557b92505c91bdab36f40b7b5a3e752e16cdc3baedebe2be34eea4fcd20e24c8" gracePeriod=45
Jul 27 18:50:47.506906 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.506877    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:50:47.579944 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.579840    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04f045b2a76e32ea2\") pod \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\" (UID: \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\") "
Jul 27 18:50:47.579944 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.579904    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74lvd\" (UniqueName: \"kubernetes.io/projected/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-kube-api-access-74lvd\") pod \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\" (UID: \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\") "
Jul 27 18:50:47.579944 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.579930    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-model-cache\") pod \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\" (UID: \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\") "
Jul 27 18:50:47.580238 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.579954    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-dshm\") pod \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\" (UID: \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\") "
Jul 27 18:50:47.580238 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.579973    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-tls-certs\") pod \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\" (UID: \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\") "
Jul 27 18:50:47.580238 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.579996    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-tmp-dir\") pod \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\" (UID: \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\") "
Jul 27 18:50:47.580238 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.580052    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-home\") pod \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\" (UID: \"0e52d58a-a79c-4871-bea8-fb8a4a76bd18\") "
Jul 27 18:50:47.580238 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.580226    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-model-cache" (OuterVolumeSpecName: "model-cache") pod "0e52d58a-a79c-4871-bea8-fb8a4a76bd18" (UID: "0e52d58a-a79c-4871-bea8-fb8a4a76bd18"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:50:47.580497 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.580323    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-model-cache\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:50:47.580845 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.580821    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-home" (OuterVolumeSpecName: "home") pod "0e52d58a-a79c-4871-bea8-fb8a4a76bd18" (UID: "0e52d58a-a79c-4871-bea8-fb8a4a76bd18"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:50:47.582807 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.582744    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-dshm" (OuterVolumeSpecName: "dshm") pod "0e52d58a-a79c-4871-bea8-fb8a4a76bd18" (UID: "0e52d58a-a79c-4871-bea8-fb8a4a76bd18"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:50:47.582807 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.582784    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0e52d58a-a79c-4871-bea8-fb8a4a76bd18" (UID: "0e52d58a-a79c-4871-bea8-fb8a4a76bd18"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:50:47.582983 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.582809    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-kube-api-access-74lvd" (OuterVolumeSpecName: "kube-api-access-74lvd") pod "0e52d58a-a79c-4871-bea8-fb8a4a76bd18" (UID: "0e52d58a-a79c-4871-bea8-fb8a4a76bd18"). InnerVolumeSpecName "kube-api-access-74lvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:50:47.583155 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.583134    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-04f045b2a76e32ea2" (OuterVolumeSpecName: "kserve-pvc-source") pod "0e52d58a-a79c-4871-bea8-fb8a4a76bd18" (UID: "0e52d58a-a79c-4871-bea8-fb8a4a76bd18"). InnerVolumeSpecName "pvc-a73e9557-7c78-435b-b2fe-17034bfc65b1". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 27 18:50:47.592904 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.592876    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0e52d58a-a79c-4871-bea8-fb8a4a76bd18" (UID: "0e52d58a-a79c-4871-bea8-fb8a4a76bd18"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:50:47.680746 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.680709    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74lvd\" (UniqueName: \"kubernetes.io/projected/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-kube-api-access-74lvd\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:50:47.680746 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.680739    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-dshm\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:50:47.680746 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.680770    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-tls-certs\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:50:47.680746 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.680780    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-tmp-dir\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:50:47.681054 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.680788    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e52d58a-a79c-4871-bea8-fb8a4a76bd18-home\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:50:47.681054 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.680814    2557 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-a73e9557-7c78-435b-b2fe-17034bfc65b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04f045b2a76e32ea2\") on node \"ip-10-0-131-114.ec2.internal\" "
Jul 27 18:50:47.708414 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.708393    2557 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-a73e9557-7c78-435b-b2fe-17034bfc65b1" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-04f045b2a76e32ea2") on node "ip-10-0-131-114.ec2.internal"
Jul 27 18:50:47.781681 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.781643    2557 reconciler_common.go:299] "Volume detached for volume \"pvc-a73e9557-7c78-435b-b2fe-17034bfc65b1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04f045b2a76e32ea2\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"csi-effe7b06b46bdf10d56898c66e78d47c7fe3a9615a8eae84b539ba087163f687\""
Jul 27 18:50:47.787350 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.787318    2557 generic.go:356] "Generic (PLEG): container finished" podID="0e52d58a-a79c-4871-bea8-fb8a4a76bd18" containerID="557b92505c91bdab36f40b7b5a3e752e16cdc3baedebe2be34eea4fcd20e24c8" exitCode=0
Jul 27 18:50:47.787496 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.787369    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 27 18:50:47.787496 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.787403    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"0e52d58a-a79c-4871-bea8-fb8a4a76bd18","Type":"ContainerDied","Data":"557b92505c91bdab36f40b7b5a3e752e16cdc3baedebe2be34eea4fcd20e24c8"}
Jul 27 18:50:47.787496 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.787459    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"0e52d58a-a79c-4871-bea8-fb8a4a76bd18","Type":"ContainerDied","Data":"5a5c14942fd506cf7d2546d1e65bafe49ea47fc388cecddc3ef6dfce49361776"}
Jul 27 18:50:47.787496 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.787481    2557 scope.go:117] "RemoveContainer" containerID="557b92505c91bdab36f40b7b5a3e752e16cdc3baedebe2be34eea4fcd20e24c8"
Jul 27 18:50:47.798165 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.798143    2557 scope.go:117] "RemoveContainer" containerID="557b92505c91bdab36f40b7b5a3e752e16cdc3baedebe2be34eea4fcd20e24c8"
Jul 27 18:50:47.798423 ip-10-0-131-114 kubenswrapper[2557]: E0727 18:50:47.798403    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"557b92505c91bdab36f40b7b5a3e752e16cdc3baedebe2be34eea4fcd20e24c8\": container with ID starting with 557b92505c91bdab36f40b7b5a3e752e16cdc3baedebe2be34eea4fcd20e24c8 not found: ID does not exist" containerID="557b92505c91bdab36f40b7b5a3e752e16cdc3baedebe2be34eea4fcd20e24c8"
Jul 27 18:50:47.798489 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.798434    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"557b92505c91bdab36f40b7b5a3e752e16cdc3baedebe2be34eea4fcd20e24c8"} err="failed to get container status \"557b92505c91bdab36f40b7b5a3e752e16cdc3baedebe2be34eea4fcd20e24c8\": rpc error: code = NotFound desc = could not find container \"557b92505c91bdab36f40b7b5a3e752e16cdc3baedebe2be34eea4fcd20e24c8\": container with ID starting with 557b92505c91bdab36f40b7b5a3e752e16cdc3baedebe2be34eea4fcd20e24c8 not found: ID does not exist"
Jul 27 18:50:47.809635 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.809613    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 27 18:50:47.815321 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:47.815299    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 27 18:50:48.119330 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:48.119296    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e52d58a-a79c-4871-bea8-fb8a4a76bd18" path="/var/lib/kubelet/pods/0e52d58a-a79c-4871-bea8-fb8a4a76bd18/volumes"
Jul 27 18:50:48.479894 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:48.479850    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0e52d58a-a79c-4871-bea8-fb8a4a76bd18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 27 18:50:56.333370 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:56.333295    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-bwgsg"]
Jul 27 18:50:56.337432 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:56.337400    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-bwgsg"]
Jul 27 18:50:58.118632 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:50:58.118600    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="998b183e-1ab2-44c4-8ff8-96b69eb7b07b" path="/var/lib/kubelet/pods/998b183e-1ab2-44c4-8ff8-96b69eb7b07b/volumes"
Jul 27 18:51:09.389352 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.389320    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq"]
Jul 27 18:51:09.389739 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.389692    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e52d58a-a79c-4871-bea8-fb8a4a76bd18" containerName="main"
Jul 27 18:51:09.389739 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.389706    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e52d58a-a79c-4871-bea8-fb8a4a76bd18" containerName="main"
Jul 27 18:51:09.389838 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.389784    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e52d58a-a79c-4871-bea8-fb8a4a76bd18" containerName="main"
Jul 27 18:51:09.392947 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.392929    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq"
Jul 27 18:51:09.394971 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.394947    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"seaweedfs-s3-creds\""
Jul 27 18:51:09.394971 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.394961    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-n4grp\""
Jul 27 18:51:09.395206 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.395190    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Jul 27 18:51:09.395312 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.395298    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Jul 27 18:51:09.395392 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.395377    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Jul 27 18:51:09.403709 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.403684    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq"]
Jul 27 18:51:09.472872 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.472841    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq\" (UID: \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq"
Jul 27 18:51:09.473048 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.472893    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-tls-certs\") pod \"e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq\" (UID: \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq"
Jul 27 18:51:09.473048 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.472917    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkmbk\" (UniqueName: \"kubernetes.io/projected/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-kube-api-access-vkmbk\") pod \"e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq\" (UID: \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq"
Jul 27 18:51:09.473048 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.473007    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-dshm\") pod \"e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq\" (UID: \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq"
Jul 27 18:51:09.473048 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.473043    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-home\") pod \"e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq\" (UID: \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq"
Jul 27 18:51:09.473201 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.473067    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-model-cache\") pod \"e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq\" (UID: \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq"
Jul 27 18:51:09.473201 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.473099    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq\" (UID: \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq"
Jul 27 18:51:09.574150 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.574114    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-dshm\") pod \"e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq\" (UID: \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq"
Jul 27 18:51:09.574150 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.574155    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-home\") pod \"e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq\" (UID: \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq"
Jul 27 18:51:09.574406 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.574176    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-model-cache\") pod \"e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq\" (UID: \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq"
Jul 27 18:51:09.574406 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.574192    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq\" (UID: \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq"
Jul 27 18:51:09.574406 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.574240    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq\" (UID: \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq"
Jul 27 18:51:09.574406 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.574269    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-tls-certs\") pod \"e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq\" (UID: \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq"
Jul 27 18:51:09.574406 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.574286    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkmbk\" (UniqueName: \"kubernetes.io/projected/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-kube-api-access-vkmbk\") pod \"e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq\" (UID: \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq"
Jul 27 18:51:09.574682 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.574616    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-home\") pod \"e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq\" (UID: \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq"
Jul 27 18:51:09.574739 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.574677    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-model-cache\") pod \"e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq\" (UID: \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq"
Jul 27 18:51:09.574821 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.574746    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq\" (UID: \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq"
Jul 27 18:51:09.574875 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.574818    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq\" (UID: \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq"
Jul 27 18:51:09.576561 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.576533    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-dshm\") pod \"e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq\" (UID: \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq"
Jul 27 18:51:09.576865 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.576843    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-tls-certs\") pod \"e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq\" (UID: \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq"
Jul 27 18:51:09.580958 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.580942    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkmbk\" (UniqueName: \"kubernetes.io/projected/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-kube-api-access-vkmbk\") pod \"e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq\" (UID: \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq"
Jul 27 18:51:09.704953 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.704916    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq"
Jul 27 18:51:09.846911 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.846876    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq"]
Jul 27 18:51:09.850271 ip-10-0-131-114 kubenswrapper[2557]: W0727 18:51:09.850242    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ea0d14e_a18e_4e36_92c4_32b05d2de9d9.slice/crio-da709c12904c51729d56fbcd610fd2dd1e9857ac7bd450832c0f6aa274e06812 WatchSource:0}: Error finding container da709c12904c51729d56fbcd610fd2dd1e9857ac7bd450832c0f6aa274e06812: Status 404 returned error can't find the container with id da709c12904c51729d56fbcd610fd2dd1e9857ac7bd450832c0f6aa274e06812
Jul 27 18:51:09.892805 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:09.892779    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq" event={"ID":"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9","Type":"ContainerStarted","Data":"da709c12904c51729d56fbcd610fd2dd1e9857ac7bd450832c0f6aa274e06812"}
Jul 27 18:51:10.900595 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:10.900554    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq" event={"ID":"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9","Type":"ContainerStarted","Data":"ae49849bb56d7c67e0230cc8ca3af398eb81629b8e46e31fbc0895d3d7368340"}
Jul 27 18:51:18.939076 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:18.939037    2557 generic.go:356] "Generic (PLEG): container finished" podID="0ea0d14e-a18e-4e36-92c4-32b05d2de9d9" containerID="ae49849bb56d7c67e0230cc8ca3af398eb81629b8e46e31fbc0895d3d7368340" exitCode=0
Jul 27 18:51:18.939507 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:18.939111    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq" event={"ID":"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9","Type":"ContainerDied","Data":"ae49849bb56d7c67e0230cc8ca3af398eb81629b8e46e31fbc0895d3d7368340"}
Jul 27 18:51:19.109450 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:19.109415    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq"]
Jul 27 18:51:19.947118 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:19.947075    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq" event={"ID":"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9","Type":"ContainerStarted","Data":"9a179b906a758f69c7341b583c3bbece5b7ed0eafff9b4765c69061a3a4adfcc"}
Jul 27 18:51:19.969145 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:19.969077    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq" podStartSLOduration=2.553770119 podStartE2EDuration="10.969054266s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.415284147s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:51:19.966745506 +0000 UTC m=+4972.453556818" watchObservedRunningTime="2026-07-27 18:51:19.969054266 +0000 UTC m=+4972.455865553"
Jul 27 18:51:20.705934 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:20.705900    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq"
Jul 27 18:51:34.971746 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:51:34.971671    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq" podUID="0ea0d14e-a18e-4e36-92c4-32b05d2de9d9" containerName="main" containerID="cri-o://9a179b906a758f69c7341b583c3bbece5b7ed0eafff9b4765c69061a3a4adfcc" gracePeriod=45
Jul 27 18:52:20.270224 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:20.270117    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-config-deletion-blocked-when-referenced-0f029b85_e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq_0ea0d14e-a18e-4e36-92c4-32b05d2de9d9/main/0.log"
Jul 27 18:52:20.270671 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:20.270499    2557 generic.go:356] "Generic (PLEG): container finished" podID="0ea0d14e-a18e-4e36-92c4-32b05d2de9d9" containerID="9a179b906a758f69c7341b583c3bbece5b7ed0eafff9b4765c69061a3a4adfcc" exitCode=137
Jul 27 18:52:20.270671 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:20.270534    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq" event={"ID":"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9","Type":"ContainerDied","Data":"9a179b906a758f69c7341b583c3bbece5b7ed0eafff9b4765c69061a3a4adfcc"}
Jul 27 18:52:20.624539 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:20.624515    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-config-deletion-blocked-when-referenced-0f029b85_e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq_0ea0d14e-a18e-4e36-92c4-32b05d2de9d9/main/0.log"
Jul 27 18:52:20.627459 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:20.627439    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq"
Jul 27 18:52:20.633279 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:20.633259    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-model-cache\") pod \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\" (UID: \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\") "
Jul 27 18:52:20.633370 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:20.633287    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-tmp-dir\") pod \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\" (UID: \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\") "
Jul 27 18:52:20.633370 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:20.633314    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-tls-certs\") pod \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\" (UID: \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\") "
Jul 27 18:52:20.633370 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:20.633345    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-dshm\") pod \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\" (UID: \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\") "
Jul 27 18:52:20.633370 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:20.633365    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-kserve-provision-location\") pod \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\" (UID: \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\") "
Jul 27 18:52:20.633586 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:20.633395    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-home\") pod \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\" (UID: \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\") "
Jul 27 18:52:20.633586 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:20.633463    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkmbk\" (UniqueName: \"kubernetes.io/projected/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-kube-api-access-vkmbk\") pod \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\" (UID: \"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9\") "
Jul 27 18:52:20.633586 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:20.633540    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "0ea0d14e-a18e-4e36-92c4-32b05d2de9d9" (UID: "0ea0d14e-a18e-4e36-92c4-32b05d2de9d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:52:20.633780 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:20.633735    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-model-cache\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:52:20.633900 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:20.633807    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0ea0d14e-a18e-4e36-92c4-32b05d2de9d9" (UID: "0ea0d14e-a18e-4e36-92c4-32b05d2de9d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:52:20.634144 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:20.634118    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-home" (OuterVolumeSpecName: "home") pod "0ea0d14e-a18e-4e36-92c4-32b05d2de9d9" (UID: "0ea0d14e-a18e-4e36-92c4-32b05d2de9d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:52:20.636124 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:20.636101    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0ea0d14e-a18e-4e36-92c4-32b05d2de9d9" (UID: "0ea0d14e-a18e-4e36-92c4-32b05d2de9d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:52:20.636224 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:20.636146    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-dshm" (OuterVolumeSpecName: "dshm") pod "0ea0d14e-a18e-4e36-92c4-32b05d2de9d9" (UID: "0ea0d14e-a18e-4e36-92c4-32b05d2de9d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:52:20.636224 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:20.636163    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-kube-api-access-vkmbk" (OuterVolumeSpecName: "kube-api-access-vkmbk") pod "0ea0d14e-a18e-4e36-92c4-32b05d2de9d9" (UID: "0ea0d14e-a18e-4e36-92c4-32b05d2de9d9"). InnerVolumeSpecName "kube-api-access-vkmbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:52:20.704217 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:20.704164    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0ea0d14e-a18e-4e36-92c4-32b05d2de9d9" (UID: "0ea0d14e-a18e-4e36-92c4-32b05d2de9d9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:52:20.737224 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:20.737076    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkmbk\" (UniqueName: \"kubernetes.io/projected/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-kube-api-access-vkmbk\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:52:20.737394 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:20.737235    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-tmp-dir\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:52:20.737394 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:20.737255    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-tls-certs\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:52:20.737394 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:20.737266    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-dshm\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:52:20.737394 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:20.737279    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-kserve-provision-location\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:52:20.737394 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:20.737290    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9-home\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:52:21.276674 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:21.276644    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-config-deletion-blocked-when-referenced-0f029b85_e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq_0ea0d14e-a18e-4e36-92c4-32b05d2de9d9/main/0.log"
Jul 27 18:52:21.277110 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:21.277043    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq" event={"ID":"0ea0d14e-a18e-4e36-92c4-32b05d2de9d9","Type":"ContainerDied","Data":"da709c12904c51729d56fbcd610fd2dd1e9857ac7bd450832c0f6aa274e06812"}
Jul 27 18:52:21.277110 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:21.277063    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq"
Jul 27 18:52:21.277110 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:21.277103    2557 scope.go:117] "RemoveContainer" containerID="9a179b906a758f69c7341b583c3bbece5b7ed0eafff9b4765c69061a3a4adfcc"
Jul 27 18:52:21.286731 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:21.286715    2557 scope.go:117] "RemoveContainer" containerID="ae49849bb56d7c67e0230cc8ca3af398eb81629b8e46e31fbc0895d3d7368340"
Jul 27 18:52:21.299603 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:21.299576    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq"]
Jul 27 18:52:21.304485 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:21.304458    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-58c69ddfd-gt6lq"]
Jul 27 18:52:22.120473 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:52:22.120432    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ea0d14e-a18e-4e36-92c4-32b05d2de9d9" path="/var/lib/kubelet/pods/0ea0d14e-a18e-4e36-92c4-32b05d2de9d9/volumes"
Jul 27 18:54:27.509499 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.509460    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"]
Jul 27 18:54:27.510397 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.510061    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ea0d14e-a18e-4e36-92c4-32b05d2de9d9" containerName="storage-initializer"
Jul 27 18:54:27.510397 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.510081    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ea0d14e-a18e-4e36-92c4-32b05d2de9d9" containerName="storage-initializer"
Jul 27 18:54:27.510397 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.510092    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ea0d14e-a18e-4e36-92c4-32b05d2de9d9" containerName="main"
Jul 27 18:54:27.510397 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.510101    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ea0d14e-a18e-4e36-92c4-32b05d2de9d9" containerName="main"
Jul 27 18:54:27.510397 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.510205    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ea0d14e-a18e-4e36-92c4-32b05d2de9d9" containerName="main"
Jul 27 18:54:27.513344 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.513323    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"
Jul 27 18:54:27.515725 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.515697    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Jul 27 18:54:27.515725 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.515709    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 27 18:54:27.515941 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.515732    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-wt2gj\""
Jul 27 18:54:27.515941 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.515699    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Jul 27 18:54:27.515941 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.515824    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Jul 27 18:54:27.523849 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.523826    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"]
Jul 27 18:54:27.634881 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.634835    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772d0671-cc65-4489-b2d7-4ecb17336934-home\") pod \"stop-feature-test-kserve-57d79c4c49-c89fn\" (UID: \"772d0671-cc65-4489-b2d7-4ecb17336934\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"
Jul 27 18:54:27.635096 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.634889    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772d0671-cc65-4489-b2d7-4ecb17336934-tls-certs\") pod \"stop-feature-test-kserve-57d79c4c49-c89fn\" (UID: \"772d0671-cc65-4489-b2d7-4ecb17336934\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"
Jul 27 18:54:27.635096 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.634933    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772d0671-cc65-4489-b2d7-4ecb17336934-dshm\") pod \"stop-feature-test-kserve-57d79c4c49-c89fn\" (UID: \"772d0671-cc65-4489-b2d7-4ecb17336934\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"
Jul 27 18:54:27.635096 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.635021    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772d0671-cc65-4489-b2d7-4ecb17336934-model-cache\") pod \"stop-feature-test-kserve-57d79c4c49-c89fn\" (UID: \"772d0671-cc65-4489-b2d7-4ecb17336934\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"
Jul 27 18:54:27.635096 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.635064    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/772d0671-cc65-4489-b2d7-4ecb17336934-kserve-provision-location\") pod \"stop-feature-test-kserve-57d79c4c49-c89fn\" (UID: \"772d0671-cc65-4489-b2d7-4ecb17336934\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"
Jul 27 18:54:27.635269 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.635158    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772d0671-cc65-4489-b2d7-4ecb17336934-tmp-dir\") pod \"stop-feature-test-kserve-57d79c4c49-c89fn\" (UID: \"772d0671-cc65-4489-b2d7-4ecb17336934\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"
Jul 27 18:54:27.635269 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.635195    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h87mg\" (UniqueName: \"kubernetes.io/projected/772d0671-cc65-4489-b2d7-4ecb17336934-kube-api-access-h87mg\") pod \"stop-feature-test-kserve-57d79c4c49-c89fn\" (UID: \"772d0671-cc65-4489-b2d7-4ecb17336934\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"
Jul 27 18:54:27.736247 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.736196    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/772d0671-cc65-4489-b2d7-4ecb17336934-kserve-provision-location\") pod \"stop-feature-test-kserve-57d79c4c49-c89fn\" (UID: \"772d0671-cc65-4489-b2d7-4ecb17336934\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"
Jul 27 18:54:27.736445 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.736286    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772d0671-cc65-4489-b2d7-4ecb17336934-tmp-dir\") pod \"stop-feature-test-kserve-57d79c4c49-c89fn\" (UID: \"772d0671-cc65-4489-b2d7-4ecb17336934\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"
Jul 27 18:54:27.736445 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.736321    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h87mg\" (UniqueName: \"kubernetes.io/projected/772d0671-cc65-4489-b2d7-4ecb17336934-kube-api-access-h87mg\") pod \"stop-feature-test-kserve-57d79c4c49-c89fn\" (UID: \"772d0671-cc65-4489-b2d7-4ecb17336934\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"
Jul 27 18:54:27.736445 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.736376    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772d0671-cc65-4489-b2d7-4ecb17336934-home\") pod \"stop-feature-test-kserve-57d79c4c49-c89fn\" (UID: \"772d0671-cc65-4489-b2d7-4ecb17336934\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"
Jul 27 18:54:27.736445 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.736401    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772d0671-cc65-4489-b2d7-4ecb17336934-tls-certs\") pod \"stop-feature-test-kserve-57d79c4c49-c89fn\" (UID: \"772d0671-cc65-4489-b2d7-4ecb17336934\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"
Jul 27 18:54:27.736445 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.736437    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772d0671-cc65-4489-b2d7-4ecb17336934-dshm\") pod \"stop-feature-test-kserve-57d79c4c49-c89fn\" (UID: \"772d0671-cc65-4489-b2d7-4ecb17336934\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"
Jul 27 18:54:27.736713 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.736479    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772d0671-cc65-4489-b2d7-4ecb17336934-model-cache\") pod \"stop-feature-test-kserve-57d79c4c49-c89fn\" (UID: \"772d0671-cc65-4489-b2d7-4ecb17336934\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"
Jul 27 18:54:27.736713 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.736628    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/772d0671-cc65-4489-b2d7-4ecb17336934-kserve-provision-location\") pod \"stop-feature-test-kserve-57d79c4c49-c89fn\" (UID: \"772d0671-cc65-4489-b2d7-4ecb17336934\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"
Jul 27 18:54:27.736713 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.736682    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772d0671-cc65-4489-b2d7-4ecb17336934-tmp-dir\") pod \"stop-feature-test-kserve-57d79c4c49-c89fn\" (UID: \"772d0671-cc65-4489-b2d7-4ecb17336934\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"
Jul 27 18:54:27.736913 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.736830    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772d0671-cc65-4489-b2d7-4ecb17336934-home\") pod \"stop-feature-test-kserve-57d79c4c49-c89fn\" (UID: \"772d0671-cc65-4489-b2d7-4ecb17336934\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"
Jul 27 18:54:27.736913 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.736885    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772d0671-cc65-4489-b2d7-4ecb17336934-model-cache\") pod \"stop-feature-test-kserve-57d79c4c49-c89fn\" (UID: \"772d0671-cc65-4489-b2d7-4ecb17336934\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"
Jul 27 18:54:27.738793 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.738745    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772d0671-cc65-4489-b2d7-4ecb17336934-dshm\") pod \"stop-feature-test-kserve-57d79c4c49-c89fn\" (UID: \"772d0671-cc65-4489-b2d7-4ecb17336934\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"
Jul 27 18:54:27.739058 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.739042    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772d0671-cc65-4489-b2d7-4ecb17336934-tls-certs\") pod \"stop-feature-test-kserve-57d79c4c49-c89fn\" (UID: \"772d0671-cc65-4489-b2d7-4ecb17336934\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"
Jul 27 18:54:27.744014 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.743996    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h87mg\" (UniqueName: \"kubernetes.io/projected/772d0671-cc65-4489-b2d7-4ecb17336934-kube-api-access-h87mg\") pod \"stop-feature-test-kserve-57d79c4c49-c89fn\" (UID: \"772d0671-cc65-4489-b2d7-4ecb17336934\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"
Jul 27 18:54:27.824997 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.824907    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"
Jul 27 18:54:27.962698 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.962666    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"]
Jul 27 18:54:27.965418 ip-10-0-131-114 kubenswrapper[2557]: W0727 18:54:27.965393    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod772d0671_cc65_4489_b2d7_4ecb17336934.slice/crio-6ebe96d1b89b83fc0a57dab0dd41b86aad0910ab5c5ce6793ea41b9e8b6c987a WatchSource:0}: Error finding container 6ebe96d1b89b83fc0a57dab0dd41b86aad0910ab5c5ce6793ea41b9e8b6c987a: Status 404 returned error can't find the container with id 6ebe96d1b89b83fc0a57dab0dd41b86aad0910ab5c5ce6793ea41b9e8b6c987a
Jul 27 18:54:27.969567 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:27.969549    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Jul 27 18:54:28.645014 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:28.644981    2557 scope.go:117] "RemoveContainer" containerID="7066c31067b760052d6aab411fa2d74950caec9205104ab1ef356e8a8ddfc3de"
Jul 27 18:54:28.897927 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:28.897836    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn" event={"ID":"772d0671-cc65-4489-b2d7-4ecb17336934","Type":"ContainerStarted","Data":"2250726a535acb6faa9b0d955638db33bdbbe5d4dd99cd4a916d8c860031ed56"}
Jul 27 18:54:28.897927 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:28.897899    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn" event={"ID":"772d0671-cc65-4489-b2d7-4ecb17336934","Type":"ContainerStarted","Data":"6ebe96d1b89b83fc0a57dab0dd41b86aad0910ab5c5ce6793ea41b9e8b6c987a"}
Jul 27 18:54:35.939245 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:35.939208    2557 generic.go:356] "Generic (PLEG): container finished" podID="772d0671-cc65-4489-b2d7-4ecb17336934" containerID="2250726a535acb6faa9b0d955638db33bdbbe5d4dd99cd4a916d8c860031ed56" exitCode=0
Jul 27 18:54:35.939678 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:35.939282    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn" event={"ID":"772d0671-cc65-4489-b2d7-4ecb17336934","Type":"ContainerDied","Data":"2250726a535acb6faa9b0d955638db33bdbbe5d4dd99cd4a916d8c860031ed56"}
Jul 27 18:54:36.947005 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:36.946972    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn" event={"ID":"772d0671-cc65-4489-b2d7-4ecb17336934","Type":"ContainerStarted","Data":"bc92b4d968b673bf3288bd370386245d3272b27c85f034e28dacf27259f7522b"}
Jul 27 18:54:36.967108 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:36.967053    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn" podStartSLOduration=2.855929031 podStartE2EDuration="9.967026797s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.111097766s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:54:36.965013751 +0000 UTC m=+5169.451825038" watchObservedRunningTime="2026-07-27 18:54:36.967026797 +0000 UTC m=+5169.453838085"
Jul 27 18:54:37.825181 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:37.825147    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"
Jul 27 18:54:37.825355 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:37.825206    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"
Jul 27 18:54:37.826744 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:37.826712    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn" podUID="772d0671-cc65-4489-b2d7-4ecb17336934" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 27 18:54:47.825657 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:47.825616    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn" podUID="772d0671-cc65-4489-b2d7-4ecb17336934" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 27 18:54:57.825777 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:54:57.825705    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn" podUID="772d0671-cc65-4489-b2d7-4ecb17336934" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 27 18:55:07.826098 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:55:07.826055    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn" podUID="772d0671-cc65-4489-b2d7-4ecb17336934" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 27 18:55:17.826377 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:55:17.826328    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn" podUID="772d0671-cc65-4489-b2d7-4ecb17336934" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 27 18:55:27.825968 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:55:27.825850    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn" podUID="772d0671-cc65-4489-b2d7-4ecb17336934" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 27 18:55:37.826204 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:55:37.826158    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn" podUID="772d0671-cc65-4489-b2d7-4ecb17336934" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 27 18:55:47.825809 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:55:47.825743    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn" podUID="772d0671-cc65-4489-b2d7-4ecb17336934" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 27 18:55:57.825594 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:55:57.825550    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn" podUID="772d0671-cc65-4489-b2d7-4ecb17336934" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 27 18:56:07.825357 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:56:07.825309    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn" podUID="772d0671-cc65-4489-b2d7-4ecb17336934" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 27 18:56:17.825746 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:56:17.825696    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn" podUID="772d0671-cc65-4489-b2d7-4ecb17336934" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 27 18:56:27.835574 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:56:27.835543    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"
Jul 27 18:56:27.843191 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:56:27.843173    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"
Jul 27 18:56:49.492618 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:56:49.492582    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"]
Jul 27 18:57:04.515734 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:04.515633    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn" podUID="772d0671-cc65-4489-b2d7-4ecb17336934" containerName="main" containerID="cri-o://bc92b4d968b673bf3288bd370386245d3272b27c85f034e28dacf27259f7522b" gracePeriod=45
Jul 27 18:57:04.836686 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:04.836581    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn" podUID="772d0671-cc65-4489-b2d7-4ecb17336934" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 27 18:57:05.199131 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.199104    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"
Jul 27 18:57:05.272449 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.272423    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772d0671-cc65-4489-b2d7-4ecb17336934-tls-certs\") pod \"772d0671-cc65-4489-b2d7-4ecb17336934\" (UID: \"772d0671-cc65-4489-b2d7-4ecb17336934\") "
Jul 27 18:57:05.272628 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.272490    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772d0671-cc65-4489-b2d7-4ecb17336934-tmp-dir\") pod \"772d0671-cc65-4489-b2d7-4ecb17336934\" (UID: \"772d0671-cc65-4489-b2d7-4ecb17336934\") "
Jul 27 18:57:05.272628 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.272520    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772d0671-cc65-4489-b2d7-4ecb17336934-dshm\") pod \"772d0671-cc65-4489-b2d7-4ecb17336934\" (UID: \"772d0671-cc65-4489-b2d7-4ecb17336934\") "
Jul 27 18:57:05.272628 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.272597    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772d0671-cc65-4489-b2d7-4ecb17336934-model-cache\") pod \"772d0671-cc65-4489-b2d7-4ecb17336934\" (UID: \"772d0671-cc65-4489-b2d7-4ecb17336934\") "
Jul 27 18:57:05.272628 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.272627    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h87mg\" (UniqueName: \"kubernetes.io/projected/772d0671-cc65-4489-b2d7-4ecb17336934-kube-api-access-h87mg\") pod \"772d0671-cc65-4489-b2d7-4ecb17336934\" (UID: \"772d0671-cc65-4489-b2d7-4ecb17336934\") "
Jul 27 18:57:05.272895 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.272656    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772d0671-cc65-4489-b2d7-4ecb17336934-home\") pod \"772d0671-cc65-4489-b2d7-4ecb17336934\" (UID: \"772d0671-cc65-4489-b2d7-4ecb17336934\") "
Jul 27 18:57:05.272895 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.272729    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/772d0671-cc65-4489-b2d7-4ecb17336934-kserve-provision-location\") pod \"772d0671-cc65-4489-b2d7-4ecb17336934\" (UID: \"772d0671-cc65-4489-b2d7-4ecb17336934\") "
Jul 27 18:57:05.272895 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.272870    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772d0671-cc65-4489-b2d7-4ecb17336934-model-cache" (OuterVolumeSpecName: "model-cache") pod "772d0671-cc65-4489-b2d7-4ecb17336934" (UID: "772d0671-cc65-4489-b2d7-4ecb17336934"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:57:05.273122 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.273103    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772d0671-cc65-4489-b2d7-4ecb17336934-model-cache\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:57:05.273456 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.273422    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772d0671-cc65-4489-b2d7-4ecb17336934-home" (OuterVolumeSpecName: "home") pod "772d0671-cc65-4489-b2d7-4ecb17336934" (UID: "772d0671-cc65-4489-b2d7-4ecb17336934"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:57:05.274914 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.274859    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772d0671-cc65-4489-b2d7-4ecb17336934-kube-api-access-h87mg" (OuterVolumeSpecName: "kube-api-access-h87mg") pod "772d0671-cc65-4489-b2d7-4ecb17336934" (UID: "772d0671-cc65-4489-b2d7-4ecb17336934"). InnerVolumeSpecName "kube-api-access-h87mg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:57:05.274914 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.274868    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/772d0671-cc65-4489-b2d7-4ecb17336934-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "772d0671-cc65-4489-b2d7-4ecb17336934" (UID: "772d0671-cc65-4489-b2d7-4ecb17336934"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:57:05.275091 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.274957    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772d0671-cc65-4489-b2d7-4ecb17336934-dshm" (OuterVolumeSpecName: "dshm") pod "772d0671-cc65-4489-b2d7-4ecb17336934" (UID: "772d0671-cc65-4489-b2d7-4ecb17336934"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:57:05.285838 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.285810    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772d0671-cc65-4489-b2d7-4ecb17336934-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "772d0671-cc65-4489-b2d7-4ecb17336934" (UID: "772d0671-cc65-4489-b2d7-4ecb17336934"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:57:05.319425 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.319373    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772d0671-cc65-4489-b2d7-4ecb17336934-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "772d0671-cc65-4489-b2d7-4ecb17336934" (UID: "772d0671-cc65-4489-b2d7-4ecb17336934"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:57:05.374084 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.374006    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772d0671-cc65-4489-b2d7-4ecb17336934-home\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:57:05.374084 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.374036    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/772d0671-cc65-4489-b2d7-4ecb17336934-kserve-provision-location\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:57:05.374084 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.374045    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772d0671-cc65-4489-b2d7-4ecb17336934-tls-certs\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:57:05.374084 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.374055    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772d0671-cc65-4489-b2d7-4ecb17336934-tmp-dir\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:57:05.374084 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.374063    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772d0671-cc65-4489-b2d7-4ecb17336934-dshm\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:57:05.374084 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.374071    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h87mg\" (UniqueName: \"kubernetes.io/projected/772d0671-cc65-4489-b2d7-4ecb17336934-kube-api-access-h87mg\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:57:05.698926 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.698893    2557 generic.go:356] "Generic (PLEG): container finished" podID="772d0671-cc65-4489-b2d7-4ecb17336934" containerID="bc92b4d968b673bf3288bd370386245d3272b27c85f034e28dacf27259f7522b" exitCode=0
Jul 27 18:57:05.699374 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.698947    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"
Jul 27 18:57:05.699374 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.698974    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn" event={"ID":"772d0671-cc65-4489-b2d7-4ecb17336934","Type":"ContainerDied","Data":"bc92b4d968b673bf3288bd370386245d3272b27c85f034e28dacf27259f7522b"}
Jul 27 18:57:05.699374 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.699026    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn" event={"ID":"772d0671-cc65-4489-b2d7-4ecb17336934","Type":"ContainerDied","Data":"6ebe96d1b89b83fc0a57dab0dd41b86aad0910ab5c5ce6793ea41b9e8b6c987a"}
Jul 27 18:57:05.699374 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.699049    2557 scope.go:117] "RemoveContainer" containerID="bc92b4d968b673bf3288bd370386245d3272b27c85f034e28dacf27259f7522b"
Jul 27 18:57:05.711312 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.711280    2557 scope.go:117] "RemoveContainer" containerID="2250726a535acb6faa9b0d955638db33bdbbe5d4dd99cd4a916d8c860031ed56"
Jul 27 18:57:05.719886 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.719867    2557 scope.go:117] "RemoveContainer" containerID="bc92b4d968b673bf3288bd370386245d3272b27c85f034e28dacf27259f7522b"
Jul 27 18:57:05.720286 ip-10-0-131-114 kubenswrapper[2557]: E0727 18:57:05.720251    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc92b4d968b673bf3288bd370386245d3272b27c85f034e28dacf27259f7522b\": container with ID starting with bc92b4d968b673bf3288bd370386245d3272b27c85f034e28dacf27259f7522b not found: ID does not exist" containerID="bc92b4d968b673bf3288bd370386245d3272b27c85f034e28dacf27259f7522b"
Jul 27 18:57:05.720521 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.720285    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc92b4d968b673bf3288bd370386245d3272b27c85f034e28dacf27259f7522b"} err="failed to get container status \"bc92b4d968b673bf3288bd370386245d3272b27c85f034e28dacf27259f7522b\": rpc error: code = NotFound desc = could not find container \"bc92b4d968b673bf3288bd370386245d3272b27c85f034e28dacf27259f7522b\": container with ID starting with bc92b4d968b673bf3288bd370386245d3272b27c85f034e28dacf27259f7522b not found: ID does not exist"
Jul 27 18:57:05.720521 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.720311    2557 scope.go:117] "RemoveContainer" containerID="2250726a535acb6faa9b0d955638db33bdbbe5d4dd99cd4a916d8c860031ed56"
Jul 27 18:57:05.720692 ip-10-0-131-114 kubenswrapper[2557]: E0727 18:57:05.720648    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2250726a535acb6faa9b0d955638db33bdbbe5d4dd99cd4a916d8c860031ed56\": container with ID starting with 2250726a535acb6faa9b0d955638db33bdbbe5d4dd99cd4a916d8c860031ed56 not found: ID does not exist" containerID="2250726a535acb6faa9b0d955638db33bdbbe5d4dd99cd4a916d8c860031ed56"
Jul 27 18:57:05.720692 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.720679    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2250726a535acb6faa9b0d955638db33bdbbe5d4dd99cd4a916d8c860031ed56"} err="failed to get container status \"2250726a535acb6faa9b0d955638db33bdbbe5d4dd99cd4a916d8c860031ed56\": rpc error: code = NotFound desc = could not find container \"2250726a535acb6faa9b0d955638db33bdbbe5d4dd99cd4a916d8c860031ed56\": container with ID starting with 2250726a535acb6faa9b0d955638db33bdbbe5d4dd99cd4a916d8c860031ed56 not found: ID does not exist"
Jul 27 18:57:05.722339 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.722318    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"]
Jul 27 18:57:05.726092 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:05.726070    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d79c4c49-c89fn"]
Jul 27 18:57:06.120574 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:06.120491    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="772d0671-cc65-4489-b2d7-4ecb17336934" path="/var/lib/kubelet/pods/772d0671-cc65-4489-b2d7-4ecb17336934/volumes"
Jul 27 18:57:27.493597 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.493563    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"]
Jul 27 18:57:27.494119 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.493947    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="772d0671-cc65-4489-b2d7-4ecb17336934" containerName="main"
Jul 27 18:57:27.494119 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.493959    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="772d0671-cc65-4489-b2d7-4ecb17336934" containerName="main"
Jul 27 18:57:27.494119 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.493968    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="772d0671-cc65-4489-b2d7-4ecb17336934" containerName="storage-initializer"
Jul 27 18:57:27.494119 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.493975    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="772d0671-cc65-4489-b2d7-4ecb17336934" containerName="storage-initializer"
Jul 27 18:57:27.494119 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.494035    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="772d0671-cc65-4489-b2d7-4ecb17336934" containerName="main"
Jul 27 18:57:27.497417 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.497385    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"
Jul 27 18:57:27.499710 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.499685    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Jul 27 18:57:27.499918 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.499696    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Jul 27 18:57:27.499918 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.499847    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-xct24\""
Jul 27 18:57:27.500113 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.500098    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 18:57:27.500201 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.500185    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Jul 27 18:57:27.500261 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.500217    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Jul 27 18:57:27.509918 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.509876    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"]
Jul 27 18:57:27.675261 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.675221    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eb51ac8-f025-47b3-8209-95a514f0ac7d-tmp-dir\") pod \"lora-single-adapter-test-kserve-9f49df58-9vnrk\" (UID: \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"
Jul 27 18:57:27.675261 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.675271    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eb51ac8-f025-47b3-8209-95a514f0ac7d-tls-certs\") pod \"lora-single-adapter-test-kserve-9f49df58-9vnrk\" (UID: \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"
Jul 27 18:57:27.675502 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.675292    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eb51ac8-f025-47b3-8209-95a514f0ac7d-dshm\") pod \"lora-single-adapter-test-kserve-9f49df58-9vnrk\" (UID: \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"
Jul 27 18:57:27.675502 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.675360    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9sc8\" (UniqueName: \"kubernetes.io/projected/2eb51ac8-f025-47b3-8209-95a514f0ac7d-kube-api-access-k9sc8\") pod \"lora-single-adapter-test-kserve-9f49df58-9vnrk\" (UID: \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"
Jul 27 18:57:27.675502 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.675437    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eb51ac8-f025-47b3-8209-95a514f0ac7d-model-cache\") pod \"lora-single-adapter-test-kserve-9f49df58-9vnrk\" (UID: \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"
Jul 27 18:57:27.675502 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.675462    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2eb51ac8-f025-47b3-8209-95a514f0ac7d-cabundle-cert\") pod \"lora-single-adapter-test-kserve-9f49df58-9vnrk\" (UID: \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"
Jul 27 18:57:27.675635 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.675514    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2eb51ac8-f025-47b3-8209-95a514f0ac7d-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-9f49df58-9vnrk\" (UID: \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"
Jul 27 18:57:27.675635 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.675561    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eb51ac8-f025-47b3-8209-95a514f0ac7d-home\") pod \"lora-single-adapter-test-kserve-9f49df58-9vnrk\" (UID: \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"
Jul 27 18:57:27.776344 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.776251    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9sc8\" (UniqueName: \"kubernetes.io/projected/2eb51ac8-f025-47b3-8209-95a514f0ac7d-kube-api-access-k9sc8\") pod \"lora-single-adapter-test-kserve-9f49df58-9vnrk\" (UID: \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"
Jul 27 18:57:27.776344 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.776306    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eb51ac8-f025-47b3-8209-95a514f0ac7d-model-cache\") pod \"lora-single-adapter-test-kserve-9f49df58-9vnrk\" (UID: \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"
Jul 27 18:57:27.776344 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.776333    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2eb51ac8-f025-47b3-8209-95a514f0ac7d-cabundle-cert\") pod \"lora-single-adapter-test-kserve-9f49df58-9vnrk\" (UID: \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"
Jul 27 18:57:27.776639 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.776376    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2eb51ac8-f025-47b3-8209-95a514f0ac7d-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-9f49df58-9vnrk\" (UID: \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"
Jul 27 18:57:27.776639 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.776431    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eb51ac8-f025-47b3-8209-95a514f0ac7d-home\") pod \"lora-single-adapter-test-kserve-9f49df58-9vnrk\" (UID: \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"
Jul 27 18:57:27.776639 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.776485    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eb51ac8-f025-47b3-8209-95a514f0ac7d-tmp-dir\") pod \"lora-single-adapter-test-kserve-9f49df58-9vnrk\" (UID: \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"
Jul 27 18:57:27.776639 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.776518    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eb51ac8-f025-47b3-8209-95a514f0ac7d-tls-certs\") pod \"lora-single-adapter-test-kserve-9f49df58-9vnrk\" (UID: \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"
Jul 27 18:57:27.776639 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.776545    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eb51ac8-f025-47b3-8209-95a514f0ac7d-dshm\") pod \"lora-single-adapter-test-kserve-9f49df58-9vnrk\" (UID: \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"
Jul 27 18:57:27.776913 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.776809    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eb51ac8-f025-47b3-8209-95a514f0ac7d-model-cache\") pod \"lora-single-adapter-test-kserve-9f49df58-9vnrk\" (UID: \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"
Jul 27 18:57:27.776913 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.776844    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2eb51ac8-f025-47b3-8209-95a514f0ac7d-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-9f49df58-9vnrk\" (UID: \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"
Jul 27 18:57:27.777021 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.776962    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eb51ac8-f025-47b3-8209-95a514f0ac7d-home\") pod \"lora-single-adapter-test-kserve-9f49df58-9vnrk\" (UID: \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"
Jul 27 18:57:27.777021 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.776963    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eb51ac8-f025-47b3-8209-95a514f0ac7d-tmp-dir\") pod \"lora-single-adapter-test-kserve-9f49df58-9vnrk\" (UID: \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"
Jul 27 18:57:27.777177 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.777156    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2eb51ac8-f025-47b3-8209-95a514f0ac7d-cabundle-cert\") pod \"lora-single-adapter-test-kserve-9f49df58-9vnrk\" (UID: \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"
Jul 27 18:57:27.779040 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.779011    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eb51ac8-f025-47b3-8209-95a514f0ac7d-dshm\") pod \"lora-single-adapter-test-kserve-9f49df58-9vnrk\" (UID: \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"
Jul 27 18:57:27.779211 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.779196    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eb51ac8-f025-47b3-8209-95a514f0ac7d-tls-certs\") pod \"lora-single-adapter-test-kserve-9f49df58-9vnrk\" (UID: \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"
Jul 27 18:57:27.783639 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.783618    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9sc8\" (UniqueName: \"kubernetes.io/projected/2eb51ac8-f025-47b3-8209-95a514f0ac7d-kube-api-access-k9sc8\") pod \"lora-single-adapter-test-kserve-9f49df58-9vnrk\" (UID: \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"
Jul 27 18:57:27.811893 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.811871    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"
Jul 27 18:57:27.953370 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.953340    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"]
Jul 27 18:57:27.957214 ip-10-0-131-114 kubenswrapper[2557]: W0727 18:57:27.957187    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb51ac8_f025_47b3_8209_95a514f0ac7d.slice/crio-4a78ee3d4264ad9e1a58890711dbba3af224f4b12ba196ea44f1a439b917b4ef WatchSource:0}: Error finding container 4a78ee3d4264ad9e1a58890711dbba3af224f4b12ba196ea44f1a439b917b4ef: Status 404 returned error can't find the container with id 4a78ee3d4264ad9e1a58890711dbba3af224f4b12ba196ea44f1a439b917b4ef
Jul 27 18:57:27.961076 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:27.961051    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Jul 27 18:57:28.814561 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:28.814525    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk" event={"ID":"2eb51ac8-f025-47b3-8209-95a514f0ac7d","Type":"ContainerStarted","Data":"947a25aebefa139f44e5955cc3018ac3bd27233ca355b3635ebb3dbc662439a9"}
Jul 27 18:57:28.815029 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:28.814586    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk" event={"ID":"2eb51ac8-f025-47b3-8209-95a514f0ac7d","Type":"ContainerStarted","Data":"4a78ee3d4264ad9e1a58890711dbba3af224f4b12ba196ea44f1a439b917b4ef"}
Jul 27 18:57:40.877931 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:40.877895    2557 generic.go:356] "Generic (PLEG): container finished" podID="2eb51ac8-f025-47b3-8209-95a514f0ac7d" containerID="947a25aebefa139f44e5955cc3018ac3bd27233ca355b3635ebb3dbc662439a9" exitCode=0
Jul 27 18:57:40.878527 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:40.877972    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk" event={"ID":"2eb51ac8-f025-47b3-8209-95a514f0ac7d","Type":"ContainerDied","Data":"947a25aebefa139f44e5955cc3018ac3bd27233ca355b3635ebb3dbc662439a9"}
Jul 27 18:57:41.884292 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:41.884251    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk" event={"ID":"2eb51ac8-f025-47b3-8209-95a514f0ac7d","Type":"ContainerStarted","Data":"40f4419af1df3d6bcddf6d80336b0c0c8e8a4b3c0e37a02661c180df7315469e"}
Jul 27 18:57:41.906491 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:41.906422    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk" podStartSLOduration=3.046621118 podStartE2EDuration="14.906402995s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.859781877s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:57:41.903895001 +0000 UTC m=+5354.390706288" watchObservedRunningTime="2026-07-27 18:57:41.906402995 +0000 UTC m=+5354.393214283"
Jul 27 18:57:42.812209 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:42.812176    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"
Jul 27 18:57:47.812545 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:47.812494    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"
Jul 27 18:57:47.814171 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:47.814137    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk" podUID="2eb51ac8-f025-47b3-8209-95a514f0ac7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Jul 27 18:57:57.812656 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:57:57.812615    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk" podUID="2eb51ac8-f025-47b3-8209-95a514f0ac7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Jul 27 18:58:07.812857 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:58:07.812814    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk" podUID="2eb51ac8-f025-47b3-8209-95a514f0ac7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Jul 27 18:58:17.812734 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:58:17.812693    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk" podUID="2eb51ac8-f025-47b3-8209-95a514f0ac7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Jul 27 18:58:27.813233 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:58:27.813188    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk" podUID="2eb51ac8-f025-47b3-8209-95a514f0ac7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Jul 27 18:58:37.812939 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:58:37.812883    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk" podUID="2eb51ac8-f025-47b3-8209-95a514f0ac7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Jul 27 18:58:47.812603 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:58:47.812547    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk" podUID="2eb51ac8-f025-47b3-8209-95a514f0ac7d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Jul 27 18:58:57.824275 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:58:57.824241    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"
Jul 27 18:58:57.832205 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:58:57.832176    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"
Jul 27 18:59:35.501684 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:35.501627    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-9f49df58-9vnrk_2eb51ac8-f025-47b3-8209-95a514f0ac7d/storage-initializer/0.log"
Jul 27 18:59:35.517311 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:35.517277    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-9f49df58-9vnrk_2eb51ac8-f025-47b3-8209-95a514f0ac7d/main/0.log"
Jul 27 18:59:41.086191 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:41.086157    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"]
Jul 27 18:59:56.109594 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:56.109499    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk" podUID="2eb51ac8-f025-47b3-8209-95a514f0ac7d" containerName="main" containerID="cri-o://40f4419af1df3d6bcddf6d80336b0c0c8e8a4b3c0e37a02661c180df7315469e" gracePeriod=45
Jul 27 18:59:56.763116 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:56.763093    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"
Jul 27 18:59:56.858223 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:56.858138    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eb51ac8-f025-47b3-8209-95a514f0ac7d-tls-certs\") pod \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\" (UID: \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\") "
Jul 27 18:59:56.858223 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:56.858199    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eb51ac8-f025-47b3-8209-95a514f0ac7d-home\") pod \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\" (UID: \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\") "
Jul 27 18:59:56.858474 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:56.858298    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eb51ac8-f025-47b3-8209-95a514f0ac7d-model-cache\") pod \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\" (UID: \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\") "
Jul 27 18:59:56.858474 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:56.858334    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2eb51ac8-f025-47b3-8209-95a514f0ac7d-cabundle-cert\") pod \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\" (UID: \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\") "
Jul 27 18:59:56.858474 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:56.858377    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eb51ac8-f025-47b3-8209-95a514f0ac7d-tmp-dir\") pod \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\" (UID: \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\") "
Jul 27 18:59:56.858474 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:56.858402    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eb51ac8-f025-47b3-8209-95a514f0ac7d-dshm\") pod \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\" (UID: \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\") "
Jul 27 18:59:56.858474 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:56.858437    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2eb51ac8-f025-47b3-8209-95a514f0ac7d-kserve-provision-location\") pod \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\" (UID: \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\") "
Jul 27 18:59:56.859016 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:56.858510    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9sc8\" (UniqueName: \"kubernetes.io/projected/2eb51ac8-f025-47b3-8209-95a514f0ac7d-kube-api-access-k9sc8\") pod \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\" (UID: \"2eb51ac8-f025-47b3-8209-95a514f0ac7d\") "
Jul 27 18:59:56.859016 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:56.858577    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eb51ac8-f025-47b3-8209-95a514f0ac7d-model-cache" (OuterVolumeSpecName: "model-cache") pod "2eb51ac8-f025-47b3-8209-95a514f0ac7d" (UID: "2eb51ac8-f025-47b3-8209-95a514f0ac7d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:59:56.859016 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:56.858592    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eb51ac8-f025-47b3-8209-95a514f0ac7d-home" (OuterVolumeSpecName: "home") pod "2eb51ac8-f025-47b3-8209-95a514f0ac7d" (UID: "2eb51ac8-f025-47b3-8209-95a514f0ac7d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:59:56.859016 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:56.858848    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eb51ac8-f025-47b3-8209-95a514f0ac7d-home\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:59:56.859016 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:56.858866    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eb51ac8-f025-47b3-8209-95a514f0ac7d-model-cache\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:59:56.859016 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:56.858957    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eb51ac8-f025-47b3-8209-95a514f0ac7d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2eb51ac8-f025-47b3-8209-95a514f0ac7d" (UID: "2eb51ac8-f025-47b3-8209-95a514f0ac7d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:59:56.861071 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:56.861035    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eb51ac8-f025-47b3-8209-95a514f0ac7d-kube-api-access-k9sc8" (OuterVolumeSpecName: "kube-api-access-k9sc8") pod "2eb51ac8-f025-47b3-8209-95a514f0ac7d" (UID: "2eb51ac8-f025-47b3-8209-95a514f0ac7d"). InnerVolumeSpecName "kube-api-access-k9sc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:59:56.861185 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:56.861148    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eb51ac8-f025-47b3-8209-95a514f0ac7d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2eb51ac8-f025-47b3-8209-95a514f0ac7d" (UID: "2eb51ac8-f025-47b3-8209-95a514f0ac7d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:59:56.861371 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:56.861346    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eb51ac8-f025-47b3-8209-95a514f0ac7d-dshm" (OuterVolumeSpecName: "dshm") pod "2eb51ac8-f025-47b3-8209-95a514f0ac7d" (UID: "2eb51ac8-f025-47b3-8209-95a514f0ac7d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:59:56.870944 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:56.870922    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eb51ac8-f025-47b3-8209-95a514f0ac7d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2eb51ac8-f025-47b3-8209-95a514f0ac7d" (UID: "2eb51ac8-f025-47b3-8209-95a514f0ac7d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:59:56.935767 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:56.935716    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eb51ac8-f025-47b3-8209-95a514f0ac7d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2eb51ac8-f025-47b3-8209-95a514f0ac7d" (UID: "2eb51ac8-f025-47b3-8209-95a514f0ac7d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:59:56.959478 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:56.959447    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eb51ac8-f025-47b3-8209-95a514f0ac7d-tls-certs\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:59:56.959478 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:56.959471    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2eb51ac8-f025-47b3-8209-95a514f0ac7d-cabundle-cert\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:59:56.959478 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:56.959480    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eb51ac8-f025-47b3-8209-95a514f0ac7d-tmp-dir\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:59:56.959692 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:56.959488    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eb51ac8-f025-47b3-8209-95a514f0ac7d-dshm\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:59:56.959692 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:56.959497    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2eb51ac8-f025-47b3-8209-95a514f0ac7d-kserve-provision-location\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:59:56.959692 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:56.959507    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9sc8\" (UniqueName: \"kubernetes.io/projected/2eb51ac8-f025-47b3-8209-95a514f0ac7d-kube-api-access-k9sc8\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 18:59:57.592106 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:57.592063    2557 generic.go:356] "Generic (PLEG): container finished" podID="2eb51ac8-f025-47b3-8209-95a514f0ac7d" containerID="40f4419af1df3d6bcddf6d80336b0c0c8e8a4b3c0e37a02661c180df7315469e" exitCode=0
Jul 27 18:59:57.592607 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:57.592123    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"
Jul 27 18:59:57.592607 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:57.592129    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk" event={"ID":"2eb51ac8-f025-47b3-8209-95a514f0ac7d","Type":"ContainerDied","Data":"40f4419af1df3d6bcddf6d80336b0c0c8e8a4b3c0e37a02661c180df7315469e"}
Jul 27 18:59:57.592607 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:57.592183    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk" event={"ID":"2eb51ac8-f025-47b3-8209-95a514f0ac7d","Type":"ContainerDied","Data":"4a78ee3d4264ad9e1a58890711dbba3af224f4b12ba196ea44f1a439b917b4ef"}
Jul 27 18:59:57.592607 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:57.592206    2557 scope.go:117] "RemoveContainer" containerID="40f4419af1df3d6bcddf6d80336b0c0c8e8a4b3c0e37a02661c180df7315469e"
Jul 27 18:59:57.601882 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:57.601863    2557 scope.go:117] "RemoveContainer" containerID="947a25aebefa139f44e5955cc3018ac3bd27233ca355b3635ebb3dbc662439a9"
Jul 27 18:59:57.611688 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:57.611671    2557 scope.go:117] "RemoveContainer" containerID="40f4419af1df3d6bcddf6d80336b0c0c8e8a4b3c0e37a02661c180df7315469e"
Jul 27 18:59:57.612022 ip-10-0-131-114 kubenswrapper[2557]: E0727 18:59:57.612000    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40f4419af1df3d6bcddf6d80336b0c0c8e8a4b3c0e37a02661c180df7315469e\": container with ID starting with 40f4419af1df3d6bcddf6d80336b0c0c8e8a4b3c0e37a02661c180df7315469e not found: ID does not exist" containerID="40f4419af1df3d6bcddf6d80336b0c0c8e8a4b3c0e37a02661c180df7315469e"
Jul 27 18:59:57.612090 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:57.612031    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40f4419af1df3d6bcddf6d80336b0c0c8e8a4b3c0e37a02661c180df7315469e"} err="failed to get container status \"40f4419af1df3d6bcddf6d80336b0c0c8e8a4b3c0e37a02661c180df7315469e\": rpc error: code = NotFound desc = could not find container \"40f4419af1df3d6bcddf6d80336b0c0c8e8a4b3c0e37a02661c180df7315469e\": container with ID starting with 40f4419af1df3d6bcddf6d80336b0c0c8e8a4b3c0e37a02661c180df7315469e not found: ID does not exist"
Jul 27 18:59:57.612090 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:57.612048    2557 scope.go:117] "RemoveContainer" containerID="947a25aebefa139f44e5955cc3018ac3bd27233ca355b3635ebb3dbc662439a9"
Jul 27 18:59:57.612324 ip-10-0-131-114 kubenswrapper[2557]: E0727 18:59:57.612303    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"947a25aebefa139f44e5955cc3018ac3bd27233ca355b3635ebb3dbc662439a9\": container with ID starting with 947a25aebefa139f44e5955cc3018ac3bd27233ca355b3635ebb3dbc662439a9 not found: ID does not exist" containerID="947a25aebefa139f44e5955cc3018ac3bd27233ca355b3635ebb3dbc662439a9"
Jul 27 18:59:57.612389 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:57.612330    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"947a25aebefa139f44e5955cc3018ac3bd27233ca355b3635ebb3dbc662439a9"} err="failed to get container status \"947a25aebefa139f44e5955cc3018ac3bd27233ca355b3635ebb3dbc662439a9\": rpc error: code = NotFound desc = could not find container \"947a25aebefa139f44e5955cc3018ac3bd27233ca355b3635ebb3dbc662439a9\": container with ID starting with 947a25aebefa139f44e5955cc3018ac3bd27233ca355b3635ebb3dbc662439a9 not found: ID does not exist"
Jul 27 18:59:57.617975 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:57.617949    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"]
Jul 27 18:59:57.621314 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:57.621281    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9f49df58-9vnrk"]
Jul 27 18:59:58.118745 ip-10-0-131-114 kubenswrapper[2557]: I0727 18:59:58.118711    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2eb51ac8-f025-47b3-8209-95a514f0ac7d" path="/var/lib/kubelet/pods/2eb51ac8-f025-47b3-8209-95a514f0ac7d/volumes"
Jul 27 19:08:46.497899 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.497864    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk"]
Jul 27 19:08:46.500846 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.498243    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eb51ac8-f025-47b3-8209-95a514f0ac7d" containerName="main"
Jul 27 19:08:46.500846 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.498255    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eb51ac8-f025-47b3-8209-95a514f0ac7d" containerName="main"
Jul 27 19:08:46.500846 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.498275    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eb51ac8-f025-47b3-8209-95a514f0ac7d" containerName="storage-initializer"
Jul 27 19:08:46.500846 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.498282    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eb51ac8-f025-47b3-8209-95a514f0ac7d" containerName="storage-initializer"
Jul 27 19:08:46.500846 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.498354    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eb51ac8-f025-47b3-8209-95a514f0ac7d" containerName="main"
Jul 27 19:08:46.501487 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.501469    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk"
Jul 27 19:08:46.503665 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.503636    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Jul 27 19:08:46.503811 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.503685    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Jul 27 19:08:46.503811 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.503706    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-6685r\""
Jul 27 19:08:46.503926 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.503809    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 27 19:08:46.503926 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.503905    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Jul 27 19:08:46.512673 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.512648    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk"]
Jul 27 19:08:46.601349 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.601316    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c465fea4-73ab-4035-8556-0294e7e6f0c8-home\") pod \"prestop-hook-test-kserve-54c766cd44-r74jk\" (UID: \"c465fea4-73ab-4035-8556-0294e7e6f0c8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk"
Jul 27 19:08:46.601349 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.601352    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c465fea4-73ab-4035-8556-0294e7e6f0c8-dshm\") pod \"prestop-hook-test-kserve-54c766cd44-r74jk\" (UID: \"c465fea4-73ab-4035-8556-0294e7e6f0c8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk"
Jul 27 19:08:46.601598 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.601381    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c465fea4-73ab-4035-8556-0294e7e6f0c8-tls-certs\") pod \"prestop-hook-test-kserve-54c766cd44-r74jk\" (UID: \"c465fea4-73ab-4035-8556-0294e7e6f0c8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk"
Jul 27 19:08:46.601598 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.601446    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w59f\" (UniqueName: \"kubernetes.io/projected/c465fea4-73ab-4035-8556-0294e7e6f0c8-kube-api-access-9w59f\") pod \"prestop-hook-test-kserve-54c766cd44-r74jk\" (UID: \"c465fea4-73ab-4035-8556-0294e7e6f0c8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk"
Jul 27 19:08:46.601598 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.601530    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c465fea4-73ab-4035-8556-0294e7e6f0c8-tmp-dir\") pod \"prestop-hook-test-kserve-54c766cd44-r74jk\" (UID: \"c465fea4-73ab-4035-8556-0294e7e6f0c8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk"
Jul 27 19:08:46.601598 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.601576    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c465fea4-73ab-4035-8556-0294e7e6f0c8-kserve-provision-location\") pod \"prestop-hook-test-kserve-54c766cd44-r74jk\" (UID: \"c465fea4-73ab-4035-8556-0294e7e6f0c8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk"
Jul 27 19:08:46.601782 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.601607    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c465fea4-73ab-4035-8556-0294e7e6f0c8-model-cache\") pod \"prestop-hook-test-kserve-54c766cd44-r74jk\" (UID: \"c465fea4-73ab-4035-8556-0294e7e6f0c8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk"
Jul 27 19:08:46.702772 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.702725    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c465fea4-73ab-4035-8556-0294e7e6f0c8-home\") pod \"prestop-hook-test-kserve-54c766cd44-r74jk\" (UID: \"c465fea4-73ab-4035-8556-0294e7e6f0c8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk"
Jul 27 19:08:46.702947 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.702789    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c465fea4-73ab-4035-8556-0294e7e6f0c8-dshm\") pod \"prestop-hook-test-kserve-54c766cd44-r74jk\" (UID: \"c465fea4-73ab-4035-8556-0294e7e6f0c8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk"
Jul 27 19:08:46.702947 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.702833    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c465fea4-73ab-4035-8556-0294e7e6f0c8-tls-certs\") pod \"prestop-hook-test-kserve-54c766cd44-r74jk\" (UID: \"c465fea4-73ab-4035-8556-0294e7e6f0c8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk"
Jul 27 19:08:46.702947 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.702872    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w59f\" (UniqueName: \"kubernetes.io/projected/c465fea4-73ab-4035-8556-0294e7e6f0c8-kube-api-access-9w59f\") pod \"prestop-hook-test-kserve-54c766cd44-r74jk\" (UID: \"c465fea4-73ab-4035-8556-0294e7e6f0c8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk"
Jul 27 19:08:46.702947 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.702932    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c465fea4-73ab-4035-8556-0294e7e6f0c8-tmp-dir\") pod \"prestop-hook-test-kserve-54c766cd44-r74jk\" (UID: \"c465fea4-73ab-4035-8556-0294e7e6f0c8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk"
Jul 27 19:08:46.703171 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.702954    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c465fea4-73ab-4035-8556-0294e7e6f0c8-kserve-provision-location\") pod \"prestop-hook-test-kserve-54c766cd44-r74jk\" (UID: \"c465fea4-73ab-4035-8556-0294e7e6f0c8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk"
Jul 27 19:08:46.703171 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.702992    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c465fea4-73ab-4035-8556-0294e7e6f0c8-model-cache\") pod \"prestop-hook-test-kserve-54c766cd44-r74jk\" (UID: \"c465fea4-73ab-4035-8556-0294e7e6f0c8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk"
Jul 27 19:08:46.703276 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.703209    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c465fea4-73ab-4035-8556-0294e7e6f0c8-home\") pod \"prestop-hook-test-kserve-54c766cd44-r74jk\" (UID: \"c465fea4-73ab-4035-8556-0294e7e6f0c8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk"
Jul 27 19:08:46.703344 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.703318    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c465fea4-73ab-4035-8556-0294e7e6f0c8-kserve-provision-location\") pod \"prestop-hook-test-kserve-54c766cd44-r74jk\" (UID: \"c465fea4-73ab-4035-8556-0294e7e6f0c8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk"
Jul 27 19:08:46.703387 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.703344    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c465fea4-73ab-4035-8556-0294e7e6f0c8-tmp-dir\") pod \"prestop-hook-test-kserve-54c766cd44-r74jk\" (UID: \"c465fea4-73ab-4035-8556-0294e7e6f0c8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk"
Jul 27 19:08:46.703422 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.703401    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c465fea4-73ab-4035-8556-0294e7e6f0c8-model-cache\") pod \"prestop-hook-test-kserve-54c766cd44-r74jk\" (UID: \"c465fea4-73ab-4035-8556-0294e7e6f0c8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk"
Jul 27 19:08:46.705244 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.705217    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c465fea4-73ab-4035-8556-0294e7e6f0c8-dshm\") pod \"prestop-hook-test-kserve-54c766cd44-r74jk\" (UID: \"c465fea4-73ab-4035-8556-0294e7e6f0c8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk"
Jul 27 19:08:46.705472 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.705457    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c465fea4-73ab-4035-8556-0294e7e6f0c8-tls-certs\") pod \"prestop-hook-test-kserve-54c766cd44-r74jk\" (UID: \"c465fea4-73ab-4035-8556-0294e7e6f0c8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk"
Jul 27 19:08:46.713016 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.712970    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w59f\" (UniqueName: \"kubernetes.io/projected/c465fea4-73ab-4035-8556-0294e7e6f0c8-kube-api-access-9w59f\") pod \"prestop-hook-test-kserve-54c766cd44-r74jk\" (UID: \"c465fea4-73ab-4035-8556-0294e7e6f0c8\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk"
Jul 27 19:08:46.812650 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.812559    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk"
Jul 27 19:08:46.948779 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:46.948729    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk"]
Jul 27 19:08:46.950855 ip-10-0-131-114 kubenswrapper[2557]: W0727 19:08:46.950823    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc465fea4_73ab_4035_8556_0294e7e6f0c8.slice/crio-b5f7836e4b8a59406fbb4c927c95a0bc69da92b4d76d6bcbafd2b950a6c93dd9 WatchSource:0}: Error finding container b5f7836e4b8a59406fbb4c927c95a0bc69da92b4d76d6bcbafd2b950a6c93dd9: Status 404 returned error can't find the container with id b5f7836e4b8a59406fbb4c927c95a0bc69da92b4d76d6bcbafd2b950a6c93dd9
Jul 27 19:08:47.093277 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:47.093169    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk" event={"ID":"c465fea4-73ab-4035-8556-0294e7e6f0c8","Type":"ContainerStarted","Data":"0fe5f0b540b3a9e4688d7f12590a06899dca6f3f4c768efb4448579519a6a283"}
Jul 27 19:08:47.093277 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:08:47.093229    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk" event={"ID":"c465fea4-73ab-4035-8556-0294e7e6f0c8","Type":"ContainerStarted","Data":"b5f7836e4b8a59406fbb4c927c95a0bc69da92b4d76d6bcbafd2b950a6c93dd9"}
Jul 27 19:09:12.802830 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:09:12.802780    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk"]
Jul 27 19:09:12.803374 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:09:12.803085    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk" podUID="c465fea4-73ab-4035-8556-0294e7e6f0c8" containerName="storage-initializer" containerID="cri-o://0fe5f0b540b3a9e4688d7f12590a06899dca6f3f4c768efb4448579519a6a283" gracePeriod=60
Jul 27 19:10:12.989988 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:12.989961    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-54c766cd44-r74jk_c465fea4-73ab-4035-8556-0294e7e6f0c8/storage-initializer/0.log"
Jul 27 19:10:12.991822 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:12.991801    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk"
Jul 27 19:10:13.087831 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:13.087726    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c465fea4-73ab-4035-8556-0294e7e6f0c8-model-cache\") pod \"c465fea4-73ab-4035-8556-0294e7e6f0c8\" (UID: \"c465fea4-73ab-4035-8556-0294e7e6f0c8\") "
Jul 27 19:10:13.087831 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:13.087806    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c465fea4-73ab-4035-8556-0294e7e6f0c8-dshm\") pod \"c465fea4-73ab-4035-8556-0294e7e6f0c8\" (UID: \"c465fea4-73ab-4035-8556-0294e7e6f0c8\") "
Jul 27 19:10:13.088058 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:13.087855    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9w59f\" (UniqueName: \"kubernetes.io/projected/c465fea4-73ab-4035-8556-0294e7e6f0c8-kube-api-access-9w59f\") pod \"c465fea4-73ab-4035-8556-0294e7e6f0c8\" (UID: \"c465fea4-73ab-4035-8556-0294e7e6f0c8\") "
Jul 27 19:10:13.088058 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:13.087877    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c465fea4-73ab-4035-8556-0294e7e6f0c8-kserve-provision-location\") pod \"c465fea4-73ab-4035-8556-0294e7e6f0c8\" (UID: \"c465fea4-73ab-4035-8556-0294e7e6f0c8\") "
Jul 27 19:10:13.088058 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:13.087936    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c465fea4-73ab-4035-8556-0294e7e6f0c8-home\") pod \"c465fea4-73ab-4035-8556-0294e7e6f0c8\" (UID: \"c465fea4-73ab-4035-8556-0294e7e6f0c8\") "
Jul 27 19:10:13.088058 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:13.087962    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c465fea4-73ab-4035-8556-0294e7e6f0c8-tmp-dir\") pod \"c465fea4-73ab-4035-8556-0294e7e6f0c8\" (UID: \"c465fea4-73ab-4035-8556-0294e7e6f0c8\") "
Jul 27 19:10:13.088058 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:13.088002    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c465fea4-73ab-4035-8556-0294e7e6f0c8-tls-certs\") pod \"c465fea4-73ab-4035-8556-0294e7e6f0c8\" (UID: \"c465fea4-73ab-4035-8556-0294e7e6f0c8\") "
Jul 27 19:10:13.088058 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:13.088031    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c465fea4-73ab-4035-8556-0294e7e6f0c8-model-cache" (OuterVolumeSpecName: "model-cache") pod "c465fea4-73ab-4035-8556-0294e7e6f0c8" (UID: "c465fea4-73ab-4035-8556-0294e7e6f0c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:10:13.088380 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:13.088223    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c465fea4-73ab-4035-8556-0294e7e6f0c8-home" (OuterVolumeSpecName: "home") pod "c465fea4-73ab-4035-8556-0294e7e6f0c8" (UID: "c465fea4-73ab-4035-8556-0294e7e6f0c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:10:13.088380 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:13.088246    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c465fea4-73ab-4035-8556-0294e7e6f0c8-model-cache\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 19:10:13.088380 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:13.088262    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c465fea4-73ab-4035-8556-0294e7e6f0c8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c465fea4-73ab-4035-8556-0294e7e6f0c8" (UID: "c465fea4-73ab-4035-8556-0294e7e6f0c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:10:13.090174 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:13.090147    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c465fea4-73ab-4035-8556-0294e7e6f0c8-dshm" (OuterVolumeSpecName: "dshm") pod "c465fea4-73ab-4035-8556-0294e7e6f0c8" (UID: "c465fea4-73ab-4035-8556-0294e7e6f0c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:10:13.090293 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:13.090221    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c465fea4-73ab-4035-8556-0294e7e6f0c8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c465fea4-73ab-4035-8556-0294e7e6f0c8" (UID: "c465fea4-73ab-4035-8556-0294e7e6f0c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:10:13.090293 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:13.090235    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c465fea4-73ab-4035-8556-0294e7e6f0c8-kube-api-access-9w59f" (OuterVolumeSpecName: "kube-api-access-9w59f") pod "c465fea4-73ab-4035-8556-0294e7e6f0c8" (UID: "c465fea4-73ab-4035-8556-0294e7e6f0c8"). InnerVolumeSpecName "kube-api-access-9w59f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:10:13.148258 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:13.148195    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c465fea4-73ab-4035-8556-0294e7e6f0c8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c465fea4-73ab-4035-8556-0294e7e6f0c8" (UID: "c465fea4-73ab-4035-8556-0294e7e6f0c8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:10:13.189346 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:13.189302    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c465fea4-73ab-4035-8556-0294e7e6f0c8-home\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 19:10:13.189346 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:13.189337    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c465fea4-73ab-4035-8556-0294e7e6f0c8-tmp-dir\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 19:10:13.189346 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:13.189351    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c465fea4-73ab-4035-8556-0294e7e6f0c8-tls-certs\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 19:10:13.189601 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:13.189364    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c465fea4-73ab-4035-8556-0294e7e6f0c8-dshm\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 19:10:13.189601 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:13.189376    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w59f\" (UniqueName: \"kubernetes.io/projected/c465fea4-73ab-4035-8556-0294e7e6f0c8-kube-api-access-9w59f\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 19:10:13.189601 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:13.189388    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c465fea4-73ab-4035-8556-0294e7e6f0c8-kserve-provision-location\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 19:10:13.523275 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:13.523245    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-54c766cd44-r74jk_c465fea4-73ab-4035-8556-0294e7e6f0c8/storage-initializer/0.log"
Jul 27 19:10:13.523489 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:13.523285    2557 generic.go:356] "Generic (PLEG): container finished" podID="c465fea4-73ab-4035-8556-0294e7e6f0c8" containerID="0fe5f0b540b3a9e4688d7f12590a06899dca6f3f4c768efb4448579519a6a283" exitCode=137
Jul 27 19:10:13.523489 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:13.523316    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk" event={"ID":"c465fea4-73ab-4035-8556-0294e7e6f0c8","Type":"ContainerDied","Data":"0fe5f0b540b3a9e4688d7f12590a06899dca6f3f4c768efb4448579519a6a283"}
Jul 27 19:10:13.523489 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:13.523361    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk"
Jul 27 19:10:13.523489 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:13.523397    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk" event={"ID":"c465fea4-73ab-4035-8556-0294e7e6f0c8","Type":"ContainerDied","Data":"b5f7836e4b8a59406fbb4c927c95a0bc69da92b4d76d6bcbafd2b950a6c93dd9"}
Jul 27 19:10:13.523489 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:13.523421    2557 scope.go:117] "RemoveContainer" containerID="0fe5f0b540b3a9e4688d7f12590a06899dca6f3f4c768efb4448579519a6a283"
Jul 27 19:10:13.533840 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:13.533818    2557 scope.go:117] "RemoveContainer" containerID="0fe5f0b540b3a9e4688d7f12590a06899dca6f3f4c768efb4448579519a6a283"
Jul 27 19:10:13.534126 ip-10-0-131-114 kubenswrapper[2557]: E0727 19:10:13.534106    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fe5f0b540b3a9e4688d7f12590a06899dca6f3f4c768efb4448579519a6a283\": container with ID starting with 0fe5f0b540b3a9e4688d7f12590a06899dca6f3f4c768efb4448579519a6a283 not found: ID does not exist" containerID="0fe5f0b540b3a9e4688d7f12590a06899dca6f3f4c768efb4448579519a6a283"
Jul 27 19:10:13.534175 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:13.534134    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fe5f0b540b3a9e4688d7f12590a06899dca6f3f4c768efb4448579519a6a283"} err="failed to get container status \"0fe5f0b540b3a9e4688d7f12590a06899dca6f3f4c768efb4448579519a6a283\": rpc error: code = NotFound desc = could not find container \"0fe5f0b540b3a9e4688d7f12590a06899dca6f3f4c768efb4448579519a6a283\": container with ID starting with 0fe5f0b540b3a9e4688d7f12590a06899dca6f3f4c768efb4448579519a6a283 not found: ID does not exist"
Jul 27 19:10:13.560438 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:13.560408    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk"]
Jul 27 19:10:13.564205 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:13.564180    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-54c766cd44-r74jk"]
Jul 27 19:10:14.118787 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:10:14.118739    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c465fea4-73ab-4035-8556-0294e7e6f0c8" path="/var/lib/kubelet/pods/c465fea4-73ab-4035-8556-0294e7e6f0c8/volumes"
Jul 27 19:11:14.788564 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:14.788524    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx"]
Jul 27 19:11:14.789132 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:14.788890    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c465fea4-73ab-4035-8556-0294e7e6f0c8" containerName="storage-initializer"
Jul 27 19:11:14.789132 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:14.788902    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c465fea4-73ab-4035-8556-0294e7e6f0c8" containerName="storage-initializer"
Jul 27 19:11:14.789132 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:14.788980    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c465fea4-73ab-4035-8556-0294e7e6f0c8" containerName="storage-initializer"
Jul 27 19:11:14.792233 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:14.792216    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx"
Jul 27 19:11:14.794310 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:14.794287    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Jul 27 19:11:14.794963 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:14.794941    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 27 19:11:14.794963 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:14.794959    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-4mb28\""
Jul 27 19:11:14.795156 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:14.794943    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Jul 27 19:11:14.802477 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:14.802447    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx"]
Jul 27 19:11:14.819746 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:14.819701    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f3d5340-6181-48e3-9875-899c0cea2057-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-lrvvx\" (UID: \"0f3d5340-6181-48e3-9875-899c0cea2057\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx"
Jul 27 19:11:14.819957 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:14.819827    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f3d5340-6181-48e3-9875-899c0cea2057-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-lrvvx\" (UID: \"0f3d5340-6181-48e3-9875-899c0cea2057\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx"
Jul 27 19:11:14.819957 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:14.819904    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f3d5340-6181-48e3-9875-899c0cea2057-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-lrvvx\" (UID: \"0f3d5340-6181-48e3-9875-899c0cea2057\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx"
Jul 27 19:11:14.819957 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:14.819950    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f3d5340-6181-48e3-9875-899c0cea2057-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-lrvvx\" (UID: \"0f3d5340-6181-48e3-9875-899c0cea2057\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx"
Jul 27 19:11:14.820136 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:14.820004    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f3d5340-6181-48e3-9875-899c0cea2057-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-lrvvx\" (UID: \"0f3d5340-6181-48e3-9875-899c0cea2057\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx"
Jul 27 19:11:14.820136 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:14.820035    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkd5g\" (UniqueName: \"kubernetes.io/projected/0f3d5340-6181-48e3-9875-899c0cea2057-kube-api-access-bkd5g\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-lrvvx\" (UID: \"0f3d5340-6181-48e3-9875-899c0cea2057\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx"
Jul 27 19:11:14.921307 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:14.921264    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f3d5340-6181-48e3-9875-899c0cea2057-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-lrvvx\" (UID: \"0f3d5340-6181-48e3-9875-899c0cea2057\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx"
Jul 27 19:11:14.921503 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:14.921325    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f3d5340-6181-48e3-9875-899c0cea2057-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-lrvvx\" (UID: \"0f3d5340-6181-48e3-9875-899c0cea2057\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx"
Jul 27 19:11:14.921503 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:14.921345    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkd5g\" (UniqueName: \"kubernetes.io/projected/0f3d5340-6181-48e3-9875-899c0cea2057-kube-api-access-bkd5g\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-lrvvx\" (UID: \"0f3d5340-6181-48e3-9875-899c0cea2057\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx"
Jul 27 19:11:14.921503 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:14.921392    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f3d5340-6181-48e3-9875-899c0cea2057-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-lrvvx\" (UID: \"0f3d5340-6181-48e3-9875-899c0cea2057\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx"
Jul 27 19:11:14.921503 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:14.921423    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f3d5340-6181-48e3-9875-899c0cea2057-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-lrvvx\" (UID: \"0f3d5340-6181-48e3-9875-899c0cea2057\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx"
Jul 27 19:11:14.921503 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:14.921484    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f3d5340-6181-48e3-9875-899c0cea2057-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-lrvvx\" (UID: \"0f3d5340-6181-48e3-9875-899c0cea2057\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx"
Jul 27 19:11:14.921810 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:14.921785    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f3d5340-6181-48e3-9875-899c0cea2057-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-lrvvx\" (UID: \"0f3d5340-6181-48e3-9875-899c0cea2057\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx"
Jul 27 19:11:14.922001 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:14.921981    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f3d5340-6181-48e3-9875-899c0cea2057-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-lrvvx\" (UID: \"0f3d5340-6181-48e3-9875-899c0cea2057\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx"
Jul 27 19:11:14.922079 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:14.922036    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f3d5340-6181-48e3-9875-899c0cea2057-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-lrvvx\" (UID: \"0f3d5340-6181-48e3-9875-899c0cea2057\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx"
Jul 27 19:11:14.923727 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:14.923703    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f3d5340-6181-48e3-9875-899c0cea2057-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-lrvvx\" (UID: \"0f3d5340-6181-48e3-9875-899c0cea2057\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx"
Jul 27 19:11:14.924310 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:14.924291    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f3d5340-6181-48e3-9875-899c0cea2057-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-lrvvx\" (UID: \"0f3d5340-6181-48e3-9875-899c0cea2057\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx"
Jul 27 19:11:14.928901 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:14.928877    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkd5g\" (UniqueName: \"kubernetes.io/projected/0f3d5340-6181-48e3-9875-899c0cea2057-kube-api-access-bkd5g\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-lrvvx\" (UID: \"0f3d5340-6181-48e3-9875-899c0cea2057\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx"
Jul 27 19:11:15.105249 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:15.105159    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx"
Jul 27 19:11:15.240248 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:15.240220    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx"]
Jul 27 19:11:15.243041 ip-10-0-131-114 kubenswrapper[2557]: W0727 19:11:15.243009    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f3d5340_6181_48e3_9875_899c0cea2057.slice/crio-0e0d003f01436bcbc785b56a9bdc4212d7b141f14bd877f48fc6cdc3820f7dcf WatchSource:0}: Error finding container 0e0d003f01436bcbc785b56a9bdc4212d7b141f14bd877f48fc6cdc3820f7dcf: Status 404 returned error can't find the container with id 0e0d003f01436bcbc785b56a9bdc4212d7b141f14bd877f48fc6cdc3820f7dcf
Jul 27 19:11:15.830682 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:15.830644    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx" event={"ID":"0f3d5340-6181-48e3-9875-899c0cea2057","Type":"ContainerStarted","Data":"86a1c16c7c119877f9f9d03ee356e6013f93d19c739b7cee388ca67bd0b1876b"}
Jul 27 19:11:15.831096 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:15.830707    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx" event={"ID":"0f3d5340-6181-48e3-9875-899c0cea2057","Type":"ContainerStarted","Data":"0e0d003f01436bcbc785b56a9bdc4212d7b141f14bd877f48fc6cdc3820f7dcf"}
Jul 27 19:11:15.847923 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:15.847873    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx" podStartSLOduration=1.847853894 podStartE2EDuration="1.847853894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:11:15.845979118 +0000 UTC m=+6168.332790426" watchObservedRunningTime="2026-07-27 19:11:15.847853894 +0000 UTC m=+6168.334665181"
Jul 27 19:11:16.105548 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:16.105463    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx"
Jul 27 19:11:25.106002 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:25.105970    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx"
Jul 27 19:11:25.118297 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:25.118274    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx"
Jul 27 19:11:25.890675 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:25.890637    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx"
Jul 27 19:11:32.288854 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:32.288821    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx"]
Jul 27 19:11:32.297938 ip-10-0-131-114 kubenswrapper[2557]: E0727 19:11:32.297898    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 19:11:32.297938 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 19:11:32.297938 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 19:11:32.297938 ip-10-0-131-114 kubenswrapper[2557]:  > containerID="86a1c16c7c119877f9f9d03ee356e6013f93d19c739b7cee388ca67bd0b1876b" cmd=["/bin/sleep","15"]
Jul 27 19:11:32.298185 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:32.297952    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx" message="" err=<
Jul 27 19:11:32.298185 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 19:11:32.298185 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 19:11:32.298185 ip-10-0-131-114 kubenswrapper[2557]:  >
Jul 27 19:11:32.298185 ip-10-0-131-114 kubenswrapper[2557]: E0727 19:11:32.297965    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 19:11:32.298185 ip-10-0-131-114 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 19:11:32.298185 ip-10-0-131-114 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 27 19:11:32.298185 ip-10-0-131-114 kubenswrapper[2557]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx" podUID="0f3d5340-6181-48e3-9875-899c0cea2057" containerName="main" containerID="cri-o://86a1c16c7c119877f9f9d03ee356e6013f93d19c739b7cee388ca67bd0b1876b"
Jul 27 19:11:32.298185 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:32.298004    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx" podUID="0f3d5340-6181-48e3-9875-899c0cea2057" containerName="main" containerID="cri-o://86a1c16c7c119877f9f9d03ee356e6013f93d19c739b7cee388ca67bd0b1876b" gracePeriod=60
Jul 27 19:11:32.659628 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:32.659602    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx"
Jul 27 19:11:32.757957 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:32.757924    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f3d5340-6181-48e3-9875-899c0cea2057-tmp-dir\") pod \"0f3d5340-6181-48e3-9875-899c0cea2057\" (UID: \"0f3d5340-6181-48e3-9875-899c0cea2057\") "
Jul 27 19:11:32.758137 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:32.758003    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f3d5340-6181-48e3-9875-899c0cea2057-dshm\") pod \"0f3d5340-6181-48e3-9875-899c0cea2057\" (UID: \"0f3d5340-6181-48e3-9875-899c0cea2057\") "
Jul 27 19:11:32.758137 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:32.758030    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f3d5340-6181-48e3-9875-899c0cea2057-home\") pod \"0f3d5340-6181-48e3-9875-899c0cea2057\" (UID: \"0f3d5340-6181-48e3-9875-899c0cea2057\") "
Jul 27 19:11:32.758137 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:32.758053    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f3d5340-6181-48e3-9875-899c0cea2057-tls-certs\") pod \"0f3d5340-6181-48e3-9875-899c0cea2057\" (UID: \"0f3d5340-6181-48e3-9875-899c0cea2057\") "
Jul 27 19:11:32.758137 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:32.758087    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f3d5340-6181-48e3-9875-899c0cea2057-model-cache\") pod \"0f3d5340-6181-48e3-9875-899c0cea2057\" (UID: \"0f3d5340-6181-48e3-9875-899c0cea2057\") "
Jul 27 19:11:32.758347 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:32.758236    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3d5340-6181-48e3-9875-899c0cea2057-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0f3d5340-6181-48e3-9875-899c0cea2057" (UID: "0f3d5340-6181-48e3-9875-899c0cea2057"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:11:32.758347 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:32.758276    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3d5340-6181-48e3-9875-899c0cea2057-home" (OuterVolumeSpecName: "home") pod "0f3d5340-6181-48e3-9875-899c0cea2057" (UID: "0f3d5340-6181-48e3-9875-899c0cea2057"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:11:32.758347 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:32.758342    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f3d5340-6181-48e3-9875-899c0cea2057-home\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 19:11:32.758460 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:32.758353    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f3d5340-6181-48e3-9875-899c0cea2057-tmp-dir\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 19:11:32.758460 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:32.758403    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3d5340-6181-48e3-9875-899c0cea2057-model-cache" (OuterVolumeSpecName: "model-cache") pod "0f3d5340-6181-48e3-9875-899c0cea2057" (UID: "0f3d5340-6181-48e3-9875-899c0cea2057"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:11:32.760388 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:32.760359    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f3d5340-6181-48e3-9875-899c0cea2057-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0f3d5340-6181-48e3-9875-899c0cea2057" (UID: "0f3d5340-6181-48e3-9875-899c0cea2057"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 19:11:32.760388 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:32.760372    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3d5340-6181-48e3-9875-899c0cea2057-dshm" (OuterVolumeSpecName: "dshm") pod "0f3d5340-6181-48e3-9875-899c0cea2057" (UID: "0f3d5340-6181-48e3-9875-899c0cea2057"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 19:11:32.858836 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:32.858720    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkd5g\" (UniqueName: \"kubernetes.io/projected/0f3d5340-6181-48e3-9875-899c0cea2057-kube-api-access-bkd5g\") pod \"0f3d5340-6181-48e3-9875-899c0cea2057\" (UID: \"0f3d5340-6181-48e3-9875-899c0cea2057\") "
Jul 27 19:11:32.858987 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:32.858893    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f3d5340-6181-48e3-9875-899c0cea2057-dshm\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 19:11:32.858987 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:32.858906    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f3d5340-6181-48e3-9875-899c0cea2057-tls-certs\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 19:11:32.858987 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:32.858915    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f3d5340-6181-48e3-9875-899c0cea2057-model-cache\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 19:11:32.860928 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:32.860901    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f3d5340-6181-48e3-9875-899c0cea2057-kube-api-access-bkd5g" (OuterVolumeSpecName: "kube-api-access-bkd5g") pod "0f3d5340-6181-48e3-9875-899c0cea2057" (UID: "0f3d5340-6181-48e3-9875-899c0cea2057"). InnerVolumeSpecName "kube-api-access-bkd5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 19:11:32.915376 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:32.915340    2557 generic.go:356] "Generic (PLEG): container finished" podID="0f3d5340-6181-48e3-9875-899c0cea2057" containerID="86a1c16c7c119877f9f9d03ee356e6013f93d19c739b7cee388ca67bd0b1876b" exitCode=0
Jul 27 19:11:32.915533 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:32.915383    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx"
Jul 27 19:11:32.915533 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:32.915422    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx" event={"ID":"0f3d5340-6181-48e3-9875-899c0cea2057","Type":"ContainerDied","Data":"86a1c16c7c119877f9f9d03ee356e6013f93d19c739b7cee388ca67bd0b1876b"}
Jul 27 19:11:32.915533 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:32.915472    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx" event={"ID":"0f3d5340-6181-48e3-9875-899c0cea2057","Type":"ContainerDied","Data":"0e0d003f01436bcbc785b56a9bdc4212d7b141f14bd877f48fc6cdc3820f7dcf"}
Jul 27 19:11:32.915533 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:32.915493    2557 scope.go:117] "RemoveContainer" containerID="86a1c16c7c119877f9f9d03ee356e6013f93d19c739b7cee388ca67bd0b1876b"
Jul 27 19:11:32.933680 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:32.933660    2557 scope.go:117] "RemoveContainer" containerID="86a1c16c7c119877f9f9d03ee356e6013f93d19c739b7cee388ca67bd0b1876b"
Jul 27 19:11:32.933981 ip-10-0-131-114 kubenswrapper[2557]: E0727 19:11:32.933959    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86a1c16c7c119877f9f9d03ee356e6013f93d19c739b7cee388ca67bd0b1876b\": container with ID starting with 86a1c16c7c119877f9f9d03ee356e6013f93d19c739b7cee388ca67bd0b1876b not found: ID does not exist" containerID="86a1c16c7c119877f9f9d03ee356e6013f93d19c739b7cee388ca67bd0b1876b"
Jul 27 19:11:32.934034 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:32.933989    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86a1c16c7c119877f9f9d03ee356e6013f93d19c739b7cee388ca67bd0b1876b"} err="failed to get container status \"86a1c16c7c119877f9f9d03ee356e6013f93d19c739b7cee388ca67bd0b1876b\": rpc error: code = NotFound desc = could not find container \"86a1c16c7c119877f9f9d03ee356e6013f93d19c739b7cee388ca67bd0b1876b\": container with ID starting with 86a1c16c7c119877f9f9d03ee356e6013f93d19c739b7cee388ca67bd0b1876b not found: ID does not exist"
Jul 27 19:11:32.941139 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:32.941116    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx"]
Jul 27 19:11:32.944793 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:32.944770    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-lrvvx"]
Jul 27 19:11:32.960129 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:32.960107    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkd5g\" (UniqueName: \"kubernetes.io/projected/0f3d5340-6181-48e3-9875-899c0cea2057-kube-api-access-bkd5g\") on node \"ip-10-0-131-114.ec2.internal\" DevicePath \"\""
Jul 27 19:11:34.119124 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:34.119091    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f3d5340-6181-48e3-9875-899c0cea2057" path="/var/lib/kubelet/pods/0f3d5340-6181-48e3-9875-899c0cea2057/volumes"
Jul 27 19:11:53.829628 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:53.829539    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-9c4f55bbb-9kklv"]
Jul 27 19:11:53.830182 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:53.829976    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f3d5340-6181-48e3-9875-899c0cea2057" containerName="main"
Jul 27 19:11:53.830182 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:53.829992    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f3d5340-6181-48e3-9875-899c0cea2057" containerName="main"
Jul 27 19:11:53.830182 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:53.830072    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f3d5340-6181-48e3-9875-899c0cea2057" containerName="main"
Jul 27 19:11:53.834448 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:53.834430    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-9c4f55bbb-9kklv"
Jul 27 19:11:53.837050 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:53.837029    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-zgd9l\""
Jul 27 19:11:53.837176 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:53.837090    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 27 19:11:53.842361 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:53.842340    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-9c4f55bbb-9kklv"]
Jul 27 19:11:53.937307 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:53.937267    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wcvj\" (UniqueName: \"kubernetes.io/projected/7f9d8262-b42e-4a50-9165-138ffb702a6a-kube-api-access-6wcvj\") pod \"llmisvc-controller-manager-9c4f55bbb-9kklv\" (UID: \"7f9d8262-b42e-4a50-9165-138ffb702a6a\") " pod="kserve/llmisvc-controller-manager-9c4f55bbb-9kklv"
Jul 27 19:11:53.937487 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:53.937417    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f9d8262-b42e-4a50-9165-138ffb702a6a-cert\") pod \"llmisvc-controller-manager-9c4f55bbb-9kklv\" (UID: \"7f9d8262-b42e-4a50-9165-138ffb702a6a\") " pod="kserve/llmisvc-controller-manager-9c4f55bbb-9kklv"
Jul 27 19:11:54.038517 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:54.038483    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f9d8262-b42e-4a50-9165-138ffb702a6a-cert\") pod \"llmisvc-controller-manager-9c4f55bbb-9kklv\" (UID: \"7f9d8262-b42e-4a50-9165-138ffb702a6a\") " pod="kserve/llmisvc-controller-manager-9c4f55bbb-9kklv"
Jul 27 19:11:54.038693 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:54.038545    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wcvj\" (UniqueName: \"kubernetes.io/projected/7f9d8262-b42e-4a50-9165-138ffb702a6a-kube-api-access-6wcvj\") pod \"llmisvc-controller-manager-9c4f55bbb-9kklv\" (UID: \"7f9d8262-b42e-4a50-9165-138ffb702a6a\") " pod="kserve/llmisvc-controller-manager-9c4f55bbb-9kklv"
Jul 27 19:11:54.041011 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:54.040986    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f9d8262-b42e-4a50-9165-138ffb702a6a-cert\") pod \"llmisvc-controller-manager-9c4f55bbb-9kklv\" (UID: \"7f9d8262-b42e-4a50-9165-138ffb702a6a\") " pod="kserve/llmisvc-controller-manager-9c4f55bbb-9kklv"
Jul 27 19:11:54.046001 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:54.045968    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wcvj\" (UniqueName: \"kubernetes.io/projected/7f9d8262-b42e-4a50-9165-138ffb702a6a-kube-api-access-6wcvj\") pod \"llmisvc-controller-manager-9c4f55bbb-9kklv\" (UID: \"7f9d8262-b42e-4a50-9165-138ffb702a6a\") " pod="kserve/llmisvc-controller-manager-9c4f55bbb-9kklv"
Jul 27 19:11:54.144732 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:54.144648    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-9c4f55bbb-9kklv"
Jul 27 19:11:54.286263 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:54.286118    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-9c4f55bbb-9kklv"]
Jul 27 19:11:54.287966 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:54.287943    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 19:11:55.018610 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:55.018571    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-9c4f55bbb-9kklv" event={"ID":"7f9d8262-b42e-4a50-9165-138ffb702a6a","Type":"ContainerStarted","Data":"abb66d22b6f816a87971428eb5eefdec8d81da57af18635693eb95354cd6f77c"}
Jul 27 19:11:58.035416 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:58.035381    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-9c4f55bbb-9kklv" event={"ID":"7f9d8262-b42e-4a50-9165-138ffb702a6a","Type":"ContainerStarted","Data":"a200cbfe58d9a46605482d76971e51823eb80c1a6817db66f1b41b064ca779fa"}
Jul 27 19:11:58.035893 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:58.035597    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-9c4f55bbb-9kklv"
Jul 27 19:11:58.050208 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:11:58.050154    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-9c4f55bbb-9kklv" podStartSLOduration=2.023213572 podStartE2EDuration="5.050135589s" totalImagesPullingTime="3.026922017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:11:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:11:58.049813409 +0000 UTC m=+6210.536624710" watchObservedRunningTime="2026-07-27 19:11:58.050135589 +0000 UTC m=+6210.536946877"
Jul 27 19:12:29.042426 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:12:29.042392    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-9c4f55bbb-9kklv"
Jul 27 19:14:18.206262 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:18.206227    2557 ???:1] "http: TLS handshake error from 10.0.131.74:53118: EOF"
Jul 27 19:14:18.527720 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:18.527629    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-w4xqt_37c62431-f7c3-409a-8d32-15cfa386cec4/authorino/0.log"
Jul 27 19:14:18.554678 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:18.554646    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-447cz_a1235f6f-c04c-46d7-aa23-0c2c78f13c21/manager/0.log"
Jul 27 19:14:18.569212 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:18.569185    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ncggk_1244ce20-3490-453e-8bf3-716230099686/kuadrant-console-plugin/0.log"
Jul 27 19:14:18.703393 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:18.703351    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-5z76z_3882525b-8726-46b2-b717-9736e21fd723/manager/0.log"
Jul 27 19:14:18.826719 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:18.826626    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-5z76z_3882525b-8726-46b2-b717-9736e21fd723/manager/1.log"
Jul 27 19:14:19.873167 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:19.873130    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-w4xqt_37c62431-f7c3-409a-8d32-15cfa386cec4/authorino/0.log"
Jul 27 19:14:19.911442 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:19.911407    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-447cz_a1235f6f-c04c-46d7-aa23-0c2c78f13c21/manager/0.log"
Jul 27 19:14:19.923544 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:19.923516    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ncggk_1244ce20-3490-453e-8bf3-716230099686/kuadrant-console-plugin/0.log"
Jul 27 19:14:20.053630 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:20.053597    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-5z76z_3882525b-8726-46b2-b717-9736e21fd723/manager/0.log"
Jul 27 19:14:20.184241 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:20.184207    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-5z76z_3882525b-8726-46b2-b717-9736e21fd723/manager/1.log"
Jul 27 19:14:21.194461 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:21.194424    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-w4xqt_37c62431-f7c3-409a-8d32-15cfa386cec4/authorino/0.log"
Jul 27 19:14:21.228731 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:21.228696    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-447cz_a1235f6f-c04c-46d7-aa23-0c2c78f13c21/manager/0.log"
Jul 27 19:14:21.241516 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:21.241490    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ncggk_1244ce20-3490-453e-8bf3-716230099686/kuadrant-console-plugin/0.log"
Jul 27 19:14:21.367659 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:21.367627    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-5z76z_3882525b-8726-46b2-b717-9736e21fd723/manager/0.log"
Jul 27 19:14:21.491459 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:21.491371    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-5z76z_3882525b-8726-46b2-b717-9736e21fd723/manager/1.log"
Jul 27 19:14:22.488449 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:22.488394    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-w4xqt_37c62431-f7c3-409a-8d32-15cfa386cec4/authorino/0.log"
Jul 27 19:14:22.519326 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:22.519299    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-447cz_a1235f6f-c04c-46d7-aa23-0c2c78f13c21/manager/0.log"
Jul 27 19:14:22.532729 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:22.532709    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ncggk_1244ce20-3490-453e-8bf3-716230099686/kuadrant-console-plugin/0.log"
Jul 27 19:14:22.652492 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:22.652457    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-5z76z_3882525b-8726-46b2-b717-9736e21fd723/manager/0.log"
Jul 27 19:14:22.779999 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:22.779893    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-5z76z_3882525b-8726-46b2-b717-9736e21fd723/manager/1.log"
Jul 27 19:14:23.776367 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:23.776326    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-w4xqt_37c62431-f7c3-409a-8d32-15cfa386cec4/authorino/0.log"
Jul 27 19:14:23.813353 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:23.813324    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-447cz_a1235f6f-c04c-46d7-aa23-0c2c78f13c21/manager/0.log"
Jul 27 19:14:23.826436 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:23.826416    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ncggk_1244ce20-3490-453e-8bf3-716230099686/kuadrant-console-plugin/0.log"
Jul 27 19:14:23.947647 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:23.947619    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-5z76z_3882525b-8726-46b2-b717-9736e21fd723/manager/0.log"
Jul 27 19:14:24.074586 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:24.074470    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-5z76z_3882525b-8726-46b2-b717-9736e21fd723/manager/1.log"
Jul 27 19:14:25.075618 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:25.075587    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-w4xqt_37c62431-f7c3-409a-8d32-15cfa386cec4/authorino/0.log"
Jul 27 19:14:25.112090 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:25.112052    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-447cz_a1235f6f-c04c-46d7-aa23-0c2c78f13c21/manager/0.log"
Jul 27 19:14:25.125031 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:25.125008    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ncggk_1244ce20-3490-453e-8bf3-716230099686/kuadrant-console-plugin/0.log"
Jul 27 19:14:25.246837 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:25.246772    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-5z76z_3882525b-8726-46b2-b717-9736e21fd723/manager/0.log"
Jul 27 19:14:25.385243 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:25.385145    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-5z76z_3882525b-8726-46b2-b717-9736e21fd723/manager/1.log"
Jul 27 19:14:26.384859 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:26.384821    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-w4xqt_37c62431-f7c3-409a-8d32-15cfa386cec4/authorino/0.log"
Jul 27 19:14:26.421124 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:26.421089    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-447cz_a1235f6f-c04c-46d7-aa23-0c2c78f13c21/manager/0.log"
Jul 27 19:14:26.434723 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:26.434698    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ncggk_1244ce20-3490-453e-8bf3-716230099686/kuadrant-console-plugin/0.log"
Jul 27 19:14:26.563667 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:26.563626    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-5z76z_3882525b-8726-46b2-b717-9736e21fd723/manager/0.log"
Jul 27 19:14:26.688452 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:26.688421    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-5z76z_3882525b-8726-46b2-b717-9736e21fd723/manager/1.log"
Jul 27 19:14:27.680669 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:27.680633    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-w4xqt_37c62431-f7c3-409a-8d32-15cfa386cec4/authorino/0.log"
Jul 27 19:14:27.714451 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:27.714415    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-447cz_a1235f6f-c04c-46d7-aa23-0c2c78f13c21/manager/0.log"
Jul 27 19:14:27.726527 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:27.726504    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ncggk_1244ce20-3490-453e-8bf3-716230099686/kuadrant-console-plugin/0.log"
Jul 27 19:14:27.858081 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:27.858004    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-5z76z_3882525b-8726-46b2-b717-9736e21fd723/manager/0.log"
Jul 27 19:14:27.964433 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:27.964335    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-5z76z_3882525b-8726-46b2-b717-9736e21fd723/manager/1.log"
Jul 27 19:14:28.946807 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:28.946747    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-w4xqt_37c62431-f7c3-409a-8d32-15cfa386cec4/authorino/0.log"
Jul 27 19:14:28.980197 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:28.980162    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-447cz_a1235f6f-c04c-46d7-aa23-0c2c78f13c21/manager/0.log"
Jul 27 19:14:28.991826 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:28.991803    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ncggk_1244ce20-3490-453e-8bf3-716230099686/kuadrant-console-plugin/0.log"
Jul 27 19:14:29.112336 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:29.112300    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-5z76z_3882525b-8726-46b2-b717-9736e21fd723/manager/0.log"
Jul 27 19:14:29.236788 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:29.236685    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-5z76z_3882525b-8726-46b2-b717-9736e21fd723/manager/1.log"
Jul 27 19:14:30.260320 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:30.260276    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-w4xqt_37c62431-f7c3-409a-8d32-15cfa386cec4/authorino/0.log"
Jul 27 19:14:30.295845 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:30.295812    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-447cz_a1235f6f-c04c-46d7-aa23-0c2c78f13c21/manager/0.log"
Jul 27 19:14:30.311445 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:30.311423    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ncggk_1244ce20-3490-453e-8bf3-716230099686/kuadrant-console-plugin/0.log"
Jul 27 19:14:30.433821 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:30.433785    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-5z76z_3882525b-8726-46b2-b717-9736e21fd723/manager/0.log"
Jul 27 19:14:30.560182 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:30.560084    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-5z76z_3882525b-8726-46b2-b717-9736e21fd723/manager/1.log"
Jul 27 19:14:31.555835 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:31.555802    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-w4xqt_37c62431-f7c3-409a-8d32-15cfa386cec4/authorino/0.log"
Jul 27 19:14:31.588838 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:31.588805    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-447cz_a1235f6f-c04c-46d7-aa23-0c2c78f13c21/manager/0.log"
Jul 27 19:14:31.601283 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:31.601257    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ncggk_1244ce20-3490-453e-8bf3-716230099686/kuadrant-console-plugin/0.log"
Jul 27 19:14:31.720644 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:31.720614    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-5z76z_3882525b-8726-46b2-b717-9736e21fd723/manager/0.log"
Jul 27 19:14:31.846506 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:31.846409    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-5z76z_3882525b-8726-46b2-b717-9736e21fd723/manager/1.log"
Jul 27 19:14:32.833310 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:32.833278    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-w4xqt_37c62431-f7c3-409a-8d32-15cfa386cec4/authorino/0.log"
Jul 27 19:14:32.872679 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:32.872640    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-447cz_a1235f6f-c04c-46d7-aa23-0c2c78f13c21/manager/0.log"
Jul 27 19:14:32.884465 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:32.884440    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ncggk_1244ce20-3490-453e-8bf3-716230099686/kuadrant-console-plugin/0.log"
Jul 27 19:14:33.007525 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:33.007495    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-5z76z_3882525b-8726-46b2-b717-9736e21fd723/manager/0.log"
Jul 27 19:14:33.142634 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:33.142543    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-5z76z_3882525b-8726-46b2-b717-9736e21fd723/manager/1.log"
Jul 27 19:14:34.139706 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:34.139671    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-w4xqt_37c62431-f7c3-409a-8d32-15cfa386cec4/authorino/0.log"
Jul 27 19:14:34.179567 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:34.179535    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-447cz_a1235f6f-c04c-46d7-aa23-0c2c78f13c21/manager/0.log"
Jul 27 19:14:34.195587 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:34.195561    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ncggk_1244ce20-3490-453e-8bf3-716230099686/kuadrant-console-plugin/0.log"
Jul 27 19:14:34.324949 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:34.324909    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-5z76z_3882525b-8726-46b2-b717-9736e21fd723/manager/0.log"
Jul 27 19:14:34.451427 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:34.451398    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-5z76z_3882525b-8726-46b2-b717-9736e21fd723/manager/1.log"
Jul 27 19:14:39.269829 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:39.269798    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zzcb2_8eac848d-fc37-4bc7-b58d-54eaf719c28f/global-pull-secret-syncer/0.log"
Jul 27 19:14:39.464451 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:39.464422    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xqqgl_9086a46f-a7bf-4e78-b049-1a1d363043a8/konnectivity-agent/0.log"
Jul 27 19:14:39.500567 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:39.500532    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-114.ec2.internal_14088cbd064fda87605e72bf989bc1ec/haproxy/0.log"
Jul 27 19:14:44.522014 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:44.521982    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-w4xqt_37c62431-f7c3-409a-8d32-15cfa386cec4/authorino/0.log"
Jul 27 19:14:44.606885 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:44.606842    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-447cz_a1235f6f-c04c-46d7-aa23-0c2c78f13c21/manager/0.log"
Jul 27 19:14:44.630887 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:44.630860    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-ncggk_1244ce20-3490-453e-8bf3-716230099686/kuadrant-console-plugin/0.log"
Jul 27 19:14:44.768573 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:44.768535    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-5z76z_3882525b-8726-46b2-b717-9736e21fd723/manager/0.log"
Jul 27 19:14:44.909156 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:44.909114    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-5z76z_3882525b-8726-46b2-b717-9736e21fd723/manager/1.log"
Jul 27 19:14:46.230976 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:46.230948    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b06529a0-2a71-4cfa-a22f-096a4c144227/alertmanager/0.log"
Jul 27 19:14:46.252125 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:46.252074    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b06529a0-2a71-4cfa-a22f-096a4c144227/config-reloader/0.log"
Jul 27 19:14:46.275166 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:46.275138    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b06529a0-2a71-4cfa-a22f-096a4c144227/kube-rbac-proxy-web/0.log"
Jul 27 19:14:46.293832 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:46.293803    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b06529a0-2a71-4cfa-a22f-096a4c144227/kube-rbac-proxy/0.log"
Jul 27 19:14:46.314886 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:46.314849    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b06529a0-2a71-4cfa-a22f-096a4c144227/kube-rbac-proxy-metric/0.log"
Jul 27 19:14:46.338777 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:46.338745    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b06529a0-2a71-4cfa-a22f-096a4c144227/prom-label-proxy/0.log"
Jul 27 19:14:46.356543 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:46.356514    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b06529a0-2a71-4cfa-a22f-096a4c144227/init-config-reloader/0.log"
Jul 27 19:14:46.591381 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:46.591355    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-pxsf7_3ccf95fe-63a3-4237-be84-db12771f35cb/kube-state-metrics/0.log"
Jul 27 19:14:46.613921 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:46.613898    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-pxsf7_3ccf95fe-63a3-4237-be84-db12771f35cb/kube-rbac-proxy-main/0.log"
Jul 27 19:14:46.630420 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:46.630398    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-pxsf7_3ccf95fe-63a3-4237-be84-db12771f35cb/kube-rbac-proxy-self/0.log"
Jul 27 19:14:46.741809 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:46.741783    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ct2bc_4e5e4d7c-9bef-4d41-a9b9-22885fe0213b/node-exporter/0.log"
Jul 27 19:14:46.771897 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:46.771858    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ct2bc_4e5e4d7c-9bef-4d41-a9b9-22885fe0213b/kube-rbac-proxy/0.log"
Jul 27 19:14:46.797907 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:46.797876    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ct2bc_4e5e4d7c-9bef-4d41-a9b9-22885fe0213b/init-textfile/0.log"
Jul 27 19:14:46.992957 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:46.992921    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-xsxnf_a2877fd1-e9e9-400c-b9b2-e90d0590e13c/kube-rbac-proxy-main/0.log"
Jul 27 19:14:47.013327 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:47.013297    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-xsxnf_a2877fd1-e9e9-400c-b9b2-e90d0590e13c/kube-rbac-proxy-self/0.log"
Jul 27 19:14:47.038487 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:47.038457    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-xsxnf_a2877fd1-e9e9-400c-b9b2-e90d0590e13c/openshift-state-metrics/0.log"
Jul 27 19:14:47.100080 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:47.099983    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_db45ee80-77ff-49f5-a4ab-698f2ed851e8/prometheus/0.log"
Jul 27 19:14:47.122415 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:47.122391    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_db45ee80-77ff-49f5-a4ab-698f2ed851e8/config-reloader/0.log"
Jul 27 19:14:47.146185 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:47.146161    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_db45ee80-77ff-49f5-a4ab-698f2ed851e8/thanos-sidecar/0.log"
Jul 27 19:14:47.168012 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:47.167990    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_db45ee80-77ff-49f5-a4ab-698f2ed851e8/kube-rbac-proxy-web/0.log"
Jul 27 19:14:47.196846 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:47.196819    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_db45ee80-77ff-49f5-a4ab-698f2ed851e8/kube-rbac-proxy/0.log"
Jul 27 19:14:47.218147 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:47.218118    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_db45ee80-77ff-49f5-a4ab-698f2ed851e8/kube-rbac-proxy-thanos/0.log"
Jul 27 19:14:47.240660 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:47.240634    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_db45ee80-77ff-49f5-a4ab-698f2ed851e8/init-config-reloader/0.log"
Jul 27 19:14:48.598748 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:48.598715    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-k5llc_3d458fd4-bb2f-4907-9431-226cac065ce0/networking-console-plugin/0.log"
Jul 27 19:14:49.084088 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:49.084046    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-r8n2x/perf-node-gather-daemonset-2jlmt"]
Jul 27 19:14:49.087607 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:49.087583    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-2jlmt"
Jul 27 19:14:49.089516 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:49.089488    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-r8n2x\"/\"default-dockercfg-xsd9b\""
Jul 27 19:14:49.089634 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:49.089488    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r8n2x\"/\"openshift-service-ca.crt\""
Jul 27 19:14:49.089944 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:49.089928    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r8n2x\"/\"kube-root-ca.crt\""
Jul 27 19:14:49.095263 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:49.095242    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r8n2x/perf-node-gather-daemonset-2jlmt"]
Jul 27 19:14:49.111638 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:49.111602    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bbcc661-e7f6-4edb-a28f-f16d1175b3a2-sys\") pod \"perf-node-gather-daemonset-2jlmt\" (UID: \"5bbcc661-e7f6-4edb-a28f-f16d1175b3a2\") " pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-2jlmt"
Jul 27 19:14:49.111847 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:49.111667    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcft7\" (UniqueName: \"kubernetes.io/projected/5bbcc661-e7f6-4edb-a28f-f16d1175b3a2-kube-api-access-fcft7\") pod \"perf-node-gather-daemonset-2jlmt\" (UID: \"5bbcc661-e7f6-4edb-a28f-f16d1175b3a2\") " pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-2jlmt"
Jul 27 19:14:49.111847 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:49.111722    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5bbcc661-e7f6-4edb-a28f-f16d1175b3a2-lib-modules\") pod \"perf-node-gather-daemonset-2jlmt\" (UID: \"5bbcc661-e7f6-4edb-a28f-f16d1175b3a2\") " pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-2jlmt"
Jul 27 19:14:49.111847 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:49.111771    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5bbcc661-e7f6-4edb-a28f-f16d1175b3a2-podres\") pod \"perf-node-gather-daemonset-2jlmt\" (UID: \"5bbcc661-e7f6-4edb-a28f-f16d1175b3a2\") " pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-2jlmt"
Jul 27 19:14:49.112007 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:49.111878    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5bbcc661-e7f6-4edb-a28f-f16d1175b3a2-proc\") pod \"perf-node-gather-daemonset-2jlmt\" (UID: \"5bbcc661-e7f6-4edb-a28f-f16d1175b3a2\") " pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-2jlmt"
Jul 27 19:14:49.212537 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:49.212503    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5bbcc661-e7f6-4edb-a28f-f16d1175b3a2-lib-modules\") pod \"perf-node-gather-daemonset-2jlmt\" (UID: \"5bbcc661-e7f6-4edb-a28f-f16d1175b3a2\") " pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-2jlmt"
Jul 27 19:14:49.212739 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:49.212542    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5bbcc661-e7f6-4edb-a28f-f16d1175b3a2-podres\") pod \"perf-node-gather-daemonset-2jlmt\" (UID: \"5bbcc661-e7f6-4edb-a28f-f16d1175b3a2\") " pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-2jlmt"
Jul 27 19:14:49.212739 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:49.212694    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5bbcc661-e7f6-4edb-a28f-f16d1175b3a2-proc\") pod \"perf-node-gather-daemonset-2jlmt\" (UID: \"5bbcc661-e7f6-4edb-a28f-f16d1175b3a2\") " pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-2jlmt"
Jul 27 19:14:49.212739 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:49.212709    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5bbcc661-e7f6-4edb-a28f-f16d1175b3a2-lib-modules\") pod \"perf-node-gather-daemonset-2jlmt\" (UID: \"5bbcc661-e7f6-4edb-a28f-f16d1175b3a2\") " pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-2jlmt"
Jul 27 19:14:49.212958 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:49.212744    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bbcc661-e7f6-4edb-a28f-f16d1175b3a2-sys\") pod \"perf-node-gather-daemonset-2jlmt\" (UID: \"5bbcc661-e7f6-4edb-a28f-f16d1175b3a2\") " pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-2jlmt"
Jul 27 19:14:49.212958 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:49.212878    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcft7\" (UniqueName: \"kubernetes.io/projected/5bbcc661-e7f6-4edb-a28f-f16d1175b3a2-kube-api-access-fcft7\") pod \"perf-node-gather-daemonset-2jlmt\" (UID: \"5bbcc661-e7f6-4edb-a28f-f16d1175b3a2\") " pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-2jlmt"
Jul 27 19:14:49.212958 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:49.212932    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bbcc661-e7f6-4edb-a28f-f16d1175b3a2-sys\") pod \"perf-node-gather-daemonset-2jlmt\" (UID: \"5bbcc661-e7f6-4edb-a28f-f16d1175b3a2\") " pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-2jlmt"
Jul 27 19:14:49.212958 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:49.212943    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5bbcc661-e7f6-4edb-a28f-f16d1175b3a2-podres\") pod \"perf-node-gather-daemonset-2jlmt\" (UID: \"5bbcc661-e7f6-4edb-a28f-f16d1175b3a2\") " pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-2jlmt"
Jul 27 19:14:49.213178 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:49.213091    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5bbcc661-e7f6-4edb-a28f-f16d1175b3a2-proc\") pod \"perf-node-gather-daemonset-2jlmt\" (UID: \"5bbcc661-e7f6-4edb-a28f-f16d1175b3a2\") " pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-2jlmt"
Jul 27 19:14:49.220386 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:49.220357    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcft7\" (UniqueName: \"kubernetes.io/projected/5bbcc661-e7f6-4edb-a28f-f16d1175b3a2-kube-api-access-fcft7\") pod \"perf-node-gather-daemonset-2jlmt\" (UID: \"5bbcc661-e7f6-4edb-a28f-f16d1175b3a2\") " pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-2jlmt"
Jul 27 19:14:49.398503 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:49.398404    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-2jlmt"
Jul 27 19:14:49.531388 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:49.531361    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r8n2x/perf-node-gather-daemonset-2jlmt"]
Jul 27 19:14:49.533367 ip-10-0-131-114 kubenswrapper[2557]: W0727 19:14:49.533337    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5bbcc661_e7f6_4edb_a28f_f16d1175b3a2.slice/crio-48b83b27d33b36b261ed60d0acd7e3ea2263b25f09b1eab20a43459eb47114b1 WatchSource:0}: Error finding container 48b83b27d33b36b261ed60d0acd7e3ea2263b25f09b1eab20a43459eb47114b1: Status 404 returned error can't find the container with id 48b83b27d33b36b261ed60d0acd7e3ea2263b25f09b1eab20a43459eb47114b1
Jul 27 19:14:49.607871 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:49.607848    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-s7hbx_823de302-a6b7-4e92-8cea-80322d2507ce/download-server/0.log"
Jul 27 19:14:49.887826 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:49.887788    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-2jlmt" event={"ID":"5bbcc661-e7f6-4edb-a28f-f16d1175b3a2","Type":"ContainerStarted","Data":"b181576ce9f6ec2f9eceb74b4bb0b8c9dbf00b136c4ed83104df01b2968efae5"}
Jul 27 19:14:49.888039 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:49.887849    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-2jlmt"
Jul 27 19:14:49.888039 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:49.887864    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-2jlmt" event={"ID":"5bbcc661-e7f6-4edb-a28f-f16d1175b3a2","Type":"ContainerStarted","Data":"48b83b27d33b36b261ed60d0acd7e3ea2263b25f09b1eab20a43459eb47114b1"}
Jul 27 19:14:49.902218 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:49.902135    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-2jlmt" podStartSLOduration=0.902117762 podStartE2EDuration="902.117762ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:14:49.901638293 +0000 UTC m=+6382.388449581" watchObservedRunningTime="2026-07-27 19:14:49.902117762 +0000 UTC m=+6382.388929052"
Jul 27 19:14:51.021236 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:51.021207    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pj6vq_2c6446ac-f2c3-4897-a995-94176302c6b6/dns/0.log"
Jul 27 19:14:51.039293 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:51.039264    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pj6vq_2c6446ac-f2c3-4897-a995-94176302c6b6/kube-rbac-proxy/0.log"
Jul 27 19:14:51.173724 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:51.173695    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k56kz_17308a6d-e821-4aae-afe4-592c1f5f0175/dns-node-resolver/0.log"
Jul 27 19:14:51.789536 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:51.789507    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q8nkb_231ae1f3-5af5-445f-92d1-2a8901275b9f/node-ca/0.log"
Jul 27 19:14:53.671588 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:53.671558    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-f7c9g_5f29005f-0e4f-4b5f-9e9a-8a2c3089fd08/serve-healthcheck-canary/0.log"
Jul 27 19:14:54.214153 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:54.214122    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cz9hj_5a1f84b3-6930-4b20-95e9-75d46cddb36b/kube-rbac-proxy/0.log"
Jul 27 19:14:54.234290 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:54.234267    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cz9hj_5a1f84b3-6930-4b20-95e9-75d46cddb36b/exporter/0.log"
Jul 27 19:14:54.255779 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:54.255740    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cz9hj_5a1f84b3-6930-4b20-95e9-75d46cddb36b/extractor/0.log"
Jul 27 19:14:55.905141 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:55.905115    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-r8n2x/perf-node-gather-daemonset-2jlmt"
Jul 27 19:14:57.142222 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:57.142193    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-67d9b7fdbd-h8v7t_564cbc55-94c2-475f-9577-6006b84d1177/manager/0.log"
Jul 27 19:14:57.881594 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:57.881555    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-9c4f55bbb-9kklv_7f9d8262-b42e-4a50-9165-138ffb702a6a/manager/0.log"
Jul 27 19:14:59.008369 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:14:59.008327    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-ddz92_b5d18178-0bcc-449b-ac6a-0e7f41c0ea41/manager/0.log"
Jul 27 19:15:03.860779 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:15:03.860730    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-wc267_f3577d8f-d13f-4cb9-948f-3d09033f62d9/migrator/0.log"
Jul 27 19:15:03.884606 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:15:03.884573    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-wc267_f3577d8f-d13f-4cb9-948f-3d09033f62d9/graceful-termination/0.log"
Jul 27 19:15:04.281235 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:15:04.281177    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t_0a3ad2c9-8fab-442a-9ec6-f8eacd33bc22/kube-storage-version-migrator-operator/1.log"
Jul 27 19:15:04.283553 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:15:04.283525    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-8fq2t_0a3ad2c9-8fab-442a-9ec6-f8eacd33bc22/kube-storage-version-migrator-operator/0.log"
Jul 27 19:15:05.335007 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:15:05.334968    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w7blt_ee935ed2-91cd-41fb-a3c3-fc5c52f42882/kube-multus-additional-cni-plugins/0.log"
Jul 27 19:15:05.356313 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:15:05.356282    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w7blt_ee935ed2-91cd-41fb-a3c3-fc5c52f42882/egress-router-binary-copy/0.log"
Jul 27 19:15:05.375924 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:15:05.375895    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w7blt_ee935ed2-91cd-41fb-a3c3-fc5c52f42882/cni-plugins/0.log"
Jul 27 19:15:05.396107 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:15:05.396075    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w7blt_ee935ed2-91cd-41fb-a3c3-fc5c52f42882/bond-cni-plugin/0.log"
Jul 27 19:15:05.419857 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:15:05.419833    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w7blt_ee935ed2-91cd-41fb-a3c3-fc5c52f42882/routeoverride-cni/0.log"
Jul 27 19:15:05.441523 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:15:05.441500    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w7blt_ee935ed2-91cd-41fb-a3c3-fc5c52f42882/whereabouts-cni-bincopy/0.log"
Jul 27 19:15:05.462606 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:15:05.462580    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w7blt_ee935ed2-91cd-41fb-a3c3-fc5c52f42882/whereabouts-cni/0.log"
Jul 27 19:15:05.743679 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:15:05.743646    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-tr5lq_3ffc168b-79b8-46ef-8f17-636a697cb4e7/kube-multus/0.log"
Jul 27 19:15:05.889477 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:15:05.889448    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mzhnc_57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec/network-metrics-daemon/0.log"
Jul 27 19:15:05.910899 ip-10-0-131-114 kubenswrapper[2557]: I0727 19:15:05.910871    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mzhnc_57ad8e2d-11d3-4f9d-bc88-78eec6fc0eec/kube-rbac-proxy/0.log"